Merge branch 'master' into perf/optimizer

.github/workflows/pull-request.yaml

@@ -24,6 +24,7 @@ jobs:
         with:
           types: |
             feat
+            perf
             fix
             chore
             refactor

.gitignore

@@ -49,4 +49,4 @@ vyper/vyper_git_commithash.txt
 *.spec
 
 # mac
-.DS_Store
+.DS_Store

FUNDING.json

@@ -0,0 +1,10 @@
+{
+  "drips": {
+    "ethereum": {
+      "ownedBy": "0x70CCBE10F980d80b7eBaab7D2E3A73e87D67B775"
+    }
+  },
+  "opRetro": {
+    "projectId": "0x9ca1f7b0e0d10d3bd2619e51a54f2e4175e029c87a2944cf1ebc89164ba77ea0"
+  }
+}

SECURITY.md

@@ -3,7 +3,7 @@
 ## Supported Versions
 
 - it is recommended to follow the list of known [vulnerabilities](https://github.com/vyperlang/vyper/security/advisories) and stay up-to-date with the latest releases
-  - as of May 2024, the `0.4.0` release is the most secure and the most comprehensively reviewed one and is recommended for use in production environments
+  - as of May 2024, the [`0.4.0`](https://github.com/vyperlang/vyper/releases/tag/v0.4.0) release is the most comprehensively reviewed one and is recommended for use in production environments
 - if a compiler vulnerability is found, a new compiler version with a patch will be released. The vulnerable version itself is not updated (see the examples below).
   - `example1`: suppose `0.4.0` is the latest version and a hypothetical vulnerability is found in `0.4.0`, then a patch will be released in `0.4.1`
   - `example2`: suppose `0.4.0` is the latest version and a hypothetical vulnerability is found both in `0.3.10` and `0.4.0`, then a patch will be released only in `0.4.1`
@@ -26,7 +26,22 @@ we will add an entry to the list of security advisories for posterity and refere
 
 
 ## Bug Bounty Program
-- as of May 2024, Vyper does not have a bug bounty program. It is planned to instantiate one soon.
+- Vyper runs a bug bounty program via the Ethereum Foundation.
+  - Bugs should be reported through the [Ethereum Foundation's bounty program](https://ethereum.org/bug-bounty).
+
+### Scope
+- Rules from the Ethereum Foundation's bug bounty program apply; for any questions please reach out [here](mailto:bounty@ethereum.org). Here we further clarify the scope of the Vyper bounty program.
+- If a compiler bug affects production code, it is in scope (excluding known issues).
+  - This includes bugs in older compiler versions still used in production.
+- If a compiler bug does not currently affect production but is likely to in the future, it is in scope.
+  - This mainly applies to the latest compiler release (e.g., a new release is available but contracts are not yet deployed with it).
+  - Experimental features (e.g. `--experimental-codegen`) are out of scope, as they are not intended for production and are unlikely to affect production code.
+  - Bugs in older compiler versions are generally out of scope, as they are no longer used for new contracts.
+    - There might be exceptions, e.g., when an L2 doesn't support recent compiler releases. In such cases, it might be reasonable for an older version to be used. It is up to the discretion of the EF & Vyper team to decide if the bug is in scope.
+- If a vulnerability affects multiple contracts, the whitehat is eligible for only one payout (though the severity of the bug may increase).
+- Eligibility for project-specific bounties is independent of this bounty.
+- [Security advisories](https://github.com/vyperlang/vyper/security/advisories) and [known issues](https://github.com/vyperlang/vyper/issues) are not eligible for the bounty program, as they are publicly disclosed and protocols should structure their contracts accordingly.
+- Individuals or organizations contracted or engaged specifically for security development, auditing, or testing of this project are ineligible for the bounty program.
 
 ## Reporting a Vulnerability
 

codecov.yml

@@ -0,0 +1,7 @@
+# https://docs.codecov.com/docs/codecovyml-reference
+coverage:
+  status:
+    project:
+      default:
+        # set threshold given noise in the coverage from fuzzing
+        threshold: 0.5%

docs/built-in-functions.rst

@@ -264,7 +264,7 @@ Vyper has three built-ins for contract creation; all three contract creation bui
             x: uint256 = 123
             success, response = raw_call(
                 _target,
-                _abi_encode(x, method_id=method_id("someMethodName(uint256)")),
+                abi_encode(x, method_id=method_id("someMethodName(uint256)")),
                 max_outsize=32,
                 value=msg.value,
                 revert_on_failure=False
@@ -1023,7 +1023,7 @@ Utilities
         >>> ExampleContract.foo()
 	0xa9059cbb
 
-.. py:function:: _abi_encode(*args, ensure_tuple: bool = True) -> Bytes[<depends on input>]
+.. py:function:: abi_encode(*args, ensure_tuple: bool = True) -> Bytes[<depends on input>]
 
     Takes a variable number of args as input, and returns the ABIv2-encoded bytestring. Used for packing arguments to raw_call, EIP712 and other cases where a consistent and efficient serialization method is needed.
     Once this function has seen more use we provisionally plan to put it into the ``ethereum.abi`` namespace.
@@ -1041,7 +1041,7 @@ Utilities
         def foo() -> Bytes[132]:
             x: uint256 = 1
             y: Bytes[32] = b"234"
-            return _abi_encode(x, y, method_id=method_id("foo()"))
+            return abi_encode(x, y, method_id=method_id("foo()"))
 
     .. code-block:: vyper
 
@@ -1052,15 +1052,18 @@ Utilities
         "0000000000000000000000000000000000000000000000000000000000000003"
         "3233340000000000000000000000000000000000000000000000000000000000"
 
+    .. note::
+        Prior to v0.4.0, this function was named ``_abi_encode``.
+
 
-.. py:function:: _abi_decode(b: Bytes, output_type: type_, unwrap_tuple: bool = True) -> Any
+.. py:function:: abi_decode(b: Bytes, output_type: type_, unwrap_tuple: bool = True) -> Any
 
     Takes a byte array as input, and returns the decoded values according to the specified output types. Used for unpacking ABIv2-encoded values.
     Once this function has seen more use we provisionally plan to put it into the ``ethereum.abi`` namespace.
 
     * ``b``: A byte array of a length that is between the minimum and maximum ABIv2 size bounds of the ``output type``.
     * ``output_type``: Name of the output type, or tuple of output types, to be decoded.
-    * ``unwrap_tuple``: If set to True, the input is decoded as a tuple even if only one output type is specified. In other words, ``_abi_decode(b, Bytes[32])`` gets decoded as ``(Bytes[32],)``. This is the convention for ABIv2-encoded values generated by Vyper and Solidity functions. Except for very specific use cases, this should be set to True. Must be a literal.
+    * ``unwrap_tuple``: If set to True, the input is decoded as a tuple even if only one output type is specified. In other words, ``abi_decode(b, Bytes[32])`` gets decoded as ``(Bytes[32],)``. This is the convention for ABIv2-encoded values generated by Vyper and Solidity functions. Except for very specific use cases, this should be set to True. Must be a literal.
 
     Returns the decoded value(s), with type as specified by `output_type`.
 
@@ -1071,9 +1074,12 @@ Utilities
         def foo(someInput: Bytes[128]) -> (uint256, Bytes[32]):
             x: uint256 = empty(uint256)
             y: Bytes[32] = empty(Bytes[32])
-            x, y =  _abi_decode(someInput, (uint256, Bytes[32]))
+            x, y =  abi_decode(someInput, (uint256, Bytes[32]))
             return x, y
 
+    .. note::
+        Prior to v0.4.0, this function was named ``_abi_decode``.
+
 
 .. py:function:: print(*args, hardhat_compat=False) -> None
 
@@ -1084,3 +1090,6 @@ Utilities
 .. note::
 
     Issuing of the static call is *NOT* mode-dependent (that is, it is not removed from production code), although the compiler will issue a warning whenever ``print`` is used.
+
+.. warning::
+    In Vyper, as of v0.4.0, the order of argument evaluation of builtins is not defined. That means that the compiler may choose to reorder evaluation of arguments. For example, ``extract32(x(), y())`` may yield unexpected results if ``x()`` and ``y()`` both touch the same data. For this reason, it is best to avoid calling functions with side-effects inside of builtins. For more information, see `GHSA-g2xh-c426-v8mf <https://github.com/vyperlang/vyper/security/advisories/GHSA-g2xh-c426-v8mf>`_ and `issue #4019 <https://github.com/vyperlang/vyper/issues/4019>`_.

docs/compiling-a-contract.rst

@@ -6,17 +6,19 @@ Command-Line Compiler Tools
 
 Vyper includes the following command-line scripts for compiling contracts:
 
-* ``vyper``: Compiles vyper contract files into ``IR`` or bytecode
+* ``vyper``: Compiles vyper contract or archive files
 * ``vyper-json``: Provides a JSON interface to the compiler
 
 .. note::
 
     The ``--help`` flag gives verbose explanations of how to use each of these scripts.
 
+.. _vyper-cli-command:
+
 vyper
 -----
 
-``vyper`` provides command-line access to the compiler. It can generate various outputs including simple binaries, ASTs, interfaces and source mappings.
+``vyper`` provides CLI access to the compiler. It can generate various outputs including simple binaries, ASTs, interfaces and source mappings.
 
 To compile a contract:
 
@@ -29,7 +31,7 @@ Include the ``-f`` flag to specify which output formats to return. Use ``vyper -
 
 .. code:: shell
 
-    $ vyper -f abi,abi_python,bytecode,bytecode_runtime,interface,external_interface,ast,annotated_ast,ir,ir_json,ir_runtime,hex-ir,asm,opcodes,opcodes_runtime,source_map,method_identifiers,userdoc,devdoc,metadata,combined_json,layout yourFileName.vy
+    $ vyper -f abi,abi_python,bytecode,bytecode_runtime,blueprint_bytecode,interface,external_interface,ast,annotated_ast,integrity,ir,ir_json,ir_runtime,asm,opcodes,opcodes_runtime,source_map,source_map_runtime,archive,solc_json,method_identifiers,userdoc,devdoc,metadata,combined_json,layout yourFileName.vy
 
 .. note::
     The ``opcodes`` and ``opcodes_runtime`` output of the compiler has been returning incorrect opcodes since ``0.2.0`` due to a lack of 0 padding (patched via `PR 3735 <https://github.com/vyperlang/vyper/pull/3735>`_). If you rely on these functions for debugging, please use the latest patched versions.
@@ -95,7 +97,6 @@ Importing Interfaces
 
 1. Interfaces defined in the ``interfaces`` field of the input JSON.
 2. Derived interfaces generated from contracts in the ``sources`` field of the input JSON.
-3. (Optional) The local filesystem, if a root path was explicitly declared via the ``-p`` flag.
 
 See :ref:`searching_for_imports` for more information on Vyper's import system.
 
@@ -121,7 +122,7 @@ Remix IDE
 Compiler Optimization Modes
 ===========================
 
-The vyper CLI tool accepts an optimization mode ``"none"``, ``"codesize"``, or ``"gas"`` (default). It can be set using the ``--optimize`` flag. For example, invoking ``vyper --optimize codesize MyContract.vy`` will compile the contract, optimizing for code size. As a rough summary of the differences between gas and codesize mode, in gas optimized mode, the compiler will try to generate bytecode which minimizes gas (up to a point), including:
+The Vyper CLI tool accepts an optimization mode ``"none"``, ``"codesize"``, or ``"gas"`` (default). It can be set using the ``--optimize`` flag. For example, invoking ``vyper --optimize codesize MyContract.vy`` will compile the contract, optimizing for code size. As a rough summary of the differences between gas and codesize mode, in gas optimized mode, the compiler will try to generate bytecode which minimizes gas (up to a point), including:
 
 * using a sparse selector table which optimizes for gas over codesize
 * inlining some constants, and
@@ -192,11 +193,50 @@ The following is a list of supported EVM versions, and changes in the compiler i
    - Functions marked with ``@nonreentrant`` are protected with TLOAD/TSTORE instead of SLOAD/SSTORE
    - The ``MCOPY`` opcode will be generated automatically by the compiler for most memory operations.
 
+.. _integrity-hash:
+
+Integrity Hash
+==============
+
+To help tooling detect whether two builds are the same, Vyper provides the ``-f integrity`` output, which outputs the integrity hash of a contract. The integrity hash is recursively defined as the sha256 of the source code with the integrity hashes of its dependencies (imports).
+
+.. _vyper-archives:
+
+Vyper Archives
+==============
+
+A Vyper archive is a compileable bundle of input sources and settings. Technically, it is a `ZIP file <https://en.wikipedia.org/wiki/ZIP_(file_format)>`_, with a special structure to make it useable as input to the compiler. It can use any suffix, but the convention is to use a ``.zip`` suffix or ``.vyz`` suffix. It must contain a ``MANIFEST/`` folder, with the following directory structure.
+
+::
+
+    MANIFEST
+    ├── cli_settings.txt
+    ├── compilation_targets
+    ├── compiler_version
+    ├── integrity
+    ├── searchpaths
+    └── settings.json
+
+* ``cli_settings.txt`` is a text representation of the settings that were used on the compilation run that generated this archive.
+* ``compilation_targets`` is a newline separated list of compilation targets. Currently only one compilation is supported
+* ``compiler_version`` is a text representation of the compiler version used to generate this archive
+* ``integrity`` is the :ref:`integrity hash <integrity-hash>` of the input contract
+* ``searchpaths`` is a newline-separated list of the search paths used on this compilation run
+* ``settings.json`` is a json representation of the settings used on this compilation run. It is 1:1 with ``cli_settings.txt``, but both are provided as they are convenient for different workflows (typically, manually vs automated).
+
+A Vyper archive file can be produced by requesting the ``-f archive`` output format. The compiler can also produce the archive in base64 encoded form using the ``--base64`` flag. The Vyper compiler can accept both ``.vyz`` and base64-encoded Vyper archives directly as input.
+
+.. code-block:: bash
+
+    $ vyper -f archive my_contract.vy -o my_contract.vyz  # write the archive to my_contract.vyz
+    $ vyper -f archive my_contract.vy --base64 > my_contract.vyz.b64  # write the archive, as base64-encoded text
+    $ vyper my_contract.vyz  # compile my_contract.vyz
+    $ vyper my_contract.vyz.b64  # compile my_contract.vyz.b64
 
 Compiler Input and Output JSON Description
 ==========================================
 
-Especially when dealing with complex or automated setups, the recommended way to compile is to use :ref:`vyper-json` and the JSON-input-output interface.
+JSON input/output is provided for compatibility with solidity, however, the recommended way is to use the aforementioned :ref:`Vyper archives <vyper-archives>`. So-called "standard json" input can be generated from a contract using the ``vyper -f solc_json`` output format.
 
 Where possible, the Vyper JSON compiler formats follow those of `Solidity <https://solidity.readthedocs.io/en/latest/using-the-compiler.html#compiler-input-and-output-json-description>`_.
 
@@ -205,7 +245,7 @@ Where possible, the Vyper JSON compiler formats follow those of `Solidity <https
 Input JSON Description
 ----------------------
 
-The following example describes the expected input format of ``vyper-json``. Comments are of course not permitted and used here *only for explanatory purposes*.
+The following example describes the expected input format of ``vyper-json``. (Comments are not normally permitted in JSON and are used here for explanatory purposes).
 
 .. code-block:: json
 
@@ -223,10 +263,10 @@ The following example describes the expected input format of ``vyper-json``. Com
             }
         },
         // Optional
-        // Interfaces given here are made available for import by the sources
+        // Sources given here are made available for import by the contracts
         // that are compiled. If the suffix is ".vy", the compiler will expect
-        // a contract-as-interface using proper Vyper syntax. If the suffix is
-        // "abi" the compiler will expect an ABI object.
+        // Vyper syntax. If the suffix is "abi" the compiler will expect an
+        // ABI object.
         "interfaces": {
             "contracts/bar.vy": {
                 "content": ""
@@ -245,6 +285,11 @@ The following example describes the expected input format of ``vyper-json``. Com
             // optional, whether or not the bytecode should include Vyper's signature
             // defaults to true
             "bytecodeMetadata": true,
+            // optional, whether to use the experimental venom pipeline
+            // defaults to false
+            "experimentalCodegen": false,
+            // the search paths to use for resolving imports
+            "search_paths": [],
             // The following is used to select desired outputs based on file names.
             // File names are given as keys, a star as a file name matches all files.
             // Outputs can also follow the Solidity format where second level keys
@@ -263,10 +308,10 @@ The following example describes the expected input format of ``vyper-json``. Com
             //    devdoc - Natspec developer documentation
             //    evm.bytecode.object - Bytecode object
             //    evm.bytecode.opcodes - Opcodes list
+            //    evm.bytecode.sourceMap - Source mapping (useful for debugging)
             //    evm.deployedBytecode.object - Deployed bytecode object
             //    evm.deployedBytecode.opcodes - Deployed opcodes list
-            //    evm.deployedBytecode.sourceMap - Solidity-style source mapping
-            //    evm.deployedBytecode.sourceMapFull - Deployed source mapping (useful for debugging)
+            //    evm.deployedBytecode.sourceMap - Deployed source mapping (useful for debugging)
             //    evm.methodIdentifiers - The list of function hashes
             //
             // Using `evm`, `evm.bytecode`, etc. will select every target part of that output.
@@ -343,15 +388,37 @@ The following example describes the output format of ``vyper-json``. Comments ar
                             // The bytecode as a hex string.
                             "object": "00fe",
                             // Opcodes list (string)
-                            "opcodes": ""
+                            "opcodes": "",
+                            // The deployed source mapping.
+                            "sourceMap": {
+                                "breakpoints": [],
+                                "error_map": {},
+                                "pc_ast_map": {},
+                                "pc_ast_map_item_keys": [],
+                                "pc_breakpoints": [],
+                                "pc_jump_map": {},
+                                "pc_pos_map": {},
+                                // The deployed source mapping as a string.
+                                "pc_pos_map_compressed": ""
+                            }
                         },
                         "deployedBytecode": {
                             // The deployed bytecode as a hex string.
                             "object": "00fe",
                             // Deployed opcodes list (string)
                             "opcodes": "",
-                            // The deployed source mapping as a string.
-                            "sourceMap": ""
+                            // The deployed source mapping.
+                            "sourceMap": {
+                                "breakpoints": [],
+                                "error_map": {},
+                                "pc_ast_map": {},
+                                "pc_ast_map_item_keys": [],
+                                "pc_breakpoints": [],
+                                "pc_jump_map": {},
+                                "pc_pos_map": {},
+                                // The deployed source mapping as a string.
+                                "pc_pos_map_compressed": ""
+                            }
                         },
                         // The list of function hashes
                         "methodIdentifiers": {