Running
Configure
@@ -524,7 +526,7 @@ Linux
The basic tooling is provided as part of the core operating system,
but you will most likely need to install developer packages.
For apt-based distributions (Debian, Ubuntu, etc), try this:
-sudo apt-get install build-essentialsudo apt-get install build-essential autoconfFor rpm-based distributions (Fedora, Red Hat, etc), try this:
sudo yum groupinstall "Development Tools"For Alpine Linux, aside from basic tooling, install the GNU versions
@@ -862,6 +864,13 @@
GNU Bash
href="https://www.gnu.org/software/bash">GNU Bash. No other shells
are supported.
At least version 3.2 of GNU Bash must be used.
+Graphviz and Pandoc
+In order to build the full docs (see the
+--enable-full-docs configure option) Graphviz and Pandoc are required. Any recent versions
+should work. For reference, and subject to change, Oracle builds use
+Graphviz 9.0.0 and Pandoc 2.19.2.
To build the JDK, you need a "configuration", which consists of a
directory where to store the build output, coupled with information
@@ -2157,15 +2166,26 @@
Using Multiple
configure from there, e.g.
mkdir build/<name> && cd build/<name> && bash ../../configure.
Then you can build that configuration using
-make CONF_NAME=<name> or
-make CONF=<pattern>, where
-<pattern> is a substring matching one or several
-configurations, e.g. CONF=debug. The special empty pattern
-(CONF=) will match all available configuration, so
-make CONF= hotspot will build the hotspot
-target for all configurations. Alternatively, you can execute
-make in the configuration directory, e.g.
-cd build/<name> && make.
+make CONF=<selector>, where
+<selector> is interpreted as follows:
+
+If <selector> exacly matches the name of a
+configuration, this and only this configuration will be selected.
+If <selector> matches (i.e. is a substring of)
+the names of several configurations, then all these configurations will
+be selected.
+If <selector> is empty (i.e. CONF=),
+then all configurations will be selected.
+If <selector> begins with !, then
+all configurations not matching the string following
+! will be selected.
+
+A more specialized version, CONF_NAME=<name> also
+exists, which will only match if the given <name>
+exactly matches a single configuration.
+Alternatively, you can execute make in the configuration
+directory, e.g. cd build/<name> && make.
+make CONF_NAME=<name> or
Handling Reconfigurations
If you update the repository and part of the configure script has
changed, the build system will force you to re-run
diff --git a/doc/building.md b/doc/building.md
index de410439446..ed8a0669355 100644
--- a/doc/building.md
+++ b/doc/building.md
@@ -349,7 +349,7 @@ will most likely need to install developer packages.
For apt-based distributions (Debian, Ubuntu, etc), try this:
```
-sudo apt-get install build-essential
+sudo apt-get install build-essential autoconf
```
For rpm-based distributions (Fedora, Red Hat, etc), try this:
@@ -685,6 +685,14 @@ shells are supported.
At least version 3.2 of GNU Bash must be used.
+### Graphviz and Pandoc
+
+In order to build the full docs (see the `--enable-full-docs`
+configure option) [Graphviz](https://www.graphviz.org) and
+[Pandoc](https://pandoc.org) are required. Any recent versions should
+work. For reference, and subject to change, Oracle builds use Graphviz
+9.0.0 and Pandoc 2.19.2.
+
## Running Configure
To build the JDK, you need a "configuration", which consists of a directory
@@ -1944,12 +1952,25 @@ configuration with the name ``. Alternatively, you can create a directory
under `build` and run `configure` from there, e.g. `mkdir build/ && cd
build/ && bash ../../configure`.
-Then you can build that configuration using `make CONF_NAME=` or `make
-CONF=`, where `` is a substring matching one or several
-configurations, e.g. `CONF=debug`. The special empty pattern (`CONF=`) will
-match *all* available configuration, so `make CONF= hotspot` will build the
-`hotspot` target for all configurations. Alternatively, you can execute `make`
-in the configuration directory, e.g. `cd build/ && make`.
+Then you can build that configuration using `make CONF=`, where
+`` is interpreted as follows:
+
+* If `` exacly matches the name of a configuration, this and only
+ this configuration will be selected.
+* If `` matches (i.e. is a substring of) the names of several
+ configurations, then all these configurations will be selected.
+* If `` is empty (i.e. `CONF=`), then all configurations will be
+ selected.
+* If `` begins with `!`, then all configurations **not** matching the
+ string following `!` will be selected.
+
+A more specialized version, `CONF_NAME=` also exists, which will only
+match if the given `` exactly matches a single configuration.
+
+Alternatively, you can execute `make` in the configuration directory, e.g. `cd
+build/ && make`.
+
+`make CONF_NAME=` or
### Handling Reconfigurations
diff --git a/make/Docs.gmk b/make/Docs.gmk
index 5f9ec99fcae..d0c01f0283d 100644
--- a/make/Docs.gmk
+++ b/make/Docs.gmk
@@ -139,11 +139,6 @@ ifeq ($(IS_DRAFT), true)
endif
DRAFT_TEXT := This specification is not final and is subject to change. \
Use is subject to license terms .
-
- # Workaround stylesheet bug
- HEADER_STYLE := style="margin-top: 9px;"
-else
- HEADER_STYLE := style="margin-top: 14px;"
endif
# $1 - Relative prefix to COPYRIGHT_URL
@@ -339,7 +334,7 @@ define SetupApiDocsGenerationBody
$1_DOC_TITLE := $$($1_LONG_NAME)$$($1_SHORT_NAME) \
+ $1_HEADER_TITLE :=
$$($1_SHORT_NAME) \
$$(DRAFT_MARKER_STR)
ifneq ($$($1_OTHER_VERSIONS), )
$1_JAVADOC_BOTTOM := $$(call JAVADOC_BOTTOM,
Other versions. )
@@ -647,7 +642,7 @@ ifeq ($(ENABLE_PANDOC), true)
GLOBAL_SPECS_DEFAULT_CSS_FILE := $(DOCS_OUTPUTDIR)/resources/jdk-default.css
# Unset the following to suppress the link to the tool guides
NAV_LINK_GUIDES := --nav-link-guides
- HEADER_RIGHT_SIDE_INFO :=
$(subst &,&,$(JDK_SHORT_NAME))$(DRAFT_MARKER_STR)
+ HEADER_RIGHT_SIDE_INFO :=
$(subst &,&,$(JDK_SHORT_NAME)) $(DRAFT_MARKER_STR)
$(foreach m, $(ALL_MODULES), \
$(eval SPECS_$m := $(call FindModuleSpecsDirs, $m)) \
diff --git a/make/Global.gmk b/make/Global.gmk
index e5e76b475b9..1df6c5fb6bc 100644
--- a/make/Global.gmk
+++ b/make/Global.gmk
@@ -87,10 +87,9 @@ help:
$(info $(_) # (gensrc, java, copy, libs, launchers, gendata))
$(info )
$(info Make control variables)
- $(info $(_) CONF= # Build all configurations (note, assignment is empty))
- $(info $(_) CONF=
# Build the configuration(s) with a name matching)
- $(info $(_) # )
- $(info $(_) CONF_NAME= # Build the configuration with exactly the )
+ $(info $(_) CONF= # Select which configuration(s) to build)
+ $(info $(_) CONF= # Select all configurations (note, assignment is empty))
+ $(info $(_) CONF_NAME= # Select the configuration with the name )
$(info $(_) SPEC= # Build the configuration given by the spec file)
$(info $(_) LOG= # Change the log level from warn to )
$(info $(_) # Available log levels are:)
diff --git a/make/InitSupport.gmk b/make/InitSupport.gmk
index 31c80e2f726..9ea01d375ce 100644
--- a/make/InitSupport.gmk
+++ b/make/InitSupport.gmk
@@ -202,8 +202,14 @@ ifeq ($(HAS_SPEC),)
matching_confs := $$(strip $$(all_confs))
else
# Otherwise select those that contain the given CONF string
- matching_confs := $$(strip $$(foreach var, $$(all_confs), \
- $$(if $$(findstring $$(CONF), $$(var)), $$(var))))
+ ifeq ($$(patsubst !%,,$$(CONF)),)
+ # A CONF starting with ! means we should negate the search term
+ matching_confs := $$(strip $$(foreach var, $$(all_confs), \
+ $$(if $$(findstring $$(subst !,,$$(CONF)), $$(var)), ,$$(var))))
+ else
+ matching_confs := $$(strip $$(foreach var, $$(all_confs), \
+ $$(if $$(findstring $$(CONF), $$(var)), $$(var))))
+ endif
ifneq ($$(filter $$(CONF), $$(matching_confs)), )
# If we found an exact match, use that
matching_confs := $$(CONF)
diff --git a/make/Main.gmk b/make/Main.gmk
index 5534a68f13b..b0b7565c138 100644
--- a/make/Main.gmk
+++ b/make/Main.gmk
@@ -969,20 +969,28 @@ else
jdk.jdeps-gendata: java
- # The ct.sym generation uses all the moduleinfos as input
- jdk.compiler-gendata: $(GENSRC_MODULEINFO_TARGETS) $(JAVA_TARGETS)
- # jdk.compiler-gendata needs the BUILD_JDK. If the BUILD_JDK was supplied
- # externally, no extra prerequisites are needed.
+ # jdk.compiler gendata generates ct.sym, which requires all generated
+ # java source and compiled classes present.
+ jdk.compiler-gendata: $(JAVA_TARGETS)
+
+ # jdk.javadoc gendata generates element-list, which requires all java sources
+ # but not compiled classes.
+ jdk.javadoc-gendata: $(GENSRC_TARGETS)
+
+ # ct.sym and element-list generation also needs the BUILD_JDK. If the
+ # BUILD_JDK was supplied externally, no extra prerequisites are needed.
ifeq ($(CREATE_BUILDJDK), true)
ifneq ($(CREATING_BUILDJDK), true)
# When cross compiling and an external BUILD_JDK wasn't supplied, it's
# produced by the create-buildjdk target.
jdk.compiler-gendata: create-buildjdk
+ jdk.javadoc-gendata: create-buildjdk
endif
else ifeq ($(EXTERNAL_BUILDJDK), false)
# When not cross compiling, the BUILD_JDK is the interim jdk image, and
# the javac launcher is needed.
jdk.compiler-gendata: jdk.compiler-launchers
+ jdk.javadoc-gendata: jdk.compiler-launchers
endif
# Declare dependencies between jmod targets.
diff --git a/make/autoconf/basic.m4 b/make/autoconf/basic.m4
index 5a1fbc07952..91d7a550108 100644
--- a/make/autoconf/basic.m4
+++ b/make/autoconf/basic.m4
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -123,6 +123,22 @@ AC_DEFUN_ONCE([BASIC_SETUP_BUILD_ENV],
]
)
AC_SUBST(BUILD_ENV)
+
+ if test "x$LOCALE" != x; then
+ # Check if we actually have C.UTF-8; if so, use it
+ if $LOCALE -a | $GREP -q -E "^C\.(utf8|UTF-8)$"; then
+ LOCALE_USED=C.UTF-8
+ else
+ AC_MSG_WARN([C.UTF-8 locale not found, using C locale])
+ LOCALE_USED=C
+ fi
+ else
+ AC_MSG_WARN([locale command not not found, using C locale])
+ LOCALE_USED=C
+ fi
+
+ export LC_ALL=$LOCALE_USED
+ AC_SUBST(LOCALE_USED)
])
###############################################################################
diff --git a/make/autoconf/basic_tools.m4 b/make/autoconf/basic_tools.m4
index f9ecbab7ac0..0c084561f2e 100644
--- a/make/autoconf/basic_tools.m4
+++ b/make/autoconf/basic_tools.m4
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -54,6 +54,7 @@ AC_DEFUN_ONCE([BASIC_SETUP_FUNDAMENTAL_TOOLS],
UTIL_REQUIRE_SPECIAL(SED, [AC_PROG_SED])
# Tools only needed on some platforms
+ UTIL_LOOKUP_PROGS(LOCALE, locale)
UTIL_LOOKUP_PROGS(PATHTOOL, cygpath wslpath)
UTIL_LOOKUP_PROGS(CMD, cmd.exe, $PATH:/cygdrive/c/windows/system32:/mnt/c/windows/system32:/c/windows/system32)
])
diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4
index df44f7a2d59..e95f32f4f7d 100644
--- a/make/autoconf/flags-cflags.m4
+++ b/make/autoconf/flags-cflags.m4
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -117,6 +117,11 @@ AC_DEFUN([FLAGS_SETUP_DEBUG_SYMBOLS],
FLAGS_COMPILER_CHECK_ARGUMENTS(ARGUMENT: [${DEBUG_PREFIX_CFLAGS}],
IF_FALSE: [
DEBUG_PREFIX_CFLAGS=
+ ],
+ IF_TRUE: [
+ # Add debug prefix map gcc system include paths, as they cause
+ # non-deterministic debug paths depending on gcc path location.
+ DEBUG_PREFIX_MAP_GCC_INCLUDE_PATHS
]
)
fi
@@ -158,6 +163,55 @@ AC_DEFUN([FLAGS_SETUP_DEBUG_SYMBOLS],
AC_SUBST(ASFLAGS_DEBUG_SYMBOLS)
])
+# gcc will embed the full system include paths in the debug info
+# resulting in non-deterministic debug symbol files and thus
+# non-reproducible native libraries if gcc includes are located
+# in different paths.
+# Add -fdebug-prefix-map'ings for root and gcc include paths,
+# pointing to a common set of folders so that the binaries are deterministic:
+# root include : /usr/include
+# gcc include : /usr/local/gcc_include
+# g++ include : /usr/local/gxx_include
+AC_DEFUN([DEBUG_PREFIX_MAP_GCC_INCLUDE_PATHS],
+[
+ # Determine gcc system include paths.
+ # Assume default roots to start with:
+ GCC_ROOT_INCLUDE="/usr/include"
+
+ # Determine is sysroot or devkit specified?
+ if test "x$SYSROOT" != "x"; then
+ GCC_ROOT_INCLUDE="${SYSROOT%/}/usr/include"
+ fi
+
+ # Add root include mapping => /usr/include
+ GCC_INCLUDE_DEBUG_MAP_FLAGS="-fdebug-prefix-map=${GCC_ROOT_INCLUDE}/=/usr/include/"
+
+ # Add gcc system include mapping => /usr/local/gcc_include
+ # Find location of stddef.h using build C compiler
+ GCC_SYSTEM_INCLUDE=`$ECHO "#include " | \
+ $CC $CFLAGS -v -E - 2>&1 | \
+ $GREP stddef | $TAIL -1 | $TR -s " " | $CUT -d'"' -f2`
+ if test "x$GCC_SYSTEM_INCLUDE" != "x"; then
+ GCC_SYSTEM_INCLUDE=`$DIRNAME $GCC_SYSTEM_INCLUDE`
+ GCC_INCLUDE_DEBUG_MAP_FLAGS="$GCC_INCLUDE_DEBUG_MAP_FLAGS \
+ -fdebug-prefix-map=${GCC_SYSTEM_INCLUDE}/=/usr/local/gcc_include/"
+ fi
+
+ # Add g++ system include mapping => /usr/local/gxx_include
+ # Find location of cstddef using build C++ compiler
+ GXX_SYSTEM_INCLUDE=`$ECHO "#include " | \
+ $CXX $CXXFLAGS -v -E -x c++ - 2>&1 | \
+ $GREP cstddef | $TAIL -1 | $TR -s " " | $CUT -d'"' -f2`
+ if test "x$GXX_SYSTEM_INCLUDE" != "x"; then
+ GXX_SYSTEM_INCLUDE=`$DIRNAME $GXX_SYSTEM_INCLUDE`
+ GCC_INCLUDE_DEBUG_MAP_FLAGS="$GCC_INCLUDE_DEBUG_MAP_FLAGS \
+ -fdebug-prefix-map=${GXX_SYSTEM_INCLUDE}/=/usr/local/gxx_include/"
+ fi
+
+ # Add to debug prefix cflags
+ DEBUG_PREFIX_CFLAGS="$DEBUG_PREFIX_CFLAGS $GCC_INCLUDE_DEBUG_MAP_FLAGS"
+])
+
AC_DEFUN([FLAGS_SETUP_WARNINGS],
[
# Set default value.
@@ -425,13 +479,14 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
[
#### OS DEFINES, these should be independent on toolchain
if test "x$OPENJDK_TARGET_OS" = xlinux; then
- CFLAGS_OS_DEF_JVM="-DLINUX"
- CFLAGS_OS_DEF_JDK="-D_GNU_SOURCE -D_REENTRANT -D_LARGEFILE64_SOURCE"
+ CFLAGS_OS_DEF_JVM="-DLINUX -D_FILE_OFFSET_BITS=64"
+ CFLAGS_OS_DEF_JDK="-D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64"
elif test "x$OPENJDK_TARGET_OS" = xmacosx; then
CFLAGS_OS_DEF_JVM="-D_ALLBSD_SOURCE -D_DARWIN_C_SOURCE -D_XOPEN_SOURCE"
CFLAGS_OS_DEF_JDK="-D_ALLBSD_SOURCE -D_DARWIN_UNLIMITED_SELECT"
elif test "x$OPENJDK_TARGET_OS" = xaix; then
- CFLAGS_OS_DEF_JVM="-DAIX"
+ CFLAGS_OS_DEF_JVM="-DAIX -D_LARGE_FILES"
+ CFLAGS_OS_DEF_JDK="-D_LARGE_FILES"
elif test "x$OPENJDK_TARGET_OS" = xbsd; then
CFLAGS_OS_DEF_JDK="-D_ALLBSD_SOURCE"
elif test "x$OPENJDK_TARGET_OS" = xwindows; then
@@ -489,7 +544,7 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER],
ALWAYS_DEFINES_JVM="-D_GNU_SOURCE"
elif test "x$TOOLCHAIN_TYPE" = xxlc; then
ALWAYS_DEFINES_JVM="-D_REENTRANT"
- ALWAYS_DEFINES_JDK="-D_GNU_SOURCE -D_REENTRANT -D_LARGEFILE64_SOURCE -DSTDC"
+ ALWAYS_DEFINES_JDK="-D_GNU_SOURCE -D_REENTRANT -DSTDC"
elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then
# Access APIs for Windows 8 and above
# see https://docs.microsoft.com/en-us/cpp/porting/modifying-winver-and-win32-winnt?view=msvc-170
diff --git a/make/autoconf/spec.gmk.template b/make/autoconf/spec.gmk.template
index fc9fbb3b912..325e8a0eaca 100644
--- a/make/autoconf/spec.gmk.template
+++ b/make/autoconf/spec.gmk.template
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -56,8 +56,8 @@ COMMA := ,
# What make to use for main processing, after bootstrapping top-level Makefile.
MAKE := @MAKE@
-# Make sure all shell commands are executed with the C locale
-export LC_ALL := C
+# Make sure all shell commands are executed with a proper locale
+export LC_ALL := @LOCALE_USED@
# Make sure we override any local CLASSPATH variable
export CLASSPATH := @CLASSPATH@
diff --git a/make/conf/github-actions.conf b/make/conf/github-actions.conf
index 05c6db9b032..77daeb8c6e1 100644
--- a/make/conf/github-actions.conf
+++ b/make/conf/github-actions.conf
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -29,13 +29,17 @@ GTEST_VERSION=1.14.0
JTREG_VERSION=7.3.1+1
LINUX_X64_BOOT_JDK_EXT=tar.gz
-LINUX_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.1/sapmachine-jdk-21.0.1_linux-x64_bin.tar.gz
-LINUX_X64_BOOT_JDK_SHA256=450426abc41695696c49c02df437882e6763344522c70a46ae6b0b682ba370ff
+LINUX_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.2/sapmachine-jdk-21.0.2_linux-x64_bin.tar.gz
+LINUX_X64_BOOT_JDK_SHA256=3123189ec5b99eed78de0328e2fd49d7c13cc7d4524c341f1fe8fbd5165be31f
MACOS_X64_BOOT_JDK_EXT=tar.gz
-MACOS_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.1/sapmachine-jdk-21.0.1_macos-x64_bin.tar.gz
-MACOS_X64_BOOT_JDK_SHA256=498bdb5635be4b784ccfe12f83dc4c83f6a5fb2f37666a4fec408a2fd4083afa
+MACOS_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.2/sapmachine-jdk-21.0.2_macos-x64_bin.tar.gz
+MACOS_X64_BOOT_JDK_SHA256=826fb6c8c5b4c24a1150ad3e393114a8be9072b9f9a65bc10ec8343971eb48fc
+
+MACOS_AARCH64_BOOT_JDK_EXT=tar.gz
+MACOS_AARCH64_BOOT_JDK_URL=https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_macos-aarch64_bin.tar.gz
+MACOS_AARCH64_BOOT_JDK_SHA256=f12e1e0a2dffc847951598f597c8ee60fb0913932f24b2b09c62cfd2f0f4dfb9
WINDOWS_X64_BOOT_JDK_EXT=zip
-WINDOWS_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.1/sapmachine-jdk-21.0.1_windows-x64_bin.zip
-WINDOWS_X64_BOOT_JDK_SHA256=d200fc7b4e5c13293caf992aef3207dd627917353c3f189901e4d95fe23bdea5
+WINDOWS_X64_BOOT_JDK_URL=https://github.com/SAP/SapMachine/releases/download/sapmachine-21.0.2/sapmachine-jdk-21.0.2_windows-x64_bin.zip
+WINDOWS_X64_BOOT_JDK_SHA256=8b2bc8007381240728ca50a23f020a3603ca84314308df707d4b3eff5b3bcfd5
diff --git a/make/conf/jib-profiles.js b/make/conf/jib-profiles.js
index d8bca355121..af164624877 100644
--- a/make/conf/jib-profiles.js
+++ b/make/conf/jib-profiles.js
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -441,7 +441,7 @@ var getJibProfilesProfiles = function (input, common, data) {
"macosx-x64": {
target_os: "macosx",
target_cpu: "x64",
- dependencies: ["devkit", "gtest", "pandoc"],
+ dependencies: ["devkit", "gtest", "graphviz", "pandoc"],
configure_args: concat(common.configure_args_64bit, "--with-zlib=system",
"--with-macosx-version-max=11.00.00",
"--enable-compatible-cds-alignment",
@@ -453,7 +453,7 @@ var getJibProfilesProfiles = function (input, common, data) {
"macosx-aarch64": {
target_os: "macosx",
target_cpu: "aarch64",
- dependencies: ["devkit", "gtest", "pandoc"],
+ dependencies: ["devkit", "gtest", "graphviz", "pandoc"],
configure_args: concat(common.configure_args_64bit,
"--with-macosx-version-max=11.00.00"),
},
@@ -486,7 +486,7 @@ var getJibProfilesProfiles = function (input, common, data) {
"linux-aarch64": {
target_os: "linux",
target_cpu: "aarch64",
- dependencies: ["devkit", "gtest", "build_devkit", "pandoc"],
+ dependencies: ["devkit", "gtest", "build_devkit", "graphviz", "pandoc"],
configure_args: [
"--with-zlib=system",
"--disable-dtrace",
@@ -1181,12 +1181,6 @@ var getJibProfilesDependencies = function (input, common) {
revision: (input.build_cpu == "x64" ? "Xcode11.3.1-MacOSX10.15+1.2" : devkit_platform_revisions[devkit_platform])
},
- cups: {
- organization: common.organization,
- ext: "tar.gz",
- revision: "1.0118+1.0"
- },
-
jtreg: {
server: "jpg",
product: "jtreg",
@@ -1237,7 +1231,7 @@ var getJibProfilesDependencies = function (input, common) {
graphviz: {
organization: common.organization,
ext: "tar.gz",
- revision: "2.38.0-1+1.1",
+ revision: "9.0.0+1.0",
module: "graphviz-" + input.target_platform,
configure_args: "DOT=" + input.get("graphviz", "install_path") + "/dot",
environment_path: input.get("graphviz", "install_path")
diff --git a/make/devkit/createGraphvizBundle.sh b/make/devkit/createGraphvizBundle.sh
index 290e68c382c..1b890838761 100644
--- a/make/devkit/createGraphvizBundle.sh
+++ b/make/devkit/createGraphvizBundle.sh
@@ -1,6 +1,6 @@
-#!/bin/bash -e
+#!/bin/bash
#
-# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -26,38 +26,106 @@
# Create a bundle in the current directory, containing what's needed to run
# the 'dot' program from the graphviz suite by the OpenJDK build.
-TMPDIR=`mktemp -d -t graphvizbundle-XXXX`
-trap "rm -rf \"$TMPDIR\"" EXIT
-
-ORIG_DIR=`pwd`
-cd "$TMPDIR"
-GRAPHVIZ_VERSION=2.38.0-1
-PACKAGE_VERSION=1.1
-TARGET_PLATFORM=linux_x64
-BUNDLE_NAME=graphviz-$TARGET_PLATFORM-$GRAPHVIZ_VERSION+$PACKAGE_VERSION.tar.gz
-wget http://www.graphviz.org/pub/graphviz/stable/redhat/el6/x86_64/os/graphviz-$GRAPHVIZ_VERSION.el6.x86_64.rpm
-wget http://www.graphviz.org/pub/graphviz/stable/redhat/el6/x86_64/os/graphviz-libs-$GRAPHVIZ_VERSION.el6.x86_64.rpm
-wget http://www.graphviz.org/pub/graphviz/stable/redhat/el6/x86_64/os/graphviz-plugins-core-$GRAPHVIZ_VERSION.el6.x86_64.rpm
-wget http://www.graphviz.org/pub/graphviz/stable/redhat/el6/x86_64/os/graphviz-plugins-x-$GRAPHVIZ_VERSION.el6.x86_64.rpm
-wget http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/libtool-ltdl-2.2.6-15.5.el6.x86_64.rpm
-
-mkdir graphviz
-cd graphviz
-for rpm in ../*.rpm; do
- rpm2cpio $rpm | cpio --extract --make-directories
-done
-
-cat > dot << EOF
+set -eux
+
+mydir="$(cd -- $(dirname ${BASH_SOURCE[0]}) && pwd)"
+me="${mydir}/$(basename ${BASH_SOURCE[0]})"
+
+EXPAT_VERSION="2.6.0"
+EXPAT_URL="https://github.com/libexpat/libexpat/releases/download/R_${EXPAT_VERSION//./_}/expat-${EXPAT_VERSION}.tar.gz"
+EXPAT_SHA256="a13447b9aa67d7c860783fdf6820f33ebdea996900d6d8bbc50a628f55f099f7"
+
+GRAPHVIZ_VERSION="9.0.0"
+GRAPHVIZ_URL="https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/${GRAPHVIZ_VERSION}/graphviz-${GRAPHVIZ_VERSION}.tar.xz"
+GRAPHVIZ_SHA256="6c9afda06a732af7658c2619ee713d2545818c3ff19b7b8fd48effcd06d57bf6"
+
+uname_s="$(uname -s)"
+case ${uname_s} in
+ Linux)
+ bundle_os="linux"
+ shacmd="sha256sum --strict --check -"
+ lib_path_var="LD_LIBRARY_PATH"
+ ;;
+ Darwin)
+ bundle_os="macosx"
+ shacmd="shasum -a 256 --strict --check -"
+ lib_path_var="DYLD_LIBRARY_PATH"
+ ;;
+ *)
+ echo "Unknown OS: ${uname_s}"
+ exit 1
+ ;;
+esac
+uname_m="$(uname -m)"
+case ${uname_m} in
+ aarch64|arm64)
+ bundle_cpu="aarch64"
+ ;;
+ x86_64)
+ bundle_cpu="x64"
+ ;;
+esac
+bundle_platform="${bundle_os}_${bundle_cpu}"
+
+build_dir="${mydir}/../../build/graphviz"
+download_dir="${build_dir}/download"
+install_dir="${build_dir}/result/graphviz-${bundle_platform}-${GRAPHVIZ_VERSION}"
+bundle_file="${install_dir}.tar.gz"
+
+expat_dir="${build_dir}/expat"
+expat_src_dir="${expat_dir}/src"
+
+graphviz_dir="${build_dir}/graphviz"
+graphviz_src_dir="${graphviz_dir}/src"
+graphviz_doc_dir="${install_dir}/doc"
+
+mkdir -p "${build_dir}"
+cd "${build_dir}"
+
+download_and_unpack() {
+ local url="$1"
+ local sha256="$2"
+ local file="$3"
+ local dir="$4"
+
+ mkdir -p "$(dirname "${file}")"
+ if [ ! -f "${file}" ]; then
+ curl -L -o "${file}" "${url}"
+ fi
+ echo "${sha256} ${file}" | ${shacmd}
+ if [ ! -d "${dir}" ]; then
+ mkdir -p "${dir}"
+ tar --extract --file "${file}" --directory "${dir}" --strip-components 1
+ fi
+}
+
+download_and_unpack "${EXPAT_URL}" "${EXPAT_SHA256}" "${download_dir}/expat.tar.gz" "${expat_src_dir}"
+download_and_unpack "${GRAPHVIZ_URL}" "${GRAPHVIZ_SHA256}" "${download_dir}/graphviz.tar.gz" "${graphviz_src_dir}"
+
+(
+ cd "${expat_src_dir}"
+ ./configure --prefix="${install_dir}"
+ make -j install
+)
+
+(
+ cd "${graphviz_src_dir}"
+ ./configure --prefix="${install_dir}" EXPAT_CFLAGS="-I${install_dir}/include" EXPAT_LIBS="-L${install_dir}/lib -lexpat"
+ make -j install
+)
+
+cat > "${install_dir}/dot" << EOF
#!/bin/bash
# Get an absolute path to this script
-this_script_dir=\`dirname \$0\`
-this_script_dir=\`cd \$this_script_dir > /dev/null && pwd\`
-export LD_LIBRARY_PATH="\$this_script_dir/usr/lib64:\$LD_LIBRARY_PATH"
-exec \$this_script_dir/usr/bin/dot "\$@"
+this_script_dir="\$(dirname \$0)"
+this_script_dir="\$(cd \${this_script_dir} > /dev/null && pwd)"
+export ${lib_path_var}="\${this_script_dir}/lib:\${this_script_dir}/lib/graphviz"
+exec "\${this_script_dir}/bin/dot" "\$@"
EOF
-chmod +x dot
-export LD_LIBRARY_PATH="$TMPDIR/graphviz/usr/lib64:$LD_LIBRARY_PATH"
+chmod +x "${install_dir}/dot"
# create config file
-./dot -c
-tar -cvzf ../$BUNDLE_NAME *
-cp ../$BUNDLE_NAME "$ORIG_DIR"
+"${install_dir}/dot" -c
+
+cp "${me}" "${install_dir}"
+
+tar --create --gzip --file "${bundle_file}" -C "${install_dir}" .
diff --git a/make/hotspot/gensrc/GensrcAdlc.gmk b/make/hotspot/gensrc/GensrcAdlc.gmk
index 0898d91e1c2..bb356476847 100644
--- a/make/hotspot/gensrc/GensrcAdlc.gmk
+++ b/make/hotspot/gensrc/GensrcAdlc.gmk
@@ -51,7 +51,7 @@ ifeq ($(call check-jvm-feature, compiler2), true)
endif
# Set the C++ standard
- ADLC_CFLAGS += $(ADLC_LANGSTD_CXXFLAG)
+ ADLC_CFLAGS += $(ADLC_LANGSTD_CXXFLAGS)
# NOTE: The old build didn't set -DASSERT for windows but it doesn't seem to
# hurt.
diff --git a/make/hotspot/lib/JvmOverrideFiles.gmk b/make/hotspot/lib/JvmOverrideFiles.gmk
index b50d6f8bb36..ffb98b9bb9a 100644
--- a/make/hotspot/lib/JvmOverrideFiles.gmk
+++ b/make/hotspot/lib/JvmOverrideFiles.gmk
@@ -48,9 +48,6 @@ ifneq ($(FDLIBM_CFLAGS), )
endif
ifeq ($(call isTargetOs, linux), true)
- BUILD_LIBJVM_ostream.cpp_CXXFLAGS := -D_FILE_OFFSET_BITS=64
- BUILD_LIBJVM_logFileOutput.cpp_CXXFLAGS := -D_FILE_OFFSET_BITS=64
-
BUILD_LIBJVM_sharedRuntimeTrig.cpp_CXXFLAGS := -DNO_PCH $(FDLIBM_CFLAGS) $(LIBJVM_FDLIBM_COPY_OPT_FLAG)
BUILD_LIBJVM_sharedRuntimeTrans.cpp_CXXFLAGS := -DNO_PCH $(FDLIBM_CFLAGS) $(LIBJVM_FDLIBM_COPY_OPT_FLAG)
diff --git a/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java b/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java
index d24aee5eee8..921eaeee764 100644
--- a/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java
+++ b/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -35,6 +35,7 @@
import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.net.InetAddress;
+import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystems;
import java.time.LocalDateTime;
import java.time.ZoneId;
@@ -132,6 +133,8 @@ public static void main(String ... args) throws Throwable {
String oldDate = String.format("%s%n",
DateFormat.getDateInstance(DateFormat.DEFAULT, Locale.ROOT)
.format(new Date()));
+ StandardCharsets.US_ASCII.encode("");
+ StandardCharsets.UTF_8.encode("");
// A selection of trivial and common reflection operations
var instance = HelloClasslist.class.getConstructor().newInstance();
diff --git a/make/modules/java.base/Java.gmk b/make/modules/java.base/Java.gmk
index 8621ff945cc..5820c280fe9 100644
--- a/make/modules/java.base/Java.gmk
+++ b/make/modules/java.base/Java.gmk
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -23,7 +23,11 @@
# questions.
#
-DISABLED_WARNINGS_java += this-escape restricted
+# The base module should be built with all warnings enabled. When a
+# new warning is added to javac, it can be temporarily added to the
+# disabled warnings list.
+#
+# DISABLED_WARNINGS_java +=
DOCLINT += -Xdoclint:all/protected \
'-Xdoclint/package:java.*,javax.*'
@@ -37,7 +41,8 @@ EXCLUDE_FILES += \
EXCLUDES += java/lang/doc-files \
java/lang/classfile/snippet-files \
- java/lang/classfile/components/snippet-files
+ java/lang/classfile/components/snippet-files \
+ java/lang/foreign/snippet-files
# Exclude BreakIterator classes that are just used in compile process to generate
# data files and shouldn't go in the product
diff --git a/make/modules/java.compiler/Java.gmk b/make/modules/java.compiler/Java.gmk
index e2d5ac264b8..04f31a9bc66 100644
--- a/make/modules/java.compiler/Java.gmk
+++ b/make/modules/java.compiler/Java.gmk
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -23,6 +23,10 @@
# questions.
#
+# To the extent technically possible, this module should be built with
+# -Werror and all lint warnings enabled. In particular,
+# DISABLED_WARNINGS_java should not be augmented.
+
DOCLINT += -Xdoclint:all/protected \
'-Xdoclint/package:java.*,javax.*'
diff --git a/make/modules/java.desktop/lib/Awt2dLibraries.gmk b/make/modules/java.desktop/lib/Awt2dLibraries.gmk
index 4d5ff751981..e274005e607 100644
--- a/make/modules/java.desktop/lib/Awt2dLibraries.gmk
+++ b/make/modules/java.desktop/lib/Awt2dLibraries.gmk
@@ -449,7 +449,6 @@ else
LIBFREETYPE_LIBS := -lfreetype
endif
- # gcc_ftobjs.c := maybe-uninitialized required for GCC 7 builds.
$(eval $(call SetupJdkLibrary, BUILD_LIBFREETYPE, \
NAME := freetype, \
OPTIMIZATION := HIGHEST, \
@@ -458,7 +457,6 @@ else
EXTRA_HEADER_DIRS := $(BUILD_LIBFREETYPE_HEADER_DIRS), \
DISABLED_WARNINGS_microsoft := 4267 4244 4996, \
DISABLED_WARNINGS_gcc := dangling-pointer stringop-overflow, \
- DISABLED_WARNINGS_gcc_ftobjs.c := maybe-uninitialized, \
LDFLAGS := $(LDFLAGS_JDKLIB) \
$(call SET_SHARED_LIBRARY_ORIGIN), \
))
diff --git a/make/modules/jdk.compiler/Java.gmk b/make/modules/jdk.compiler/Java.gmk
index 7e6793fc637..a2dd4f60fa6 100644
--- a/make/modules/jdk.compiler/Java.gmk
+++ b/make/modules/jdk.compiler/Java.gmk
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -23,6 +23,10 @@
# questions.
#
+# To the extent technically possible, this module should be built with
+# -Werror and all lint warnings enabled. In particular,
+# DISABLED_WARNINGS_java should not be augmented.
+
DOCLINT += -Xdoclint:all/protected \
'-Xdoclint/package:-com.sun.tools.*,-jdk.internal.*,sun.tools.serialver.resources.*'
JAVAC_FLAGS += -XDstringConcat=inline
diff --git a/make/modules/jdk.hotspot.agent/Lib.gmk b/make/modules/jdk.hotspot.agent/Lib.gmk
index d21c969c188..ebdbfecd461 100644
--- a/make/modules/jdk.hotspot.agent/Lib.gmk
+++ b/make/modules/jdk.hotspot.agent/Lib.gmk
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -27,10 +27,7 @@ include LibCommon.gmk
################################################################################
-ifeq ($(call isTargetOs, linux), true)
- SA_CFLAGS := -D_FILE_OFFSET_BITS=64
-
-else ifeq ($(call isTargetOs, macosx), true)
+ifeq ($(call isTargetOs, macosx), true)
SA_CFLAGS := -D_GNU_SOURCE -mno-omit-leaf-frame-pointer \
-mstack-alignment=16 -fPIC
LIBSA_EXTRA_SRC := $(SUPPORT_OUTPUTDIR)/gensrc/jdk.hotspot.agent
diff --git a/make/modules/jdk.javadoc/Java.gmk b/make/modules/jdk.javadoc/Java.gmk
index 3035864ecba..a8b1f2c1173 100644
--- a/make/modules/jdk.javadoc/Java.gmk
+++ b/make/modules/jdk.javadoc/Java.gmk
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@@ -23,6 +23,8 @@
# questions.
#
-DISABLED_WARNINGS_java += this-escape
+# To the extent technically possible, this module should be built with
+# -Werror and all lint warnings enabled. In particular,
+# DISABLED_WARNINGS_java should not be augmented.
COPY += .xml .css .svg .js .js.template .png .txt
diff --git a/make/test/BuildMicrobenchmark.gmk b/make/test/BuildMicrobenchmark.gmk
index 628f3855728..ba502a56128 100644
--- a/make/test/BuildMicrobenchmark.gmk
+++ b/make/test/BuildMicrobenchmark.gmk
@@ -107,6 +107,7 @@ $(eval $(call SetupJavaCompilation, BUILD_JDK_MICROBENCHMARK, \
--add-exports java.base/sun.invoke.util=ALL-UNNAMED \
--add-exports java.base/sun.security.util=ALL-UNNAMED \
--enable-preview \
+ -XDsuppressNotes \
-processor org.openjdk.jmh.generators.BenchmarkProcessor, \
JAVA_FLAGS := \
--add-exports java.base/jdk.internal.vm=ALL-UNNAMED \
diff --git a/src/hotspot/cpu/aarch64/aarch64.ad b/src/hotspot/cpu/aarch64/aarch64.ad
index f637648b227..501d9df08c1 100644
--- a/src/hotspot/cpu/aarch64/aarch64.ad
+++ b/src/hotspot/cpu/aarch64/aarch64.ad
@@ -1237,7 +1237,7 @@ source %{
// r27 is not allocatable when compressed oops is on and heapbase is not
// zero, compressed klass pointers doesn't use r27 after JDK-8234794
- if (UseCompressedOops && (CompressedOops::ptrs_base() != NULL)) {
+ if (UseCompressedOops && (CompressedOops::ptrs_base() != nullptr)) {
_NO_SPECIAL_REG32_mask.Remove(OptoReg::as_OptoReg(r27->as_VMReg()));
_NO_SPECIAL_REG_mask.Remove(OptoReg::as_OptoReg(r27->as_VMReg()));
_NO_SPECIAL_PTR_REG_mask.Remove(OptoReg::as_OptoReg(r27->as_VMReg()));
@@ -1581,7 +1581,7 @@ bool needs_releasing_store(const Node *n)
{
// assert n->is_Store();
StoreNode *st = n->as_Store();
- return st->trailing_membar() != NULL;
+ return st->trailing_membar() != nullptr;
}
// predicate controlling translation of CAS
@@ -1593,9 +1593,9 @@ bool needs_acquiring_load_exclusive(const Node *n)
assert(is_CAS(n->Opcode(), true), "expecting a compare and swap");
LoadStoreNode* ldst = n->as_LoadStore();
if (is_CAS(n->Opcode(), false)) {
- assert(ldst->trailing_membar() != NULL, "expected trailing membar");
+ assert(ldst->trailing_membar() != nullptr, "expected trailing membar");
} else {
- return ldst->trailing_membar() != NULL;
+ return ldst->trailing_membar() != nullptr;
}
// so we can just return true here
@@ -1734,7 +1734,7 @@ void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
st->print("mov rscratch1, #%d\n\t", framesize - 2 * wordSize);
st->print("sub sp, sp, rscratch1");
}
- if (C->stub_function() == NULL && BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (C->stub_function() == nullptr && BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
st->print("\n\t");
st->print("ldr rscratch1, [guard]\n\t");
st->print("dmb ishld\n\t");
@@ -1783,9 +1783,9 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ build_frame(framesize);
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
- if (BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
// Dummy labels for just measuring the code size
Label dummy_slow_path;
Label dummy_continuation;
@@ -2153,12 +2153,12 @@ void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
if (!ra_)
st->print("N%d = SpillCopy(N%d)", _idx, in(1)->_idx);
else
- implementation(NULL, ra_, false, st);
+ implementation(nullptr, ra_, false, st);
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation(&cbuf, ra_, false, NULL);
+ implementation(&cbuf, ra_, false, nullptr);
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
@@ -2205,14 +2205,14 @@ void MachUEPNode::format(PhaseRegAlloc* ra_, outputStream* st) const
{
st->print_cr("# MachUEPNode");
if (UseCompressedClassPointers) {
- st->print_cr("\tldrw rscratch1, j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
- if (CompressedKlassPointers::shift() != 0) {
- st->print_cr("\tdecode_klass_not_null rscratch1, rscratch1");
- }
+ st->print_cr("\tldrw rscratch1, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tldrw r10, [rscratch2 + CompiledICData::speculated_klass_offset()]\t# compressed klass");
+ st->print_cr("\tcmpw rscratch1, r10");
} else {
- st->print_cr("\tldr rscratch1, j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tldr rscratch1, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tldr r10, [rscratch2 + CompiledICData::speculated_klass_offset()]\t# compressed klass");
+ st->print_cr("\tcmp rscratch1, r10");
}
- st->print_cr("\tcmp r0, rscratch1\t # Inline cache check");
st->print_cr("\tbne, SharedRuntime::_ic_miss_stub");
}
#endif
@@ -2221,14 +2221,7 @@ void MachUEPNode::emit(CodeBuffer& cbuf, PhaseRegAlloc* ra_) const
{
// This is the unverified entry point.
C2_MacroAssembler _masm(&cbuf);
-
- __ cmp_klass(j_rarg0, rscratch2, rscratch1);
- Label skip;
- // TODO
- // can we avoid this skip and still use a reloc?
- __ br(Assembler::EQ, skip);
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
- __ bind(skip);
+ __ ic_check(InteriorEntryAlignment);
}
uint MachUEPNode::size(PhaseRegAlloc* ra_) const
@@ -2249,7 +2242,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer& cbuf)
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -2267,7 +2260,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf)
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -2377,7 +2370,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return MIN2(size, max_size);
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
return Matcher::max_vector_size(bt);
}
@@ -2410,7 +2403,7 @@ MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* generic_opnd,
case Op_VecX: return new vecXOper();
}
ShouldNotReachHere();
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -2582,8 +2575,8 @@ Assembler::Condition to_assembler_cond(BoolTest::mask cond) {
}
// Binary src (Replicate con)
-bool is_valid_sve_arith_imm_pattern(Node* n, Node* m) {
- if (n == NULL || m == NULL) {
+static bool is_valid_sve_arith_imm_pattern(Node* n, Node* m) {
+ if (n == nullptr || m == nullptr) {
return false;
}
@@ -2623,8 +2616,8 @@ bool is_valid_sve_arith_imm_pattern(Node* n, Node* m) {
// (XorV src (Replicate m1))
// (XorVMask src (MaskAll m1))
-bool is_vector_bitwise_not_pattern(Node* n, Node* m) {
- if (n != NULL && m != NULL) {
+static bool is_vector_bitwise_not_pattern(Node* n, Node* m) {
+ if (n != nullptr && m != nullptr) {
return (n->Opcode() == Op_XorV || n->Opcode() == Op_XorVMask) &&
VectorNode::is_all_ones_vector(m);
}
@@ -3430,7 +3423,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL || con == (address)1) {
+ if (con == nullptr || con == (address)1) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -3473,7 +3466,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -3492,7 +3485,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -3675,7 +3668,7 @@ encode %{
Label miss;
C2_MacroAssembler _masm(&cbuf);
__ check_klass_subtype_slow_path(sub_reg, super_reg, temp_reg, result_reg,
- NULL, &miss,
+ nullptr, &miss,
/*set_cond_codes:*/ true);
if ($primary) {
__ mov(result_reg, zr);
@@ -3691,7 +3684,7 @@ encode %{
if (!_method) {
// A call to a runtime wrapper, e.g. new, new_typeArray_Java, uncommon_trap.
call = __ trampoline_call(Address(addr, relocInfo::runtime_call_type));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3705,7 +3698,7 @@ encode %{
RelocationHolder rspec = _optimized_virtual ? opt_virtual_call_Relocation::spec(method_index)
: static_call_Relocation::spec(method_index);
call = __ trampoline_call(Address(addr, rspec));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3715,8 +3708,8 @@ encode %{
cbuf.shared_stub_to_interp_for(_method, call - cbuf.insts_begin());
} else {
// Emit stub for static call
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf, call);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf, call);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3735,7 +3728,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
int method_index = resolved_method_index(cbuf);
address call = __ ic_call((address)$meth$$method, method_index);
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3764,7 +3757,7 @@ encode %{
CodeBlob *cb = CodeCache::find_blob(entry);
if (cb) {
address call = __ trampoline_call(Address(entry, relocInfo::runtime_call_type));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -4663,7 +4656,7 @@ operand immP()
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immP0()
%{
predicate(n->get_ptr() == 0);
@@ -4795,7 +4788,7 @@ operand immN()
interface(CONST_INTER);
%}
-// Narrow NULL Pointer Immediate
+// Narrow Null Pointer Immediate
operand immN0()
%{
predicate(n->get_narrowcon() == 0);
@@ -7219,7 +7212,7 @@ instruct loadConP0(iRegPNoSp dst, immP0 con)
match(Set dst con);
ins_cost(INSN_COST);
- format %{ "mov $dst, $con\t# NULL ptr" %}
+ format %{ "mov $dst, $con\t# null pointer" %}
ins_encode(aarch64_enc_mov_p0(dst, con));
@@ -7233,7 +7226,7 @@ instruct loadConP1(iRegPNoSp dst, immP_1 con)
match(Set dst con);
ins_cost(INSN_COST);
- format %{ "mov $dst, $con\t# NULL ptr" %}
+ format %{ "mov $dst, $con\t# null pointer" %}
ins_encode(aarch64_enc_mov_p1(dst, con));
@@ -7275,7 +7268,7 @@ instruct loadConN0(iRegNNoSp dst, immN0 con)
match(Set dst con);
ins_cost(INSN_COST);
- format %{ "mov $dst, $con\t# compressed NULL ptr" %}
+ format %{ "mov $dst, $con\t# compressed null pointer" %}
ins_encode(aarch64_enc_mov_n0(dst, con));
@@ -15256,7 +15249,7 @@ instruct clearArray_reg_reg(iRegL_R11 cnt, iRegP_R10 base, Universe dummy, rFlag
ins_encode %{
address tpc = __ zero_words($base$$Register, $cnt$$Register);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -15277,7 +15270,7 @@ instruct clearArray_imm_reg(immL cnt, iRegP_R10 base, iRegL_R11 temp, Universe d
ins_encode %{
address tpc = __ zero_words($base$$Register, (uint64_t)$cnt$$constant);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -17104,23 +17097,7 @@ instruct string_equalsL(iRegP_R1 str1, iRegP_R3 str2, iRegI_R4 cnt,
ins_encode %{
// Count is in 8-bit bytes; non-Compact chars are 16 bits.
__ string_equals($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 1);
- %}
- ins_pipe(pipe_class_memory);
-%}
-
-instruct string_equalsU(iRegP_R1 str1, iRegP_R3 str2, iRegI_R4 cnt,
- iRegI_R0 result, rFlagsReg cr)
-%{
- predicate(((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU);
- match(Set result (StrEquals (Binary str1 str2) cnt));
- effect(USE_KILL str1, USE_KILL str2, USE_KILL cnt, KILL cr);
-
- format %{ "String Equals $str1,$str2,$cnt -> $result" %}
- ins_encode %{
- // Count is in 8-bit bytes; non-Compact chars are 16 bits.
- __ string_equals($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 2);
+ $result$$Register, $cnt$$Register);
%}
ins_pipe(pipe_class_memory);
%}
@@ -17142,7 +17119,7 @@ instruct array_equalsB(iRegP_R1 ary1, iRegP_R2 ary2, iRegI_R0 result,
address tpc = __ arrays_equals($ary1$$Register, $ary2$$Register,
$tmp1$$Register, $tmp2$$Register, $tmp3$$Register,
$result$$Register, $tmp$$Register, 1);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -17167,7 +17144,7 @@ instruct array_equalsC(iRegP_R1 ary1, iRegP_R2 ary2, iRegI_R0 result,
address tpc = __ arrays_equals($ary1$$Register, $ary2$$Register,
$tmp1$$Register, $tmp2$$Register, $tmp3$$Register,
$result$$Register, $tmp$$Register, 2);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -17182,7 +17159,7 @@ instruct count_positives(iRegP_R1 ary1, iRegI_R2 len, iRegI_R0 result, rFlagsReg
format %{ "count positives byte[] $ary1,$len -> $result" %}
ins_encode %{
address tpc = __ count_positives($ary1$$Register, $len$$Register, $result$$Register);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -17225,7 +17202,7 @@ instruct string_inflate(Universe dummy, iRegP_R0 src, iRegP_R1 dst, iRegI_R2 len
address tpc = __ byte_array_inflate($src$$Register, $dst$$Register, $len$$Register,
$vtmp0$$FloatRegister, $vtmp1$$FloatRegister,
$vtmp2$$FloatRegister, $tmp$$Register);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
diff --git a/src/hotspot/cpu/aarch64/aarch64_vector.ad b/src/hotspot/cpu/aarch64/aarch64_vector.ad
index 210efa0b760..d611c14f403 100644
--- a/src/hotspot/cpu/aarch64/aarch64_vector.ad
+++ b/src/hotspot/cpu/aarch64/aarch64_vector.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2020, 2023, Arm Limited. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
@@ -126,7 +126,7 @@ source %{
}
}
- bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+ bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
if (UseSVE == 0) {
// These operations are not profitable to be vectorized on NEON, because no direct
// NEON instructions support them. But the match rule support for them is profitable for
diff --git a/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4 b/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4
index 3f4ed020f55..5c4e13d432f 100644
--- a/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4
+++ b/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2020, 2023, Arm Limited. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
@@ -116,7 +116,7 @@ source %{
}
}
- bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+ bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
if (UseSVE == 0) {
// These operations are not profitable to be vectorized on NEON, because no direct
// NEON instructions support them. But the match rule support for them is profitable for
diff --git a/src/hotspot/cpu/aarch64/assembler_aarch64.cpp b/src/hotspot/cpu/aarch64/assembler_aarch64.cpp
index afeb19e906e..c7b867a4207 100644
--- a/src/hotspot/cpu/aarch64/assembler_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/assembler_aarch64.cpp
@@ -187,6 +187,26 @@ void Address::lea(MacroAssembler *as, Register r) const {
zrf(Rd, 0);
}
+// This encoding is similar (but not quite identical) to the encoding used
+// by literal ld/st. see JDK-8324123.
+// PRFM does not support writeback or pre/post index.
+void Assembler::prfm(const Address &adr, prfop pfop) {
+ Address::mode mode = adr.getMode();
+ // PRFM does not support pre/post index
+ guarantee((mode != Address::pre) && (mode != Address::post), "prfm does not support pre/post indexing");
+ if (mode == Address::literal) {
+ starti;
+ f(0b11, 31, 30), f(0b011, 29, 27), f(0b000, 26, 24);
+ f(pfop, 4, 0);
+ int64_t offset = (adr.target() - pc()) >> 2;
+ sf(offset, 23, 5);
+ } else {
+ assert((mode == Address::base_plus_offset)
+ || (mode == Address::base_plus_offset_reg), "must be base_plus_offset/base_plus_offset_reg");
+ ld_st2(as_Register(pfop), adr, 0b11, 0b10);
+ }
+}
+
// An "all-purpose" add/subtract immediate, per ARM documentation:
// A "programmer-friendly" assembler may accept a negative immediate
// between -(2^24 -1) and -1 inclusive, causing it to convert a
diff --git a/src/hotspot/cpu/aarch64/assembler_aarch64.hpp b/src/hotspot/cpu/aarch64/assembler_aarch64.hpp
index 187c1209303..9c05c36706d 100644
--- a/src/hotspot/cpu/aarch64/assembler_aarch64.hpp
+++ b/src/hotspot/cpu/aarch64/assembler_aarch64.hpp
@@ -797,6 +797,8 @@ class Assembler : public AbstractAssembler {
void adrp(Register Rd, const Address &dest, uint64_t &offset) = delete;
+ void prfm(const Address &adr, prfop pfop = PLDL1KEEP);
+
#undef INSN
void add_sub_immediate(Instruction_aarch64 ¤t_insn, Register Rd, Register Rn,
@@ -1574,17 +1576,6 @@ class Assembler : public AbstractAssembler {
#undef INSN
-#define INSN(NAME, size, op) \
- void NAME(const Address &adr, prfop pfop = PLDL1KEEP) { \
- ld_st2(as_Register(pfop), adr, size, op); \
- }
-
- INSN(prfm, 0b11, 0b10); // FIXME: PRFM should not be used with
- // writeback modes, but the assembler
- // doesn't enfore that.
-
-#undef INSN
-
#define INSN(NAME, size, op) \
void NAME(FloatRegister Rt, const Address &adr) { \
ld_st2(as_Register(Rt), adr, size, op, 1); \
diff --git a/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp
index b83d6185062..ba613b62a3e 100644
--- a/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.cpp
@@ -53,7 +53,6 @@
#endif
NEEDS_CLEANUP // remove this definitions ?
-const Register IC_Klass = rscratch2; // where the IC klass is cached
const Register SYNC_header = r0; // synchronization header
const Register SHIFT_count = r0; // where count for shift operations must be
@@ -293,27 +292,7 @@ void LIR_Assembler::osr_entry() {
// inline cache check; done before the frame is built.
int LIR_Assembler::check_icache() {
- Register receiver = FrameMap::receiver_opr->as_register();
- Register ic_klass = IC_Klass;
- int start_offset = __ offset();
- __ inline_cache_check(receiver, ic_klass);
-
- // if icache check fails, then jump to runtime routine
- // Note: RECEIVER must still contain the receiver!
- Label dont;
- __ br(Assembler::EQ, dont);
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- // We align the verified entry point unless the method body
- // (including its inline cache check) will fit in a single 64-byte
- // icache line.
- if (! method()->is_accessor() || __ offset() - start_offset > 4 * 4) {
- // force alignment after the cache check.
- __ align(CodeEntryAlignment);
- }
-
- __ bind(dont);
- return start_offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::clinit_barrier(ciMethod* method) {
@@ -2042,7 +2021,7 @@ void LIR_Assembler::emit_static_call_stub() {
__ relocate(static_stub_Relocation::spec(call_pc));
__ emit_static_call_stub();
- assert(__ offset() - start + CompiledStaticCall::to_trampoline_stub_size()
+ assert(__ offset() - start + CompiledDirectCall::to_trampoline_stub_size()
<= call_stub_size(), "stub too big");
__ end_a_stub();
}
diff --git a/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.hpp b/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.hpp
index 43ec189255f..ef1b5fe2703 100644
--- a/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.hpp
+++ b/src/hotspot/cpu/aarch64/c1_LIRAssembler_aarch64.hpp
@@ -71,8 +71,8 @@ friend class ArrayCopyStub;
void deoptimize_trap(CodeEmitInfo *info);
enum {
- // call stub: CompiledStaticCall::to_interp_stub_size() +
- // CompiledStaticCall::to_trampoline_stub_size()
+ // call stub: CompiledDirectCall::to_interp_stub_size() +
+ // CompiledDirectCall::to_trampoline_stub_size()
_call_stub_size = 13 * NativeInstruction::instruction_size,
_exception_handler_size = DEBUG_ONLY(1*K) NOT_DEBUG(175),
_deopt_handler_size = 7 * NativeInstruction::instruction_size
diff --git a/src/hotspot/cpu/aarch64/c1_LIRGenerator_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_LIRGenerator_aarch64.cpp
index 952e060ed21..fd31d2dde5b 100644
--- a/src/hotspot/cpu/aarch64/c1_LIRGenerator_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/c1_LIRGenerator_aarch64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -831,18 +831,12 @@ void LIRGenerator::do_LibmIntrinsic(Intrinsic* x) {
}
break;
case vmIntrinsics::_dlog:
- if (StubRoutines::dlog() != nullptr) {
- __ call_runtime_leaf(StubRoutines::dlog(), getThreadTemp(), result_reg, cc->args());
- } else {
- __ call_runtime_leaf(CAST_FROM_FN_PTR(address, SharedRuntime::dlog), getThreadTemp(), result_reg, cc->args());
- }
+ // Math.log intrinsic is not implemented on AArch64 (see JDK-8210858),
+ // but we can still call the shared runtime.
+ __ call_runtime_leaf(CAST_FROM_FN_PTR(address, SharedRuntime::dlog), getThreadTemp(), result_reg, cc->args());
break;
case vmIntrinsics::_dlog10:
- if (StubRoutines::dlog10() != nullptr) {
- __ call_runtime_leaf(StubRoutines::dlog10(), getThreadTemp(), result_reg, cc->args());
- } else {
- __ call_runtime_leaf(CAST_FROM_FN_PTR(address, SharedRuntime::dlog10), getThreadTemp(), result_reg, cc->args());
- }
+ __ call_runtime_leaf(CAST_FROM_FN_PTR(address, SharedRuntime::dlog10), getThreadTemp(), result_reg, cc->args());
break;
case vmIntrinsics::_dpow:
if (StubRoutines::dpow() != nullptr) {
diff --git a/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp
index d3a746178f1..c0d1d1747ab 100644
--- a/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp
@@ -308,17 +308,6 @@ void C1_MacroAssembler::allocate_array(Register obj, Register len, Register t1,
verify_oop(obj);
}
-
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache) {
- verify_oop(receiver);
- // explicit null check not needed since load from [klass_offset] causes a trap
- // check against inline cache
- assert(!MacroAssembler::needs_explicit_null_check(oopDesc::klass_offset_in_bytes()), "must add explicit null check");
-
- cmp_klass(receiver, iCache, rscratch1);
-}
-
-
void C1_MacroAssembler::build_frame(int framesize, int bang_size_in_bytes) {
assert(bang_size_in_bytes >= framesize, "stack bang size incorrect");
// Make sure there is enough stack space for this method's activation.
diff --git a/src/hotspot/cpu/aarch64/c1_Runtime1_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_Runtime1_aarch64.cpp
index d2f4744a049..63a32e714e3 100644
--- a/src/hotspot/cpu/aarch64/c1_Runtime1_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/c1_Runtime1_aarch64.cpp
@@ -38,7 +38,6 @@
#include "interpreter/interpreter.hpp"
#include "memory/universe.hpp"
#include "nativeInst_aarch64.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_aarch64.hpp"
diff --git a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp
index 7b9784ec47a..8910fba97a5 100644
--- a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -111,10 +111,10 @@ void C2_MacroAssembler::fast_lock(Register objectReg, Register boxReg, Register
// Handle existing monitor.
bind(object_has_monitor);
- // The object's monitor m is unlocked iff m->owner == NULL,
+ // The object's monitor m is unlocked iff m->owner == nullptr,
// otherwise m->owner may contain a thread or a stack address.
//
- // Try to CAS m->owner from NULL to current thread.
+ // Try to CAS m->owner from null to current thread.
add(tmp, disp_hdr, (in_bytes(ObjectMonitor::owner_offset())-markWord::monitor_value));
cmpxchg(tmp, zr, rthread, Assembler::xword, /*acquire*/ true,
/*release*/ true, /*weak*/ false, tmp3Reg); // Sets flags for result
diff --git a/src/hotspot/cpu/aarch64/compiledIC_aarch64.cpp b/src/hotspot/cpu/aarch64/compiledIC_aarch64.cpp
index c58ff8828bc..23c08f11d1a 100644
--- a/src/hotspot/cpu/aarch64/compiledIC_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/compiledIC_aarch64.cpp
@@ -26,7 +26,6 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
@@ -36,7 +35,7 @@
// ----------------------------------------------------------------------------
#define __ _masm.
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
precond(cbuf.stubs()->start() != badAddress);
precond(cbuf.stubs()->end() != badAddress);
@@ -71,11 +70,11 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark)
}
#undef __
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return MacroAssembler::static_call_stub_size();
}
-int CompiledStaticCall::to_trampoline_stub_size() {
+int CompiledDirectCall::to_trampoline_stub_size() {
// Somewhat pessimistically, we count 3 instructions here (although
// there are only two) because we sometimes emit an alignment nop.
// Trampoline stubs are always word aligned.
@@ -83,21 +82,14 @@ int CompiledStaticCall::to_trampoline_stub_size() {
}
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 4; // 3 in emit_to_interp_stub + 1 in emit_call
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder
= nativeMovConstReg_at(stub + NativeInstruction::instruction_size);
@@ -115,7 +107,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
// Reset stub.
address stub = static_stub->addr();
assert(stub != nullptr, "stub not found");
@@ -132,7 +124,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp b/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp
index d035ab21093..47b6f1f2f38 100644
--- a/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp
@@ -1,6 +1,6 @@
/*
* Copyright (c) 2023, Red Hat, Inc. All rights reserved.
- * Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2023, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -58,7 +58,7 @@ static char* reserve_at_eor_compatible_address(size_t size, bool aslr) {
0x7ffc, 0x7ffe, 0x7fff
};
static constexpr int num_immediates = sizeof(immediates) / sizeof(immediates[0]);
- const int start_index = aslr ? os::random() : 0;
+ const int start_index = aslr ? os::next_random((int)os::javaTimeNanos()) : 0;
constexpr int max_tries = 64;
for (int ntry = 0; result == nullptr && ntry < max_tries; ntry ++) {
// As in os::attempt_reserve_memory_between, we alternate between higher and lower
diff --git a/src/hotspot/cpu/aarch64/frame_aarch64.cpp b/src/hotspot/cpu/aarch64/frame_aarch64.cpp
index c5b2ff8a4c0..8d0fa8895d1 100644
--- a/src/hotspot/cpu/aarch64/frame_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/frame_aarch64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -678,7 +678,7 @@ static void printbc(Method *m, intptr_t bcx) {
printf("%s : %s ==> %s\n", m->name_and_sig_as_C_string(), buf, name);
}
-void internal_pf(uintptr_t sp, uintptr_t fp, uintptr_t pc, uintptr_t bcx) {
+static void internal_pf(uintptr_t sp, uintptr_t fp, uintptr_t pc, uintptr_t bcx) {
if (! fp)
return;
diff --git a/src/hotspot/cpu/aarch64/globals_aarch64.hpp b/src/hotspot/cpu/aarch64/globals_aarch64.hpp
index b26eaa4bfcd..293cc6eb0d0 100644
--- a/src/hotspot/cpu/aarch64/globals_aarch64.hpp
+++ b/src/hotspot/cpu/aarch64/globals_aarch64.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2000, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2015, 2019, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -117,7 +117,7 @@ define_pd_global(intx, InlineSmallCode, 1000);
"Use prfm hint with specified distance in compiled code." \
"Value -1 means off.") \
range(-1, 4096) \
- product(ccstr, OnSpinWaitInst, "none", DIAGNOSTIC, \
+ product(ccstr, OnSpinWaitInst, "yield", DIAGNOSTIC, \
"The instruction to use to implement " \
"java.lang.Thread.onSpinWait()." \
"Options: none, nop, isb, yield.") \
diff --git a/src/hotspot/cpu/aarch64/icBuffer_aarch64.cpp b/src/hotspot/cpu/aarch64/icBuffer_aarch64.cpp
deleted file mode 100644
index bd8cfc42600..00000000000
--- a/src/hotspot/cpu/aarch64/icBuffer_aarch64.cpp
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2014, Red Hat Inc. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/macroAssembler.hpp"
-#include "asm/macroAssembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_aarch64.hpp"
-#include "oops/oop.inline.hpp"
-
-int InlineCacheBuffer::ic_stub_code_size() {
- return (MacroAssembler::far_branches() ? 6 : 4) * NativeInstruction::instruction_size;
-}
-
-#define __ masm->
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point) {
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler* masm = new MacroAssembler(&code);
- // note: even though the code contains an embedded value, we do not need reloc info
- // because
- // (1) the value is old (i.e., doesn't matter for scavenges)
- // (2) these ICStubs are removed *before* a GC happens, so the roots disappear
- // assert(cached_value == nullptr || cached_oop->is_perm(), "must be perm oop");
-
- address start = __ pc();
- Label l;
- __ ldr(rscratch2, l);
- int jump_code_size = __ far_jump(ExternalAddress(entry_point));
- // IC stub code size is not expected to vary depending on target address.
- // We use NOPs to make the [ldr + far_jump + nops + int64] stub size equal to ic_stub_code_size.
- for (int size = NativeInstruction::instruction_size + jump_code_size + 8;
- size < ic_stub_code_size(); size += NativeInstruction::instruction_size) {
- __ nop();
- }
- __ bind(l);
- assert((uintptr_t)__ pc() % wordSize == 0, "");
- __ emit_int64((int64_t)cached_value);
- // Only need to invalidate the 1st two instructions - not the whole ic stub
- ICache::invalidate_range(code_begin, InlineCacheBuffer::ic_stub_code_size());
- assert(__ pc() - start == ic_stub_code_size(), "must be");
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // creation also verifies the object
- NativeJump* jump = nativeJump_at(code_begin + 4);
- return jump->jump_destination();
-}
-
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- // The word containing the cached value is at the end of this IC buffer
- uintptr_t *p = (uintptr_t *)(code_begin + ic_stub_code_size() - wordSize);
- void* o = (void*)*p;
- return o;
-}
diff --git a/src/hotspot/cpu/aarch64/immediate_aarch64.cpp b/src/hotspot/cpu/aarch64/immediate_aarch64.cpp
index 3d87fde2b5b..7caafc19fbd 100644
--- a/src/hotspot/cpu/aarch64/immediate_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/immediate_aarch64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2023, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -53,7 +53,7 @@ struct li_pair {
static struct li_pair InverseLITable[LI_TABLE_SIZE];
// comparator to sort entries in the inverse table
-int compare_immediate_pair(const void *i1, const void *i2)
+static int compare_immediate_pair(const void *i1, const void *i2)
{
struct li_pair *li1 = (struct li_pair *)i1;
struct li_pair *li2 = (struct li_pair *)i2;
@@ -142,7 +142,7 @@ static inline uint32_t uimm(uint32_t val, int hi, int lo)
// result
// a bit string containing count copies of input bit string
//
-uint64_t replicate(uint64_t bits, int nbits, int count)
+static uint64_t replicate(uint64_t bits, int nbits, int count)
{
assert(count > 0, "must be");
assert(nbits > 0, "must be");
@@ -231,8 +231,8 @@ uint64_t replicate(uint64_t bits, int nbits, int count)
// For historical reasons the implementation of this function is much
// more convoluted than is really necessary.
-int expandLogicalImmediate(uint32_t immN, uint32_t immr,
- uint32_t imms, uint64_t &bimm)
+static int expandLogicalImmediate(uint32_t immN, uint32_t immr,
+ uint32_t imms, uint64_t &bimm)
{
int len; // ought to be <= 6
uint32_t levels; // 6 bits
@@ -446,4 +446,3 @@ uint32_t encoding_for_fp_immediate(float immediate)
res = (s << 7) | (r << 4) | f;
return res;
}
-
diff --git a/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp
index a3c560b28d3..b19587ebe76 100644
--- a/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/macroAssembler_aarch64.cpp
@@ -29,6 +29,7 @@
#include "asm/assembler.hpp"
#include "asm/assembler.inline.hpp"
#include "ci/ciEnv.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/compileTask.hpp"
#include "compiler/disassembler.hpp"
#include "compiler/oopMap.hpp"
@@ -965,7 +966,7 @@ int MacroAssembler::max_trampoline_stub_size() {
}
void MacroAssembler::emit_static_call_stub() {
- // CompiledDirectStaticCall::set_to_interpreted knows the
+ // CompiledDirectCall::set_to_interpreted knows the
// exact layout of this stub.
isb();
@@ -995,10 +996,51 @@ address MacroAssembler::ic_call(address entry, jint method_index) {
// address const_ptr = long_constant((jlong)Universe::non_oop_word());
// uintptr_t offset;
// ldr_constant(rscratch2, const_ptr);
- movptr(rscratch2, (uintptr_t)Universe::non_oop_word());
+ movptr(rscratch2, (intptr_t)Universe::non_oop_word());
return trampoline_call(Address(entry, rh));
}
+int MacroAssembler::ic_check_size() {
+ if (target_needs_far_branch(CAST_FROM_FN_PTR(address, SharedRuntime::get_ic_miss_stub()))) {
+ return NativeInstruction::instruction_size * 7;
+ } else {
+ return NativeInstruction::instruction_size * 5;
+ }
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ Register receiver = j_rarg0;
+ Register data = rscratch2;
+ Register tmp1 = rscratch1;
+ Register tmp2 = r10;
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, offset() + ic_check_size());
+
+ int uep_offset = offset();
+
+ if (UseCompressedClassPointers) {
+ ldrw(tmp1, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ ldrw(tmp2, Address(data, CompiledICData::speculated_klass_offset()));
+ cmpw(tmp1, tmp2);
+ } else {
+ ldr(tmp1, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ ldr(tmp2, Address(data, CompiledICData::speculated_klass_offset()));
+ cmp(tmp1, tmp2);
+ }
+
+ Label dont;
+ br(Assembler::EQ, dont);
+ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
+ bind(dont);
+ assert((offset() % end_alignment) == 0, "Misaligned verified entry point");
+
+ return uep_offset;
+}
+
// Implementation of call_VM versions
void MacroAssembler::call_VM(Register oop_result,
@@ -1100,7 +1142,14 @@ void MacroAssembler::get_vm_result_2(Register metadata_result, Register java_thr
}
void MacroAssembler::align(int modulus) {
- while (offset() % modulus != 0) nop();
+ align(modulus, offset());
+}
+
+// Ensure that the code at target bytes offset from the current offset() is aligned
+// according to modulus.
+void MacroAssembler::align(int modulus, int target) {
+ int delta = target - offset();
+ while ((offset() + delta) % modulus != 0) nop();
}
void MacroAssembler::post_call_nop() {
@@ -1197,7 +1246,7 @@ void MacroAssembler::lookup_interface_method(Register recv_klass,
}
// Look up the method for a megamorphic invokeinterface call in a single pass over itable:
-// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICHolder
+// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICData
// - find a holder_klass (class that implements the method) vtable offset and get the method from vtable by index
// The target method is determined by .
// The receiver klass is in recv_klass.
@@ -4258,108 +4307,117 @@ void MacroAssembler::kernel_crc32_common_fold_using_crypto_pmull(Register crc, R
}
add(table, table, table_offset);
+ // Registers v0..v7 are used as data registers.
+ // Registers v16..v31 are used as tmp registers.
sub(buf, buf, 0x10);
- ldrq(v1, Address(buf, 0x10));
- ldrq(v2, Address(buf, 0x20));
- ldrq(v3, Address(buf, 0x30));
- ldrq(v4, Address(buf, 0x40));
- ldrq(v5, Address(buf, 0x50));
- ldrq(v6, Address(buf, 0x60));
- ldrq(v7, Address(buf, 0x70));
- ldrq(v8, Address(pre(buf, 0x80)));
-
- movi(v25, T4S, 0);
- mov(v25, S, 0, crc);
- eor(v1, T16B, v1, v25);
-
- ldrq(v0, Address(table));
+ ldrq(v0, Address(buf, 0x10));
+ ldrq(v1, Address(buf, 0x20));
+ ldrq(v2, Address(buf, 0x30));
+ ldrq(v3, Address(buf, 0x40));
+ ldrq(v4, Address(buf, 0x50));
+ ldrq(v5, Address(buf, 0x60));
+ ldrq(v6, Address(buf, 0x70));
+ ldrq(v7, Address(pre(buf, 0x80)));
+
+ movi(v31, T4S, 0);
+ mov(v31, S, 0, crc);
+ eor(v0, T16B, v0, v31);
+
+ // Register v16 contains constants from the crc table.
+ ldrq(v16, Address(table));
b(CRC_by128_loop);
align(OptoLoopAlignment);
BIND(CRC_by128_loop);
- pmull (v9, T1Q, v1, v0, T1D);
- pmull2(v10, T1Q, v1, v0, T2D);
- ldrq(v1, Address(buf, 0x10));
- eor3(v1, T16B, v9, v10, v1);
-
- pmull (v11, T1Q, v2, v0, T1D);
- pmull2(v12, T1Q, v2, v0, T2D);
- ldrq(v2, Address(buf, 0x20));
- eor3(v2, T16B, v11, v12, v2);
-
- pmull (v13, T1Q, v3, v0, T1D);
- pmull2(v14, T1Q, v3, v0, T2D);
- ldrq(v3, Address(buf, 0x30));
- eor3(v3, T16B, v13, v14, v3);
-
- pmull (v15, T1Q, v4, v0, T1D);
- pmull2(v16, T1Q, v4, v0, T2D);
- ldrq(v4, Address(buf, 0x40));
- eor3(v4, T16B, v15, v16, v4);
-
- pmull (v17, T1Q, v5, v0, T1D);
- pmull2(v18, T1Q, v5, v0, T2D);
- ldrq(v5, Address(buf, 0x50));
- eor3(v5, T16B, v17, v18, v5);
-
- pmull (v19, T1Q, v6, v0, T1D);
- pmull2(v20, T1Q, v6, v0, T2D);
- ldrq(v6, Address(buf, 0x60));
- eor3(v6, T16B, v19, v20, v6);
-
- pmull (v21, T1Q, v7, v0, T1D);
- pmull2(v22, T1Q, v7, v0, T2D);
- ldrq(v7, Address(buf, 0x70));
- eor3(v7, T16B, v21, v22, v7);
-
- pmull (v23, T1Q, v8, v0, T1D);
- pmull2(v24, T1Q, v8, v0, T2D);
- ldrq(v8, Address(pre(buf, 0x80)));
- eor3(v8, T16B, v23, v24, v8);
+ pmull (v17, T1Q, v0, v16, T1D);
+ pmull2(v18, T1Q, v0, v16, T2D);
+ ldrq(v0, Address(buf, 0x10));
+ eor3(v0, T16B, v17, v18, v0);
+
+ pmull (v19, T1Q, v1, v16, T1D);
+ pmull2(v20, T1Q, v1, v16, T2D);
+ ldrq(v1, Address(buf, 0x20));
+ eor3(v1, T16B, v19, v20, v1);
+
+ pmull (v21, T1Q, v2, v16, T1D);
+ pmull2(v22, T1Q, v2, v16, T2D);
+ ldrq(v2, Address(buf, 0x30));
+ eor3(v2, T16B, v21, v22, v2);
+
+ pmull (v23, T1Q, v3, v16, T1D);
+ pmull2(v24, T1Q, v3, v16, T2D);
+ ldrq(v3, Address(buf, 0x40));
+ eor3(v3, T16B, v23, v24, v3);
+
+ pmull (v25, T1Q, v4, v16, T1D);
+ pmull2(v26, T1Q, v4, v16, T2D);
+ ldrq(v4, Address(buf, 0x50));
+ eor3(v4, T16B, v25, v26, v4);
+
+ pmull (v27, T1Q, v5, v16, T1D);
+ pmull2(v28, T1Q, v5, v16, T2D);
+ ldrq(v5, Address(buf, 0x60));
+ eor3(v5, T16B, v27, v28, v5);
+
+ pmull (v29, T1Q, v6, v16, T1D);
+ pmull2(v30, T1Q, v6, v16, T2D);
+ ldrq(v6, Address(buf, 0x70));
+ eor3(v6, T16B, v29, v30, v6);
+
+ // Reuse registers v23, v24.
+ // Using them won't block the first instruction of the next iteration.
+ pmull (v23, T1Q, v7, v16, T1D);
+ pmull2(v24, T1Q, v7, v16, T2D);
+ ldrq(v7, Address(pre(buf, 0x80)));
+ eor3(v7, T16B, v23, v24, v7);
subs(len, len, 0x80);
br(Assembler::GE, CRC_by128_loop);
// fold into 512 bits
- ldrq(v0, Address(table, 0x10));
+ // Use v31 for constants because v16 can be still in use.
+ ldrq(v31, Address(table, 0x10));
- pmull (v10, T1Q, v1, v0, T1D);
- pmull2(v11, T1Q, v1, v0, T2D);
- eor3(v1, T16B, v10, v11, v5);
+ pmull (v17, T1Q, v0, v31, T1D);
+ pmull2(v18, T1Q, v0, v31, T2D);
+ eor3(v0, T16B, v17, v18, v4);
- pmull (v12, T1Q, v2, v0, T1D);
- pmull2(v13, T1Q, v2, v0, T2D);
- eor3(v2, T16B, v12, v13, v6);
+ pmull (v19, T1Q, v1, v31, T1D);
+ pmull2(v20, T1Q, v1, v31, T2D);
+ eor3(v1, T16B, v19, v20, v5);
- pmull (v14, T1Q, v3, v0, T1D);
- pmull2(v15, T1Q, v3, v0, T2D);
- eor3(v3, T16B, v14, v15, v7);
+ pmull (v21, T1Q, v2, v31, T1D);
+ pmull2(v22, T1Q, v2, v31, T2D);
+ eor3(v2, T16B, v21, v22, v6);
- pmull (v16, T1Q, v4, v0, T1D);
- pmull2(v17, T1Q, v4, v0, T2D);
- eor3(v4, T16B, v16, v17, v8);
+ pmull (v23, T1Q, v3, v31, T1D);
+ pmull2(v24, T1Q, v3, v31, T2D);
+ eor3(v3, T16B, v23, v24, v7);
// fold into 128 bits
- ldrq(v5, Address(table, 0x20));
- pmull (v10, T1Q, v1, v5, T1D);
- pmull2(v11, T1Q, v1, v5, T2D);
- eor3(v4, T16B, v4, v10, v11);
-
- ldrq(v6, Address(table, 0x30));
- pmull (v12, T1Q, v2, v6, T1D);
- pmull2(v13, T1Q, v2, v6, T2D);
- eor3(v4, T16B, v4, v12, v13);
-
- ldrq(v7, Address(table, 0x40));
- pmull (v14, T1Q, v3, v7, T1D);
- pmull2(v15, T1Q, v3, v7, T2D);
- eor3(v1, T16B, v4, v14, v15);
+ // Use v17 for constants because v31 can be still in use.
+ ldrq(v17, Address(table, 0x20));
+ pmull (v25, T1Q, v0, v17, T1D);
+ pmull2(v26, T1Q, v0, v17, T2D);
+ eor3(v3, T16B, v3, v25, v26);
+
+ // Use v18 for constants because v17 can be still in use.
+ ldrq(v18, Address(table, 0x30));
+ pmull (v27, T1Q, v1, v18, T1D);
+ pmull2(v28, T1Q, v1, v18, T2D);
+ eor3(v3, T16B, v3, v27, v28);
+
+ // Use v19 for constants because v18 can be still in use.
+ ldrq(v19, Address(table, 0x40));
+ pmull (v29, T1Q, v2, v19, T1D);
+ pmull2(v30, T1Q, v2, v19, T2D);
+ eor3(v0, T16B, v3, v29, v30);
add(len, len, 0x80);
add(buf, buf, 0x10);
- mov(tmp0, v1, D, 0);
- mov(tmp1, v1, D, 1);
+ mov(tmp0, v0, D, 0);
+ mov(tmp1, v0, D, 1);
}
SkipIfEqual::SkipIfEqual(
@@ -5339,28 +5397,25 @@ address MacroAssembler::arrays_equals(Register a1, Register a2, Register tmp3,
// For Strings we're passed the address of the first characters in a1
// and a2 and the length in cnt1.
-// elem_size is the element size in bytes: either 1 or 2.
// There are two implementations. For arrays >= 8 bytes, all
// comparisons (including the final one, which may overlap) are
// performed 8 bytes at a time. For strings < 8 bytes, we compare a
// halfword, then a short, and then a byte.
void MacroAssembler::string_equals(Register a1, Register a2,
- Register result, Register cnt1, int elem_size)
+ Register result, Register cnt1)
{
Label SAME, DONE, SHORT, NEXT_WORD;
Register tmp1 = rscratch1;
Register tmp2 = rscratch2;
Register cnt2 = tmp2; // cnt2 only used in array length compare
- assert(elem_size == 1 || elem_size == 2, "must be 2 or 1 byte");
assert_different_registers(a1, a2, result, cnt1, rscratch1, rscratch2);
#ifndef PRODUCT
{
- const char kind = (elem_size == 2) ? 'U' : 'L';
char comment[64];
- snprintf(comment, sizeof comment, "{string_equals%c", kind);
+ snprintf(comment, sizeof comment, "{string_equalsL");
BLOCK_COMMENT(comment);
}
#endif
@@ -5408,14 +5463,12 @@ void MacroAssembler::string_equals(Register a1, Register a2,
cbnzw(tmp1, DONE);
}
bind(TAIL01);
- if (elem_size == 1) { // Only needed when comparing 1-byte elements
- tbz(cnt1, 0, SAME); // 0-1 bytes left.
+ tbz(cnt1, 0, SAME); // 0-1 bytes left.
{
- ldrb(tmp1, a1);
- ldrb(tmp2, a2);
- eorw(tmp1, tmp1, tmp2);
- cbnzw(tmp1, DONE);
- }
+ ldrb(tmp1, a1);
+ ldrb(tmp2, a2);
+ eorw(tmp1, tmp1, tmp2);
+ cbnzw(tmp1, DONE);
}
// Arrays are equal.
bind(SAME);
diff --git a/src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp b/src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp
index 84931c409cf..990e725d099 100644
--- a/src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp
+++ b/src/hotspot/cpu/aarch64/macroAssembler_aarch64.hpp
@@ -720,6 +720,7 @@ class MacroAssembler: public Assembler {
// Alignment
void align(int modulus);
+ void align(int modulus, int target);
// nop
void post_call_nop();
@@ -1247,6 +1248,8 @@ class MacroAssembler: public Assembler {
// Emit the CompiledIC call idiom
address ic_call(address entry, jint method_index = 0);
+ static int ic_check_size();
+ int ic_check(int end_alignment);
public:
@@ -1399,8 +1402,7 @@ class MacroAssembler: public Assembler {
address arrays_equals(Register a1, Register a2, Register result, Register cnt1,
Register tmp1, Register tmp2, Register tmp3, int elem_size);
- void string_equals(Register a1, Register a2, Register result, Register cnt1,
- int elem_size);
+ void string_equals(Register a1, Register a2, Register result, Register cnt1);
void fill_words(Register base, Register cnt, Register value);
address zero_words(Register base, uint64_t cnt);
@@ -1425,11 +1427,6 @@ class MacroAssembler: public Assembler {
FloatRegister vtmp2, FloatRegister vtmp3,
FloatRegister vtmp4, FloatRegister vtmp5);
- void fast_log(FloatRegister vtmp0, FloatRegister vtmp1, FloatRegister vtmp2,
- FloatRegister vtmp3, FloatRegister vtmp4, FloatRegister vtmp5,
- FloatRegister tmpC1, FloatRegister tmpC2, FloatRegister tmpC3,
- FloatRegister tmpC4, Register tmp1, Register tmp2,
- Register tmp3, Register tmp4, Register tmp5);
void generate_dsin_dcos(bool isCos, address npio2_hw, address two_over_pi,
address pio2, address dsin_coef, address dcos_coef);
private:
diff --git a/src/hotspot/cpu/aarch64/macroAssembler_aarch64_log.cpp b/src/hotspot/cpu/aarch64/macroAssembler_aarch64_log.cpp
deleted file mode 100644
index 45772ff1afd..00000000000
--- a/src/hotspot/cpu/aarch64/macroAssembler_aarch64_log.cpp
+++ /dev/null
@@ -1,366 +0,0 @@
-/* Copyright (c) 2018, Cavium. All rights reserved. (By BELLSOFT)
- * Copyright (c) 2016, 2021, Intel Corporation. All rights reserved.
- * Intel Math Library (LIBM) Source Code
- *
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/assembler.hpp"
-#include "asm/assembler.inline.hpp"
-#include "macroAssembler_aarch64.hpp"
-
-// Algorithm idea is taken from x86 hotspot intrinsic and adapted for AARCH64.
-//
-// For mathematical background please refer to the following literature:
-//
-// Tang, Ping-Tak Peter.
-// Table-driven implementation of the logarithm function
-// in IEEE floating-point arithmetic.
-// ACM Transactions on Mathematical Software (TOMS) 16, no. 4, 1990: 378-400.
-
-/******************************************************************************/
-// ALGORITHM DESCRIPTION - LOG()
-// ---------------------
-//
-// x=2^k * mx, mx in [1,2)
-//
-// Get B~1/mx based on the output of frecpe instruction (B0)
-// B = int((B0*2^7+0.5))/2^7
-//
-// Reduced argument: r=B*mx-1.0 (computed accurately in high and low parts)
-//
-// Result: k*log(2) - log(B) + p(r) if |x-1| >= small value (2^-6) and
-// p(r) is a degree 7 polynomial
-// -log(B) read from data table (high, low parts)
-// Result is formed from high and low parts
-//
-// Special cases:
-// 1. log(NaN) = quiet NaN
-// 2. log(+INF) = that INF
-// 3. log(0) = -INF
-// 4. log(1) = +0
-// 5. log(x) = NaN if x < -0, including -INF
-//
-/******************************************************************************/
-
-// Table with p(r) polynomial coefficients
-// and table representation of logarithm values (hi and low parts)
-ATTRIBUTE_ALIGNED(64) juint _L_tbl[] =
-{
- // coefficients of p(r) polynomial:
- // _coeff[]
- 0x00000000UL, 0xbfd00000UL, // C1_0 = -0.25
- 0x92492492UL, 0x3fc24924UL, // C1_1 = 0.14285714285714285
- 0x55555555UL, 0x3fd55555UL, // C2_0 = 0.3333333333333333
- 0x3d6fb175UL, 0xbfc5555eUL, // C2_1 = -0.16666772842235003
- 0x00000000UL, 0xbfe00000UL, // C3_0 = -0.5
- 0x9999999aUL, 0x3fc99999UL, // C3_1 = 0.2
- // _log2[]
- 0xfefa3800UL, 0x3fa62e42UL, // C4_0 = 0.043321698784993146
- 0x93c76730UL, 0x3ceef357UL, // C4_1 = 3.436201886692732e-15
- // _L_tbl[] with logarithm values (hi and low parts)
- 0xfefa3800UL, 0x3fe62e42UL, 0x93c76730UL, 0x3d2ef357UL, 0xaa241800UL,
- 0x3fe5ee82UL, 0x0cda46beUL, 0x3d220238UL, 0x5c364800UL, 0x3fe5af40UL,
- 0xac10c9fbUL, 0x3d2dfa63UL, 0x26bb8c00UL, 0x3fe5707aUL, 0xff3303ddUL,
- 0x3d09980bUL, 0x26867800UL, 0x3fe5322eUL, 0x5d257531UL, 0x3d05ccc4UL,
- 0x835a5000UL, 0x3fe4f45aUL, 0x6d93b8fbUL, 0xbd2e6c51UL, 0x6f970c00UL,
- 0x3fe4b6fdUL, 0xed4c541cUL, 0x3cef7115UL, 0x27e8a400UL, 0x3fe47a15UL,
- 0xf94d60aaUL, 0xbd22cb6aUL, 0xf2f92400UL, 0x3fe43d9fUL, 0x481051f7UL,
- 0xbcfd984fUL, 0x2125cc00UL, 0x3fe4019cUL, 0x30f0c74cUL, 0xbd26ce79UL,
- 0x0c36c000UL, 0x3fe3c608UL, 0x7cfe13c2UL, 0xbd02b736UL, 0x17197800UL,
- 0x3fe38ae2UL, 0xbb5569a4UL, 0xbd218b7aUL, 0xad9d8c00UL, 0x3fe35028UL,
- 0x9527e6acUL, 0x3d10b83fUL, 0x44340800UL, 0x3fe315daUL, 0xc5a0ed9cUL,
- 0xbd274e93UL, 0x57b0e000UL, 0x3fe2dbf5UL, 0x07b9dc11UL, 0xbd17a6e5UL,
- 0x6d0ec000UL, 0x3fe2a278UL, 0xe797882dUL, 0x3d206d2bUL, 0x1134dc00UL,
- 0x3fe26962UL, 0x05226250UL, 0xbd0b61f1UL, 0xd8bebc00UL, 0x3fe230b0UL,
- 0x6e48667bUL, 0x3d12fc06UL, 0x5fc61800UL, 0x3fe1f863UL, 0xc9fe81d3UL,
- 0xbd2a7242UL, 0x49ae6000UL, 0x3fe1c078UL, 0xed70e667UL, 0x3cccacdeUL,
- 0x40f23c00UL, 0x3fe188eeUL, 0xf8ab4650UL, 0x3d14cc4eUL, 0xf6f29800UL,
- 0x3fe151c3UL, 0xa293ae49UL, 0xbd2edd97UL, 0x23c75c00UL, 0x3fe11af8UL,
- 0xbb9ddcb2UL, 0xbd258647UL, 0x8611cc00UL, 0x3fe0e489UL, 0x07801742UL,
- 0x3d1c2998UL, 0xe2d05400UL, 0x3fe0ae76UL, 0x887e7e27UL, 0x3d1f486bUL,
- 0x0533c400UL, 0x3fe078bfUL, 0x41edf5fdUL, 0x3d268122UL, 0xbe760400UL,
- 0x3fe04360UL, 0xe79539e0UL, 0xbd04c45fUL, 0xe5b20800UL, 0x3fe00e5aUL,
- 0xb1727b1cUL, 0xbd053ba3UL, 0xaf7a4800UL, 0x3fdfb358UL, 0x3c164935UL,
- 0x3d0085faUL, 0xee031800UL, 0x3fdf4aa7UL, 0x6f014a8bUL, 0x3d12cde5UL,
- 0x56b41000UL, 0x3fdee2a1UL, 0x5a470251UL, 0x3d2f27f4UL, 0xc3ddb000UL,
- 0x3fde7b42UL, 0x5372bd08UL, 0xbd246550UL, 0x1a272800UL, 0x3fde148aUL,
- 0x07322938UL, 0xbd1326b2UL, 0x484c9800UL, 0x3fddae75UL, 0x60dc616aUL,
- 0xbd1ea42dUL, 0x46def800UL, 0x3fdd4902UL, 0xe9a767a8UL, 0x3d235bafUL,
- 0x18064800UL, 0x3fdce42fUL, 0x3ec7a6b0UL, 0xbd0797c3UL, 0xc7455800UL,
- 0x3fdc7ff9UL, 0xc15249aeUL, 0xbd29b6ddUL, 0x693fa000UL, 0x3fdc1c60UL,
- 0x7fe8e180UL, 0x3d2cec80UL, 0x1b80e000UL, 0x3fdbb961UL, 0xf40a666dUL,
- 0x3d27d85bUL, 0x04462800UL, 0x3fdb56faUL, 0x2d841995UL, 0x3d109525UL,
- 0x5248d000UL, 0x3fdaf529UL, 0x52774458UL, 0xbd217cc5UL, 0x3c8ad800UL,
- 0x3fda93edUL, 0xbea77a5dUL, 0x3d1e36f2UL, 0x0224f800UL, 0x3fda3344UL,
- 0x7f9d79f5UL, 0x3d23c645UL, 0xea15f000UL, 0x3fd9d32bUL, 0x10d0c0b0UL,
- 0xbd26279eUL, 0x43135800UL, 0x3fd973a3UL, 0xa502d9f0UL, 0xbd152313UL,
- 0x635bf800UL, 0x3fd914a8UL, 0x2ee6307dUL, 0xbd1766b5UL, 0xa88b3000UL,
- 0x3fd8b639UL, 0xe5e70470UL, 0xbd205ae1UL, 0x776dc800UL, 0x3fd85855UL,
- 0x3333778aUL, 0x3d2fd56fUL, 0x3bd81800UL, 0x3fd7fafaUL, 0xc812566aUL,
- 0xbd272090UL, 0x687cf800UL, 0x3fd79e26UL, 0x2efd1778UL, 0x3d29ec7dUL,
- 0x76c67800UL, 0x3fd741d8UL, 0x49dc60b3UL, 0x3d2d8b09UL, 0xe6af1800UL,
- 0x3fd6e60eUL, 0x7c222d87UL, 0x3d172165UL, 0x3e9c6800UL, 0x3fd68ac8UL,
- 0x2756eba0UL, 0x3d20a0d3UL, 0x0b3ab000UL, 0x3fd63003UL, 0xe731ae00UL,
- 0xbd2db623UL, 0xdf596000UL, 0x3fd5d5bdUL, 0x08a465dcUL, 0xbd0a0b2aUL,
- 0x53c8d000UL, 0x3fd57bf7UL, 0xee5d40efUL, 0x3d1fadedUL, 0x0738a000UL,
- 0x3fd522aeUL, 0x8164c759UL, 0x3d2ebe70UL, 0x9e173000UL, 0x3fd4c9e0UL,
- 0x1b0ad8a4UL, 0xbd2e2089UL, 0xc271c800UL, 0x3fd4718dUL, 0x0967d675UL,
- 0xbd2f27ceUL, 0x23d5e800UL, 0x3fd419b4UL, 0xec90e09dUL, 0x3d08e436UL,
- 0x77333000UL, 0x3fd3c252UL, 0xb606bd5cUL, 0x3d183b54UL, 0x76be1000UL,
- 0x3fd36b67UL, 0xb0f177c8UL, 0x3d116ecdUL, 0xe1d36000UL, 0x3fd314f1UL,
- 0xd3213cb8UL, 0xbd28e27aUL, 0x7cdc9000UL, 0x3fd2bef0UL, 0x4a5004f4UL,
- 0x3d2a9cfaUL, 0x1134d800UL, 0x3fd26962UL, 0xdf5bb3b6UL, 0x3d2c93c1UL,
- 0x6d0eb800UL, 0x3fd21445UL, 0xba46baeaUL, 0x3d0a87deUL, 0x635a6800UL,
- 0x3fd1bf99UL, 0x5147bdb7UL, 0x3d2ca6edUL, 0xcbacf800UL, 0x3fd16b5cUL,
- 0xf7a51681UL, 0x3d2b9acdUL, 0x8227e800UL, 0x3fd1178eUL, 0x63a5f01cUL,
- 0xbd2c210eUL, 0x67616000UL, 0x3fd0c42dUL, 0x163ceae9UL, 0x3d27188bUL,
- 0x604d5800UL, 0x3fd07138UL, 0x16ed4e91UL, 0x3cf89cdbUL, 0x5626c800UL,
- 0x3fd01eaeUL, 0x1485e94aUL, 0xbd16f08cUL, 0x6cb3b000UL, 0x3fcf991cUL,
- 0xca0cdf30UL, 0x3d1bcbecUL, 0xe4dd0000UL, 0x3fcef5adUL, 0x65bb8e11UL,
- 0xbcca2115UL, 0xffe71000UL, 0x3fce530eUL, 0x6041f430UL, 0x3cc21227UL,
- 0xb0d49000UL, 0x3fcdb13dUL, 0xf715b035UL, 0xbd2aff2aUL, 0xf2656000UL,
- 0x3fcd1037UL, 0x75b6f6e4UL, 0xbd084a7eUL, 0xc6f01000UL, 0x3fcc6ffbUL,
- 0xc5962bd2UL, 0xbcf1ec72UL, 0x383be000UL, 0x3fcbd087UL, 0x595412b6UL,
- 0xbd2d4bc4UL, 0x575bd000UL, 0x3fcb31d8UL, 0x4eace1aaUL, 0xbd0c358dUL,
- 0x3c8ae000UL, 0x3fca93edUL, 0x50562169UL, 0xbd287243UL, 0x07089000UL,
- 0x3fc9f6c4UL, 0x6865817aUL, 0x3d29904dUL, 0xdcf70000UL, 0x3fc95a5aUL,
- 0x58a0ff6fUL, 0x3d07f228UL, 0xeb390000UL, 0x3fc8beafUL, 0xaae92cd1UL,
- 0xbd073d54UL, 0x6551a000UL, 0x3fc823c1UL, 0x9a631e83UL, 0x3d1e0ddbUL,
- 0x85445000UL, 0x3fc7898dUL, 0x70914305UL, 0xbd1c6610UL, 0x8b757000UL,
- 0x3fc6f012UL, 0xe59c21e1UL, 0xbd25118dUL, 0xbe8c1000UL, 0x3fc6574eUL,
- 0x2c3c2e78UL, 0x3d19cf8bUL, 0x6b544000UL, 0x3fc5bf40UL, 0xeb68981cUL,
- 0xbd127023UL, 0xe4a1b000UL, 0x3fc527e5UL, 0xe5697dc7UL, 0x3d2633e8UL,
- 0x8333b000UL, 0x3fc4913dUL, 0x54fdb678UL, 0x3d258379UL, 0xa5993000UL,
- 0x3fc3fb45UL, 0x7e6a354dUL, 0xbd2cd1d8UL, 0xb0159000UL, 0x3fc365fcUL,
- 0x234b7289UL, 0x3cc62fa8UL, 0x0c868000UL, 0x3fc2d161UL, 0xcb81b4a1UL,
- 0x3d039d6cUL, 0x2a49c000UL, 0x3fc23d71UL, 0x8fd3df5cUL, 0x3d100d23UL,
- 0x7e23f000UL, 0x3fc1aa2bUL, 0x44389934UL, 0x3d2ca78eUL, 0x8227e000UL,
- 0x3fc1178eUL, 0xce2d07f2UL, 0x3d21ef78UL, 0xb59e4000UL, 0x3fc08598UL,
- 0x7009902cUL, 0xbd27e5ddUL, 0x39dbe000UL, 0x3fbfe891UL, 0x4fa10afdUL,
- 0xbd2534d6UL, 0x830a2000UL, 0x3fbec739UL, 0xafe645e0UL, 0xbd2dc068UL,
- 0x63844000UL, 0x3fbda727UL, 0x1fa71733UL, 0x3d1a8940UL, 0x01bc4000UL,
- 0x3fbc8858UL, 0xc65aacd3UL, 0x3d2646d1UL, 0x8dad6000UL, 0x3fbb6ac8UL,
- 0x2bf768e5UL, 0xbd139080UL, 0x40b1c000UL, 0x3fba4e76UL, 0xb94407c8UL,
- 0xbd0e42b6UL, 0x5d594000UL, 0x3fb9335eUL, 0x3abd47daUL, 0x3d23115cUL,
- 0x2f40e000UL, 0x3fb8197eUL, 0xf96ffdf7UL, 0x3d0f80dcUL, 0x0aeac000UL,
- 0x3fb700d3UL, 0xa99ded32UL, 0x3cec1e8dUL, 0x4d97a000UL, 0x3fb5e95aUL,
- 0x3c5d1d1eUL, 0xbd2c6906UL, 0x5d208000UL, 0x3fb4d311UL, 0x82f4e1efUL,
- 0xbcf53a25UL, 0xa7d1e000UL, 0x3fb3bdf5UL, 0xa5db4ed7UL, 0x3d2cc85eUL,
- 0xa4472000UL, 0x3fb2aa04UL, 0xae9c697dUL, 0xbd20b6e8UL, 0xd1466000UL,
- 0x3fb1973bUL, 0x560d9e9bUL, 0xbd25325dUL, 0xb59e4000UL, 0x3fb08598UL,
- 0x7009902cUL, 0xbd17e5ddUL, 0xc006c000UL, 0x3faeea31UL, 0x4fc93b7bUL,
- 0xbd0e113eUL, 0xcdddc000UL, 0x3faccb73UL, 0x47d82807UL, 0xbd1a68f2UL,
- 0xd0fb0000UL, 0x3faaaef2UL, 0x353bb42eUL, 0x3d20fc1aUL, 0x149fc000UL,
- 0x3fa894aaUL, 0xd05a267dUL, 0xbd197995UL, 0xf2d4c000UL, 0x3fa67c94UL,
- 0xec19afa2UL, 0xbd029efbUL, 0xd42e0000UL, 0x3fa466aeUL, 0x75bdfd28UL,
- 0xbd2c1673UL, 0x2f8d0000UL, 0x3fa252f3UL, 0xe021b67bUL, 0x3d283e9aUL,
- 0x89e74000UL, 0x3fa0415dUL, 0x5cf1d753UL, 0x3d0111c0UL, 0xec148000UL,
- 0x3f9c63d2UL, 0x3f9eb2f3UL, 0x3d2578c6UL, 0x28c90000UL, 0x3f984925UL,
- 0x325a0c34UL, 0xbd2aa0baUL, 0x25980000UL, 0x3f9432a9UL, 0x928637feUL,
- 0x3d098139UL, 0x58938000UL, 0x3f902056UL, 0x06e2f7d2UL, 0xbd23dc5bUL,
- 0xa3890000UL, 0x3f882448UL, 0xda74f640UL, 0xbd275577UL, 0x75890000UL,
- 0x3f801015UL, 0x999d2be8UL, 0xbd10c76bUL, 0x59580000UL, 0x3f700805UL,
- 0xcb31c67bUL, 0x3d2166afUL, 0x00000000UL, 0x00000000UL, 0x00000000UL,
- 0x80000000UL
-};
-
-// BEGIN dlog PSEUDO CODE:
-// double dlog(double X) {
-// // p(r) polynomial coefficients initialized from _L_tbl table
-// double C1_0 = _L_tbl[0];
-// double C1_1 = _L_tbl[1];
-// double C2_0 = _L_tbl[2];
-// double C2_1 = _L_tbl[3];
-// double C3_0 = _L_tbl[4];
-// double C3_1 = _L_tbl[5];
-// double C4_0 = _L_tbl[6];
-// double C4_1 = _L_tbl[7];
-// // NOTE: operations with coefficients above are mostly vectorized in assembly
-// // Check corner cases first
-// if (X == 1.0d || AS_LONG_BITS(X) + 0x0010000000000000 <= 0x0010000000000000) {
-// // NOTE: AS_LONG_BITS(X) + 0x0010000000000000 <= 0x0010000000000000 means
-// // that X < 0 or X >= 0x7FF0000000000000 (0x7FF* is NaN or INF)
-// if (X < 0 || X is NaN) return NaN;
-// if (X == 1.0d) return 0.0d;
-// if (X == 0.0d) return -INFINITY;
-// if (X is INFINITY) return INFINITY;
-// }
-// // double representation is 2^exponent * mantissa
-// // split X into two multipliers: 2^exponent and 1.0 * mantissa
-// // pseudo function: zeroExponent(X) return value of X with exponent == 0
-// float vtmp5 = 1/(float)(zeroExponent(X)); // reciprocal estimate
-// // pseudo function: HI16(X) returns high 16 bits of double value
-// int hiWord = HI16(X);
-// double vtmp1 = (double) 0x77F0 << 48 | mantissa(X);
-// hiWord -= 16;
-// if (AS_LONG_BITS(hiWord) > 0x8000) {
-// // SMALL_VALUE branch
-// vtmp0 = vtmp1 = vtmp0 * AS_DOUBLE_BITS(0x47F0000000000000);
-// hiWord = HI16(vtmp1);
-// vtmp0 = AS_DOUBLE_BITS(AS_LONG_BITS(vtmp0) |= 0x3FF0000000000000);
-// vtmp5 = (double) (1/(float)vtmp0);
-// vtmp1 <<= 12;
-// vtmp1 >>= 12;
-// }
-// // MAIN branch
-// double vtmp3 = AS_LONG_BITS(vtmp1) & 0xffffe00000000000; // hi part
-// int intB0 = AS_INT_BITS(vtmp5) + 0x8000;
-// double vtmp0 = AS_DOUBLE_BITS(0xffffe00000000000 & (intB0<<29));
-// int index = (intB0 >> 16) && 0xFF;
-// double hiTableValue = _L_tbl[8+index]; // vtmp2[0]
-// double lowTableValue = _L_tbl[16+index]; // vtmp2[1]
-// vtmp5 = AS_DOUBLE_BITS(hiWord & 0x7FF0 - 0x3FE0); // 0x3FE = 1023 << 4
-// vtmp1 -= vtmp3; // low part
-// vtmp3 = vtmp3*vtmp0 - 1.0;
-// hiTableValue += C4_0 * vtmp5;
-// lowTableValue += C4_1 * vtmp5;
-// double r = vtmp1 * vtmp0 + vtmp3; // r = B*mx-1.0, computed in hi and low parts
-// vtmp0 = hiTableValue + r;
-// hiTableValue -= vtmp0;
-// double r2 = r*r;
-// double r3 = r2*r;
-// double p7 = C3_0*r2 + C2_0*r3 + C1_0*r2*r2 + C3_1*r3*r2 + C2_1*r3*r3
-// + C1_1*r3*r2*r2; // degree 7 polynomial
-// return p7 + (vtmp0 + ((r + hiTableValue) + lowTableValue));
-// }
-//
-// END dlog PSEUDO CODE
-
-
-// Generate log(X). X passed in register v0. Return log(X) into v0.
-// Generator parameters: 10 temporary FPU registers and temporary general
-// purpose registers
-void MacroAssembler::fast_log(FloatRegister vtmp0, FloatRegister vtmp1,
- FloatRegister vtmp2, FloatRegister vtmp3,
- FloatRegister vtmp4, FloatRegister vtmp5,
- FloatRegister C1, FloatRegister C2,
- FloatRegister C3, FloatRegister C4,
- Register tmp1, Register tmp2, Register tmp3,
- Register tmp4, Register tmp5) {
- Label DONE, CHECK_CORNER_CASES, SMALL_VALUE, MAIN,
- CHECKED_CORNER_CASES, RETURN_MINF_OR_NAN;
- const int64_t INF_OR_NAN_PREFIX = 0x7FF0;
- const int64_t MINF_OR_MNAN_PREFIX = 0xFFF0;
- const int64_t ONE_PREFIX = 0x3FF0;
- movz(tmp2, ONE_PREFIX, 48);
- movz(tmp4, 0x0010, 48);
- fmovd(rscratch1, v0); // rscratch1 = AS_LONG_BITS(X)
- lea(rscratch2, ExternalAddress((address)_L_tbl));
- movz(tmp5, 0x7F);
- add(tmp1, rscratch1, tmp4);
- cmp(tmp2, rscratch1);
- lsr(tmp3, rscratch1, 29);
- ccmp(tmp1, tmp4, 0b1101 /* LE */, NE);
- bfm(tmp3, tmp5, 41, 8);
- fmovs(vtmp5, tmp3);
- // Load coefficients from table. All coefficients are organized to be
- // in specific order, because load below will load it in vectors to be used
- // later in vector instructions. Load will be performed in parallel while
- // branches are taken. C1 will contain vector of {C1_0, C1_1}, C2 =
- // {C2_0, C2_1}, C3 = {C3_0, C3_1}, C4 = {C4_0, C4_1}
- ld1(C1, C2, C3, C4, T2D, post(rscratch2, 64));
- br(LE, CHECK_CORNER_CASES);
- bind(CHECKED_CORNER_CASES);
- // all corner cases are handled
- frecpe(vtmp5, vtmp5, S); // vtmp5 ~= 1/vtmp5
- lsr(tmp2, rscratch1, 48);
- movz(tmp4, 0x77f0, 48);
- fmovd(vtmp4, 1.0);
- movz(tmp1, INF_OR_NAN_PREFIX, 48);
- bfm(tmp4, rscratch1, 0, 51); // tmp4 = 0x77F0 << 48 | mantissa(X)
- // vtmp1 = AS_DOUBLE_BITS(0x77F0 << 48 | mantissa(X)) == mx
- fmovd(vtmp1, tmp4);
- subw(tmp2, tmp2, 16);
- subs(zr, tmp2, 0x8000);
- br(GE, SMALL_VALUE);
- bind(MAIN);
- fmovs(tmp3, vtmp5); // int intB0 = AS_INT_BITS(B);
- mov(tmp5, 0x3FE0);
- uint64_t mask = UCONST64(0xffffe00000000000);
- mov(rscratch1, mask);
- andr(tmp2, tmp2, tmp1, LSR, 48); // hiWord & 0x7FF0
- sub(tmp2, tmp2, tmp5); // tmp2 = hiWord & 0x7FF0 - 0x3FE0
- scvtfwd(vtmp5, tmp2); // vtmp5 = (double)tmp2;
- addw(tmp3, tmp3, 0x8000); // tmp3 = B
- andr(tmp4, tmp4, rscratch1); // tmp4 == hi_part(mx)
- andr(rscratch1, rscratch1, tmp3, LSL, 29); // rscratch1 = hi_part(B)
- ubfm(tmp3, tmp3, 16, 23); // int index = (intB0 >> 16) && 0xFF
- ldrq(vtmp2, Address(rscratch2, tmp3, Address::lsl(4))); // vtmp2 = _L_tbl[index]
- // AS_LONG_BITS(vtmp1) & 0xffffe00000000000 // hi_part(mx)
- fmovd(vtmp3, tmp4);
- fmovd(vtmp0, rscratch1); // vtmp0 = hi_part(B)
- fsubd(vtmp1, vtmp1, vtmp3); // vtmp1 -= vtmp3; // low_part(mx)
- fnmsub(vtmp3, vtmp3, vtmp0, vtmp4); // vtmp3 = vtmp3*vtmp0 - vtmp4
- fmlavs(vtmp2, T2D, C4, vtmp5, 0); // vtmp2 += {C4} * vtmp5
- // vtmp1 = r = vtmp1 * vtmp0 + vtmp3 == low_part(mx) * hi_part(B) + (hi_part(mx)*hi_part(B) - 1.0)
- fmaddd(vtmp1, vtmp1, vtmp0, vtmp3);
- ins(vtmp5, D, vtmp2, 0, 1); // vtmp5 = vtmp2[1];
- faddd(vtmp0, vtmp2, vtmp1); // vtmp0 = vtmp2 + vtmp1
- fmlavs(C3, T2D, C2, vtmp1, 0); // {C3} += {C2}*vtmp1
- fsubd(vtmp2, vtmp2, vtmp0); // vtmp2 -= vtmp0
- fmuld(vtmp3, vtmp1, vtmp1); // vtmp3 = vtmp1*vtmp1
- faddd(C4, vtmp1, vtmp2); // C4[0] = vtmp1 + vtmp2
- fmlavs(C3, T2D, C1, vtmp3, 0); // {C3} += {C1}*vtmp3
- faddd(C4, C4, vtmp5); // C4 += vtmp5
- fmuld(vtmp4, vtmp3, vtmp1); // vtmp4 = vtmp3*vtmp1
- faddd(vtmp0, vtmp0, C4); // vtmp0 += C4
- fmlavs(C3, T2D, vtmp4, C3, 1); // {C3} += {vtmp4}*C3[1]
- fmaddd(vtmp0, C3, vtmp3, vtmp0); // vtmp0 = C3 * vtmp3 + vtmp0
- ret(lr);
-
- block_comment("if (AS_LONG_BITS(hiWord) > 0x8000)"); {
- bind(SMALL_VALUE);
- movz(tmp2, 0x47F0, 48);
- fmovd(vtmp1, tmp2);
- fmuld(vtmp0, vtmp1, v0);
- fmovd(vtmp1, vtmp0);
- umov(tmp2, vtmp1, S, 3);
- orr(vtmp0, T16B, vtmp0, vtmp4);
- ushr(vtmp5, T2D, vtmp0, 27);
- ushr(vtmp5, T4S, vtmp5, 2);
- frecpe(vtmp5, vtmp5, S);
- shl(vtmp1, T2D, vtmp1, 12);
- ushr(vtmp1, T2D, vtmp1, 12);
- b(MAIN);
- }
-
- block_comment("Corner cases"); {
- bind(RETURN_MINF_OR_NAN);
- movz(tmp1, MINF_OR_MNAN_PREFIX, 48);
- orr(rscratch1, rscratch1, tmp1);
- fmovd(v0, rscratch1);
- ret(lr);
- bind(CHECK_CORNER_CASES);
- movz(tmp1, INF_OR_NAN_PREFIX, 48);
- cmp(rscratch1, zr);
- br(LE, RETURN_MINF_OR_NAN);
- cmp(rscratch1, tmp1);
- br(GE, DONE);
- cmp(rscratch1, tmp2);
- br(NE, CHECKED_CORNER_CASES);
- fmovd(v0, 0.0);
- }
- bind(DONE);
- ret(lr);
-}
diff --git a/src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp b/src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp
index 8694734c751..216c1ff3509 100644
--- a/src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/sharedRuntime_aarch64.cpp
@@ -30,7 +30,6 @@
#include "code/codeCache.hpp"
#include "code/compiledIC.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
@@ -39,7 +38,6 @@
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
#include "nativeInst_aarch64.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
#include "prims/methodHandles.hpp"
@@ -740,9 +738,7 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
address c2i_unverified_entry = __ pc();
Label skip_fixup;
- Label ok;
-
- Register holder = rscratch2;
+ Register data = rscratch2;
Register receiver = j_rarg0;
Register tmp = r10; // A call-clobbered register not used for arg passing
@@ -757,17 +753,12 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
{
__ block_comment("c2i_unverified_entry {");
- __ load_klass(rscratch1, receiver);
- __ ldr(tmp, Address(holder, CompiledICHolder::holder_klass_offset()));
- __ cmp(rscratch1, tmp);
- __ ldr(rmethod, Address(holder, CompiledICHolder::holder_metadata_offset()));
- __ br(Assembler::EQ, ok);
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- __ bind(ok);
// Method might have been compiled since the call site was patched to
// interpreted; if that is the case treat it as a miss so we can get
// the call site corrected.
+ __ ic_check(1 /* end_alignment */);
+ __ ldr(rmethod, Address(data, CompiledICData::speculated_method_offset()));
+
__ ldr(rscratch1, Address(rmethod, in_bytes(Method::code_offset())));
__ cbz(rscratch1, skip_fixup);
__ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
@@ -1118,7 +1109,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
__ b(exit);
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, tr_call);
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, tr_call);
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1183,7 +1174,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
}
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, tr_call);
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, tr_call);
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1391,6 +1382,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
in_ByteSize(-1),
oop_maps,
exception_offset);
+ if (nm == nullptr) return nm;
if (method->is_continuation_enter_intrinsic()) {
ContinuationEntry::set_enter_code(nm, interpreted_entry_offset);
} else if (method->is_continuation_yield_intrinsic()) {
@@ -1538,25 +1530,15 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
// restoring them except rfp. rfp is the only callee save register
// as far as the interpreter and the compiler(s) are concerned.
-
- const Register ic_reg = rscratch2;
const Register receiver = j_rarg0;
- Label hit;
Label exception_pending;
- assert_different_registers(ic_reg, receiver, rscratch1);
+ assert_different_registers(receiver, rscratch1);
__ verify_oop(receiver);
- __ cmp_klass(receiver, ic_reg, rscratch1);
- __ br(Assembler::EQ, hit);
-
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
+ __ ic_check(8 /* end_alignment */);
// Verified entry point must be aligned
- __ align(8);
-
- __ bind(hit);
-
int vep_offset = ((intptr_t)__ pc()) - start;
// If we have to make this method not-entrant we'll overwrite its
diff --git a/src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp b/src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp
index 97ca90ac764..46a7d796267 100644
--- a/src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/stubGenerator_aarch64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, 2022, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -5327,19 +5327,6 @@ class StubGenerator: public StubCodeGenerator {
return start;
}
- address generate_dlog() {
- __ align(CodeEntryAlignment);
- StubCodeMark mark(this, "StubRoutines", "dlog");
- address entry = __ pc();
- FloatRegister vtmp0 = v0, vtmp1 = v1, vtmp2 = v2, vtmp3 = v3, vtmp4 = v4,
- vtmp5 = v5, tmpC1 = v16, tmpC2 = v17, tmpC3 = v18, tmpC4 = v19;
- Register tmp1 = r0, tmp2 = r1, tmp3 = r2, tmp4 = r3, tmp5 = r4;
- __ fast_log(vtmp0, vtmp1, vtmp2, vtmp3, vtmp4, vtmp5, tmpC1, tmpC2, tmpC3,
- tmpC4, tmp1, tmp2, tmp3, tmp4, tmp5);
- return entry;
- }
-
-
// code for comparing 16 characters of strings with Latin1 and Utf16 encoding
void compare_string_16_x_LU(Register tmpL, Register tmpU, Label &DIFF1,
Label &DIFF2) {
@@ -5487,6 +5474,32 @@ class StubGenerator: public StubCodeGenerator {
return entry;
}
+ // r0 = input (float16)
+ // v0 = result (float)
+ // v1 = temporary float register
+ address generate_float16ToFloat() {
+ __ align(CodeEntryAlignment);
+ StubCodeMark mark(this, "StubRoutines", "float16ToFloat");
+ address entry = __ pc();
+ BLOCK_COMMENT("Entry:");
+ __ flt16_to_flt(v0, r0, v1);
+ __ ret(lr);
+ return entry;
+ }
+
+ // v0 = input (float)
+ // r0 = result (float16)
+ // v1 = temporary float register
+ address generate_floatToFloat16() {
+ __ align(CodeEntryAlignment);
+ StubCodeMark mark(this, "StubRoutines", "floatToFloat16");
+ address entry = __ pc();
+ BLOCK_COMMENT("Entry:");
+ __ flt_to_flt16(r0, v0, v1);
+ __ ret(lr);
+ return entry;
+ }
+
address generate_method_entry_barrier() {
__ align(CodeEntryAlignment);
StubCodeMark mark(this, "StubRoutines", "nmethod_entry_barrier");
@@ -8333,11 +8346,6 @@ class StubGenerator: public StubCodeGenerator {
StubRoutines::_updateBytesCRC32C = generate_updateBytesCRC32C();
}
- // Disabled until JDK-8210858 is fixed
- // if (vmIntrinsics::is_intrinsic_available(vmIntrinsics::_dlog)) {
- // StubRoutines::_dlog = generate_dlog();
- // }
-
if (vmIntrinsics::is_intrinsic_available(vmIntrinsics::_dsin)) {
StubRoutines::_dsin = generate_dsin_dcos(/* isCos = */ false);
}
@@ -8345,6 +8353,12 @@ class StubGenerator: public StubCodeGenerator {
if (vmIntrinsics::is_intrinsic_available(vmIntrinsics::_dcos)) {
StubRoutines::_dcos = generate_dsin_dcos(/* isCos = */ true);
}
+
+ if (vmIntrinsics::is_intrinsic_available(vmIntrinsics::_float16ToFloat) &&
+ vmIntrinsics::is_intrinsic_available(vmIntrinsics::_floatToFloat16)) {
+ StubRoutines::_hf2f = generate_float16ToFloat();
+ StubRoutines::_f2hf = generate_floatToFloat16();
+ }
}
void generate_continuation_stubs() {
diff --git a/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp b/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp
index f7fe2f7dec8..18f310c746c 100644
--- a/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp
@@ -143,11 +143,19 @@ void VM_Version::initialize() {
}
}
- // Ampere CPUs: Ampere-1 and Ampere-1A
- if (_cpu == CPU_AMPERE && ((_model == CPU_MODEL_AMPERE_1) || (_model == CPU_MODEL_AMPERE_1A))) {
+ // Ampere CPUs
+ if (_cpu == CPU_AMPERE && ((_model == CPU_MODEL_AMPERE_1) ||
+ (_model == CPU_MODEL_AMPERE_1A) ||
+ (_model == CPU_MODEL_AMPERE_1B))) {
if (FLAG_IS_DEFAULT(UseSIMDForMemoryOps)) {
FLAG_SET_DEFAULT(UseSIMDForMemoryOps, true);
}
+ if (FLAG_IS_DEFAULT(OnSpinWaitInst)) {
+ FLAG_SET_DEFAULT(OnSpinWaitInst, "isb");
+ }
+ if (FLAG_IS_DEFAULT(OnSpinWaitInstCount)) {
+ FLAG_SET_DEFAULT(OnSpinWaitInstCount, 2);
+ }
}
// ThunderX
diff --git a/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp b/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp
index 4b2e5cc5a4d..0a85d339a55 100644
--- a/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp
+++ b/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp
@@ -110,7 +110,8 @@ enum Ampere_CPU_Model {
CPU_MODEL_ALTRA = 0xd0c, /* CPU implementer is CPU_ARM, Neoverse N1 */
CPU_MODEL_ALTRAMAX = 0xd0c, /* CPU implementer is CPU_ARM, Neoverse N1 */
CPU_MODEL_AMPERE_1 = 0xac3, /* CPU implementer is CPU_AMPERE */
- CPU_MODEL_AMPERE_1A = 0xac4 /* CPU implementer is CPU_AMPERE */
+ CPU_MODEL_AMPERE_1A = 0xac4, /* CPU implementer is CPU_AMPERE */
+ CPU_MODEL_AMPERE_1B = 0xac5 /* AMPERE_1B core Implements ARMv8.7 with CSSC, MTE, SM3/SM4 extensions */
};
#define CPU_FEATURE_FLAGS(decl) \
diff --git a/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp b/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp
index c895ff5cc0e..2bb53d16a3c 100644
--- a/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp
+++ b/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp
@@ -26,10 +26,10 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_aarch64.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klassVtable.hpp"
#include "runtime/sharedRuntime.hpp"
@@ -168,22 +168,22 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
assert(VtableStub::receiver_location() == j_rarg0->as_VMReg(), "receiver expected in j_rarg0");
// Entry arguments:
- // rscratch2: CompiledICHolder
+ // rscratch2: CompiledICData
// j_rarg0: Receiver
// This stub is called from compiled code which has no callee-saved registers,
// so all registers except arguments are free at this point.
const Register recv_klass_reg = r10;
- const Register holder_klass_reg = r16; // declaring interface klass (DECC)
+ const Register holder_klass_reg = r16; // declaring interface klass (DEFC)
const Register resolved_klass_reg = r17; // resolved interface klass (REFC)
const Register temp_reg = r11;
const Register temp_reg2 = r15;
- const Register icholder_reg = rscratch2;
+ const Register icdata_reg = rscratch2;
Label L_no_such_interface;
- __ ldr(resolved_klass_reg, Address(icholder_reg, CompiledICHolder::holder_klass_offset()));
- __ ldr(holder_klass_reg, Address(icholder_reg, CompiledICHolder::holder_metadata_offset()));
+ __ ldr(resolved_klass_reg, Address(icdata_reg, CompiledICData::itable_refc_klass_offset()));
+ __ ldr(holder_klass_reg, Address(icdata_reg, CompiledICData::itable_defc_klass_offset()));
start_pc = __ pc();
diff --git a/src/hotspot/cpu/arm/arm.ad b/src/hotspot/cpu/arm/arm.ad
index e31ad91613a..1a833b08c4c 100644
--- a/src/hotspot/cpu/arm/arm.ad
+++ b/src/hotspot/cpu/arm/arm.ad
@@ -195,7 +195,7 @@ void emit_call_reloc(CodeBuffer &cbuf, const MachCallNode *n, MachOper *m, Reloc
assert(maybe_far_call(n) == !__ reachable_from_cache(target), "sanity");
assert(cache_reachable() == __ cache_fully_reachable(), "sanity");
- assert(target != NULL, "need real address");
+ assert(target != nullptr, "need real address");
int ret_addr_offset = -1;
if (rspec.type() == relocInfo::runtime_call_type) {
@@ -290,7 +290,7 @@ void MachPrologNode::format( PhaseRegAlloc *ra_, outputStream *st ) const {
st->print ("SUB R_SP, R_SP, " SIZE_FORMAT,framesize);
}
- if (C->stub_function() == NULL && BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (C->stub_function() == nullptr && BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
st->print("ldr t0, [guard]\n\t");
st->print("ldr t1, [Rthread, #thread_disarmed_guard_value_offset]\n\t");
st->print("cmp t0, t1\n\t");
@@ -332,7 +332,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ sub_slow(SP, SP, framesize);
}
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
bs->nmethod_entry_barrier(&_masm);
}
@@ -454,7 +454,7 @@ uint MachSpillCopyNode::implementation( CodeBuffer *cbuf,
return size; // Self copy, no move
#ifdef TODO
- if (bottom_type()->isa_vect() != NULL) {
+ if (bottom_type()->isa_vect() != nullptr) {
}
#endif
@@ -804,16 +804,16 @@ uint MachSpillCopyNode::implementation( CodeBuffer *cbuf,
#ifndef PRODUCT
void MachSpillCopyNode::format( PhaseRegAlloc *ra_, outputStream *st ) const {
- implementation( NULL, ra_, false, st );
+ implementation(nullptr, ra_, false, st );
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation( &cbuf, ra_, false, NULL );
+ implementation( &cbuf, ra_, false, nullptr );
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
- return implementation( NULL, ra_, true, NULL );
+ return implementation( nullptr, ra_, true, nullptr );
}
//=============================================================================
@@ -869,12 +869,7 @@ uint BoxLockNode::size(PhaseRegAlloc *ra_) const {
#define R_RTEMP "R_R12"
void MachUEPNode::format( PhaseRegAlloc *ra_, outputStream *st ) const {
st->print_cr("\nUEP:");
- if (UseCompressedClassPointers) {
- st->print_cr("\tLDR_w " R_RTEMP ",[R_R0 + oopDesc::klass_offset_in_bytes]\t! Inline cache check");
- st->print_cr("\tdecode_klass " R_RTEMP);
- } else {
- st->print_cr("\tLDR " R_RTEMP ",[R_R0 + oopDesc::klass_offset_in_bytes]\t! Inline cache check");
- }
+ st->print_cr("\tLDR " R_RTEMP ",[R_R0 + oopDesc::klass_offset_in_bytes]\t! Inline cache check");
st->print_cr("\tCMP " R_RTEMP ",R_R8" );
st->print ("\tB.NE SharedRuntime::handle_ic_miss_stub");
}
@@ -882,13 +877,7 @@ void MachUEPNode::format( PhaseRegAlloc *ra_, outputStream *st ) const {
void MachUEPNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
C2_MacroAssembler _masm(&cbuf);
- Register iCache = reg_to_register_object(Matcher::inline_cache_reg_encode());
- assert(iCache == Ricklass, "should be");
- Register receiver = R0;
-
- __ load_klass(Rtemp, receiver);
- __ cmp(Rtemp, iCache);
- __ jump(SharedRuntime::get_ic_miss_stub(), relocInfo::runtime_call_type, noreg, ne);
+ __ ic_check(InteriorEntryAlignment);
}
uint MachUEPNode::size(PhaseRegAlloc *ra_) const {
@@ -903,7 +892,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer& cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -926,7 +915,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -1002,7 +991,7 @@ bool Matcher::match_rule_supported(int opcode) {
return true; // Per default match rules are supported.
}
-bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
return match_rule_supported_vector(opcode, vlen, bt);
}
@@ -1026,11 +1015,11 @@ bool Matcher::vector_needs_partial_operations(Node* node, const TypeVect* vt) {
}
const RegMask* Matcher::predicate_reg_mask(void) {
- return NULL;
+ return nullptr;
}
const TypeVectMask* Matcher::predicate_reg_type(const Type* elemTy, int length) {
- return NULL;
+ return nullptr;
}
// Vector calling convention not yet implemented.
@@ -1074,7 +1063,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return 8/type2aelembytes(bt);
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
return Matcher::max_vector_size(bt);
}
@@ -1094,7 +1083,7 @@ bool Matcher::is_short_branch_offset(int rule, int br_size, int offset) {
MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* original_opnd, uint ideal_reg, bool is_temp) {
ShouldNotReachHere(); // generic vector operands not supported
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -1241,8 +1230,8 @@ encode %{
emit_call_reloc(cbuf, as_MachCall(), $meth, rspec);
// Emit stubs for static call.
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -1987,7 +1976,7 @@ operand immNKlass()
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immN0()
%{
predicate(n->get_narrowcon() == 0);
diff --git a/src/hotspot/cpu/arm/arm_32.ad b/src/hotspot/cpu/arm/arm_32.ad
index affe5a816fc..dd7d6f491da 100644
--- a/src/hotspot/cpu/arm/arm_32.ad
+++ b/src/hotspot/cpu/arm/arm_32.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2008, 2022, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2008, 2024, Oracle and/or its affiliates. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
// This code is free software; you can redistribute it and/or modify it
@@ -431,7 +431,7 @@ OptoRegPair c2::return_value(int ideal_reg) {
// will point.
int MachCallStaticJavaNode::ret_addr_offset() {
- bool far = (_method == NULL) ? maybe_far_call(this) : !cache_reachable();
+ bool far = (_method == nullptr) ? maybe_far_call(this) : !cache_reachable();
return ((far ? 3 : 1) + (_method_handle_invoke ? 1 : 0)) *
NativeInstruction::instruction_size;
}
diff --git a/src/hotspot/cpu/arm/c1_LIRAssembler_arm.cpp b/src/hotspot/cpu/arm/c1_LIRAssembler_arm.cpp
index 999309c0225..16aeaa20c04 100644
--- a/src/hotspot/cpu/arm/c1_LIRAssembler_arm.cpp
+++ b/src/hotspot/cpu/arm/c1_LIRAssembler_arm.cpp
@@ -161,10 +161,7 @@ void LIR_Assembler::osr_entry() {
int LIR_Assembler::check_icache() {
- Register receiver = LIR_Assembler::receiverOpr()->as_register();
- int offset = __ offset();
- __ inline_cache_check(receiver, Ricklass);
- return offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::clinit_barrier(ciMethod* method) {
@@ -1950,7 +1947,7 @@ void LIR_Assembler::emit_static_call_stub() {
__ relocate(static_stub_Relocation::spec(call_pc));
// If not a single instruction, NativeMovConstReg::next_instruction_address()
// must jump over the whole following ldr_literal.
- // (See CompiledStaticCall::set_to_interpreted())
+ // (See CompiledDirectCall::set_to_interpreted())
#ifdef ASSERT
address ldr_site = __ pc();
#endif
diff --git a/src/hotspot/cpu/arm/c1_MacroAssembler_arm.cpp b/src/hotspot/cpu/arm/c1_MacroAssembler_arm.cpp
index c09e54e0e57..d9d042bb2e4 100644
--- a/src/hotspot/cpu/arm/c1_MacroAssembler_arm.cpp
+++ b/src/hotspot/cpu/arm/c1_MacroAssembler_arm.cpp
@@ -43,16 +43,6 @@
// arm [macro]assembler) and used with care in the other C1 specific
// files.
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache) {
- Label verified;
- load_klass(Rtemp, receiver);
- cmp(Rtemp, iCache);
- b(verified, eq); // jump over alignment no-ops
- jump(SharedRuntime::get_ic_miss_stub(), relocInfo::runtime_call_type);
- align(CodeEntryAlignment);
- bind(verified);
-}
-
void C1_MacroAssembler::build_frame(int frame_size_in_bytes, int bang_size_in_bytes) {
assert(bang_size_in_bytes >= frame_size_in_bytes, "stack bang size incorrect");
assert((frame_size_in_bytes % StackAlignmentInBytes) == 0, "frame size should be aligned");
diff --git a/src/hotspot/cpu/arm/c1_Runtime1_arm.cpp b/src/hotspot/cpu/arm/c1_Runtime1_arm.cpp
index 62faa617083..9862a074a68 100644
--- a/src/hotspot/cpu/arm/c1_Runtime1_arm.cpp
+++ b/src/hotspot/cpu/arm/c1_Runtime1_arm.cpp
@@ -37,7 +37,6 @@
#include "interpreter/interpreter.hpp"
#include "memory/universe.hpp"
#include "nativeInst_arm.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_arm.hpp"
diff --git a/src/hotspot/cpu/arm/compiledIC_arm.cpp b/src/hotspot/cpu/arm/compiledIC_arm.cpp
index 2d4187b7d6c..71389d2353d 100644
--- a/src/hotspot/cpu/arm/compiledIC_arm.cpp
+++ b/src/hotspot/cpu/arm/compiledIC_arm.cpp
@@ -25,7 +25,6 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/nmethod.hpp"
#include "logging/log.hpp"
@@ -37,7 +36,7 @@
#if COMPILER2_OR_JVMCI
#define __ _masm.
// emit call stub, compiled java to interpreter
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
// Stub is fixed up when the corresponding call is converted from calling
// compiled code to calling interpreted code.
// set (empty), R9
@@ -59,7 +58,7 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark)
InlinedMetadata object_literal(nullptr);
// single instruction, see NativeMovConstReg::next_instruction_address() in
- // CompiledStaticCall::set_to_interpreted()
+ // CompiledDirectCall::set_to_interpreted()
__ ldr_literal(Rmethod, object_literal);
__ set_inst_mark(); // Who uses this?
@@ -87,32 +86,25 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark)
#undef __
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 10; // 4 in emit_to_interp_stub + 1 in Java_Static_Call
}
#endif // COMPILER2_OR_JVMCI
-int CompiledStaticCall::to_trampoline_stub_size() {
+int CompiledDirectCall::to_trampoline_stub_size() {
// ARM doesn't use trampolines.
return 0;
}
// size of C2 call stub, compiled java to interpreter
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return 8 * NativeInstruction::instruction_size;
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder = nativeMovConstReg_at(stub);
NativeJump* jump = nativeJump_at(method_holder->next_instruction_address());
@@ -128,7 +120,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
// Reset stub.
address stub = static_stub->addr();
assert(stub != nullptr, "stub not found");
@@ -144,7 +136,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/arm/icBuffer_arm.cpp b/src/hotspot/cpu/arm/icBuffer_arm.cpp
deleted file mode 100644
index e3a1c148ec6..00000000000
--- a/src/hotspot/cpu/arm/icBuffer_arm.cpp
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2008, 2016, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/assembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_arm.hpp"
-#include "oops/oop.inline.hpp"
-
-#define __ masm->
-
-int InlineCacheBuffer::ic_stub_code_size() {
- return (4 * Assembler::InstructionSize);
-}
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point) {
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler* masm = new MacroAssembler(&code);
-
- InlinedAddress oop_literal((address) cached_value);
- __ ldr_literal(Ricklass, oop_literal);
- // FIXME: OK to remove reloc here?
- __ patchable_jump(entry_point, relocInfo::runtime_call_type, Rtemp);
- __ bind_literal(oop_literal);
- __ flush();
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- address jump_address;
- jump_address = code_begin + NativeInstruction::instruction_size;
- NativeJump* jump = nativeJump_at(jump_address);
- return jump->jump_destination();
-}
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin);
- return (void*)move->data();
-}
-
-#undef __
diff --git a/src/hotspot/cpu/arm/macroAssembler_arm.cpp b/src/hotspot/cpu/arm/macroAssembler_arm.cpp
index b827e69d022..99d619bddb5 100644
--- a/src/hotspot/cpu/arm/macroAssembler_arm.cpp
+++ b/src/hotspot/cpu/arm/macroAssembler_arm.cpp
@@ -28,6 +28,7 @@
#include "asm/assembler.inline.hpp"
#include "asm/macroAssembler.hpp"
#include "ci/ciEnv.hpp"
+#include "code/compiledIC.hpp"
#include "code/nativeInst.hpp"
#include "compiler/disassembler.hpp"
#include "gc/shared/barrierSet.hpp"
@@ -297,11 +298,13 @@ Address MacroAssembler::receiver_argument_address(Register params_base, Register
return Address(tmp, -Interpreter::stackElementSize);
}
+void MacroAssembler::align(int modulus, int target) {
+ int delta = target - offset();
+ while ((offset() + delta) % modulus != 0) nop();
+}
void MacroAssembler::align(int modulus) {
- while (offset() % modulus != 0) {
- nop();
- }
+ align(modulus, offset());
}
int MacroAssembler::set_last_Java_frame(Register last_java_sp,
@@ -1860,3 +1863,31 @@ void MacroAssembler::lightweight_unlock(Register obj, Register t1, Register t2,
// Fallthrough: success
}
+
+int MacroAssembler::ic_check_size() {
+ return NativeInstruction::instruction_size * 7;
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ Register receiver = j_rarg0;
+ Register tmp1 = R4;
+ Register tmp2 = R5;
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, offset() + ic_check_size());
+
+ int uep_offset = offset();
+
+ ldr(tmp1, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ ldr(tmp2, Address(Ricklass, CompiledICData::speculated_klass_offset()));
+ cmp(tmp1, tmp2);
+
+ Label dont;
+ b(dont, eq);
+ jump(SharedRuntime::get_ic_miss_stub(), relocInfo::runtime_call_type);
+ bind(dont);
+ return uep_offset;
+}
diff --git a/src/hotspot/cpu/arm/macroAssembler_arm.hpp b/src/hotspot/cpu/arm/macroAssembler_arm.hpp
index d9e49ab986c..691c8fa70ee 100644
--- a/src/hotspot/cpu/arm/macroAssembler_arm.hpp
+++ b/src/hotspot/cpu/arm/macroAssembler_arm.hpp
@@ -221,6 +221,7 @@ class MacroAssembler: public Assembler {
inline bool ignore_non_patchable_relocations() { return true; }
void align(int modulus);
+ void align(int modulus, int target);
// Support for VM calls
//
@@ -1077,6 +1078,9 @@ class MacroAssembler: public Assembler {
void safepoint_poll(Register tmp1, Label& slow_path);
void get_polling_page(Register dest);
void read_polling_page(Register dest, relocInfo::relocType rtype);
+
+ static int ic_check_size();
+ int ic_check(int end_alignment);
};
diff --git a/src/hotspot/cpu/arm/nativeInst_arm_32.cpp b/src/hotspot/cpu/arm/nativeInst_arm_32.cpp
index 23ee01d3352..6a4062f29b3 100644
--- a/src/hotspot/cpu/arm/nativeInst_arm_32.cpp
+++ b/src/hotspot/cpu/arm/nativeInst_arm_32.cpp
@@ -25,7 +25,6 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "memory/resourceArea.hpp"
#include "nativeInst_arm.hpp"
#include "oops/oop.inline.hpp"
diff --git a/src/hotspot/cpu/arm/nativeInst_arm_32.hpp b/src/hotspot/cpu/arm/nativeInst_arm_32.hpp
index 7006d770981..15b57188730 100644
--- a/src/hotspot/cpu/arm/nativeInst_arm_32.hpp
+++ b/src/hotspot/cpu/arm/nativeInst_arm_32.hpp
@@ -385,7 +385,7 @@ class NativeMovConstReg: public NativeInstruction {
}
void set_pc_relative_offset(address addr, address pc);
address next_instruction_address() const {
- // NOTE: CompiledStaticCall::set_to_interpreted() calls this but
+ // NOTE: CompiledDirectCall::set_to_interpreted() calls this but
// are restricted to single-instruction ldr. No need to jump over
// several instructions.
assert(is_ldr_literal(), "Should only use single-instructions load");
diff --git a/src/hotspot/cpu/arm/sharedRuntime_arm.cpp b/src/hotspot/cpu/arm/sharedRuntime_arm.cpp
index 716c7b7575e..3792fab082b 100644
--- a/src/hotspot/cpu/arm/sharedRuntime_arm.cpp
+++ b/src/hotspot/cpu/arm/sharedRuntime_arm.cpp
@@ -24,15 +24,14 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
#include "interpreter/interpreter.hpp"
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "prims/methodHandles.hpp"
#include "runtime/jniHandles.hpp"
@@ -626,12 +625,9 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
Label skip_fixup;
const Register receiver = R0;
const Register holder_klass = Rtemp; // XXX should be OK for C2 but not 100% sure
- const Register receiver_klass = R4;
- __ load_klass(receiver_klass, receiver);
- __ ldr(holder_klass, Address(Ricklass, CompiledICHolder::holder_klass_offset()));
- __ ldr(Rmethod, Address(Ricklass, CompiledICHolder::holder_metadata_offset()));
- __ cmp(receiver_klass, holder_klass);
+ __ ic_check(1 /* end_alignment */);
+ __ ldr(Rmethod, Address(Ricklass, CompiledICData::speculated_method_offset()));
__ ldr(Rtemp, Address(Rmethod, Method::code_offset()), eq);
__ cmp(Rtemp, 0, eq);
@@ -819,21 +815,14 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
// Unverified entry point
address start = __ pc();
- // Inline cache check, same as in C1_MacroAssembler::inline_cache_check()
const Register receiver = R0; // see receiverOpr()
- __ load_klass(Rtemp, receiver);
- __ cmp(Rtemp, Ricklass);
- Label verified;
-
- __ b(verified, eq); // jump over alignment no-ops too
- __ jump(SharedRuntime::get_ic_miss_stub(), relocInfo::runtime_call_type, Rtemp);
- __ align(CodeEntryAlignment);
+ __ verify_oop(receiver);
+ // Inline cache check
+ __ ic_check(CodeEntryAlignment /* end_alignment */);
// Verified entry point
- __ bind(verified);
int vep_offset = __ pc() - start;
-
if ((InlineObjectHash && method->intrinsic_id() == vmIntrinsics::_hashCode) || (method->intrinsic_id() == vmIntrinsics::_identityHashCode)) {
// Object.hashCode, System.identityHashCode can pull the hashCode from the header word
// instead of doing a full VM transition once it's been computed.
diff --git a/src/hotspot/cpu/arm/vtableStubs_arm.cpp b/src/hotspot/cpu/arm/vtableStubs_arm.cpp
index 539e288f63f..1229b5073f5 100644
--- a/src/hotspot/cpu/arm/vtableStubs_arm.cpp
+++ b/src/hotspot/cpu/arm/vtableStubs_arm.cpp
@@ -25,10 +25,10 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_arm.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klassVtable.hpp"
#include "oops/klass.inline.hpp"
@@ -160,7 +160,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
__ load_klass(Rclass, R0);
// Receiver subtype check against REFC.
- __ ldr(Rintf, Address(Ricklass, CompiledICHolder::holder_klass_offset()));
+ __ ldr(Rintf, Address(Ricklass, CompiledICData::itable_refc_klass_offset()));
__ lookup_interface_method(// inputs: rec. class, interface, itable index
Rclass, Rintf, noreg,
// outputs: temp reg1, temp reg2
@@ -171,7 +171,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
start_pc = __ pc();
// Get Method* and entry point for compiler
- __ ldr(Rintf, Address(Ricklass, CompiledICHolder::holder_metadata_offset()));
+ __ ldr(Rintf, Address(Ricklass, CompiledICData::itable_defc_klass_offset()));
__ lookup_interface_method(// inputs: rec. class, interface, itable index
Rclass, Rintf, itable_index,
// outputs: temp reg1, temp reg2, temp reg3
diff --git a/src/hotspot/cpu/ppc/assembler_ppc.inline.hpp b/src/hotspot/cpu/ppc/assembler_ppc.inline.hpp
index 47b681ce26b..d78dec964cb 100644
--- a/src/hotspot/cpu/ppc/assembler_ppc.inline.hpp
+++ b/src/hotspot/cpu/ppc/assembler_ppc.inline.hpp
@@ -451,7 +451,7 @@ inline void Assembler::bcctrl(int boint, int biint, int bhint, relocInfo::relocT
// helper function for b
inline bool Assembler::is_within_range_of_b(address a, address pc) {
- // Guard against illegal branch targets, e.g. -1 (see CompiledStaticCall and ad-file).
+ // Guard against illegal branch targets, e.g. -1 (see CompiledDirectCall and ad-file).
if ((((uint64_t)a) & 0x3) != 0) return false;
const int range = 1 << (29-6); // li field is from bit 6 to bit 29.
@@ -465,7 +465,7 @@ inline bool Assembler::is_within_range_of_b(address a, address pc) {
// helper functions for bcxx.
inline bool Assembler::is_within_range_of_bcxx(address a, address pc) {
- // Guard against illegal branch targets, e.g. -1 (see CompiledStaticCall and ad-file).
+ // Guard against illegal branch targets, e.g. -1 (see CompiledDirectCall and ad-file).
if ((((uint64_t)a) & 0x3) != 0) return false;
const int range = 1 << (29-16); // bd field is from bit 16 to bit 29.
diff --git a/src/hotspot/cpu/ppc/c1_LIRAssembler_ppc.cpp b/src/hotspot/cpu/ppc/c1_LIRAssembler_ppc.cpp
index d316c2b3db2..4b29bcf57e4 100644
--- a/src/hotspot/cpu/ppc/c1_LIRAssembler_ppc.cpp
+++ b/src/hotspot/cpu/ppc/c1_LIRAssembler_ppc.cpp
@@ -77,9 +77,7 @@ int LIR_Assembler::initial_frame_size_in_bytes() const {
// we fetch the class of the receiver and compare it with the cached class.
// If they do not match we jump to slow case.
int LIR_Assembler::check_icache() {
- int offset = __ offset();
- __ inline_cache_check(R3_ARG1, R19_inline_cache_reg);
- return offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::clinit_barrier(ciMethod* method) {
diff --git a/src/hotspot/cpu/ppc/c1_MacroAssembler_ppc.cpp b/src/hotspot/cpu/ppc/c1_MacroAssembler_ppc.cpp
index 577dcae25f4..b379d4141a3 100644
--- a/src/hotspot/cpu/ppc/c1_MacroAssembler_ppc.cpp
+++ b/src/hotspot/cpu/ppc/c1_MacroAssembler_ppc.cpp
@@ -40,29 +40,6 @@
#include "utilities/macros.hpp"
#include "utilities/powerOfTwo.hpp"
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache) {
- const Register temp_reg = R12_scratch2;
- Label Lmiss;
-
- verify_oop(receiver, FILE_AND_LINE);
- load_klass_check_null(temp_reg, receiver, &Lmiss);
-
- if (TrapBasedICMissChecks && TrapBasedNullChecks) {
- trap_ic_miss_check(temp_reg, iCache);
- } else {
- Label Lok;
- cmpd(CCR0, temp_reg, iCache);
- beq(CCR0, Lok);
- bind(Lmiss);
- //load_const_optimized(temp_reg, SharedRuntime::get_ic_miss_stub(), R0);
- calculate_address_from_global_toc(temp_reg, SharedRuntime::get_ic_miss_stub(), true, true, false);
- mtctr(temp_reg);
- bctr();
- align(32, 12);
- bind(Lok);
- }
-}
-
void C1_MacroAssembler::explicit_null_check(Register base) {
Unimplemented();
diff --git a/src/hotspot/cpu/ppc/c1_Runtime1_ppc.cpp b/src/hotspot/cpu/ppc/c1_Runtime1_ppc.cpp
index 2ba6a6bca4e..63914c5d1cb 100644
--- a/src/hotspot/cpu/ppc/c1_Runtime1_ppc.cpp
+++ b/src/hotspot/cpu/ppc/c1_Runtime1_ppc.cpp
@@ -34,7 +34,6 @@
#include "gc/shared/cardTableBarrierSet.hpp"
#include "interpreter/interpreter.hpp"
#include "nativeInst_ppc.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_ppc.hpp"
diff --git a/src/hotspot/cpu/ppc/compiledIC_ppc.cpp b/src/hotspot/cpu/ppc/compiledIC_ppc.cpp
index 54f9cfa9367..355ac4815d5 100644
--- a/src/hotspot/cpu/ppc/compiledIC_ppc.cpp
+++ b/src/hotspot/cpu/ppc/compiledIC_ppc.cpp
@@ -26,7 +26,6 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "memory/resourceArea.hpp"
#include "runtime/mutexLocker.hpp"
@@ -37,7 +36,7 @@
// ----------------------------------------------------------------------------
-// A PPC CompiledDirectStaticCall looks like this:
+// A PPC CompiledDirectCall looks like this:
//
// >>>> consts
//
@@ -79,7 +78,7 @@
const int IC_pos_in_java_to_interp_stub = 8;
#define __ _masm.
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/* = nullptr*/) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/* = nullptr*/) {
#ifdef COMPILER2
if (mark == nullptr) {
// Get the mark within main instrs section which is set to the address of the call.
@@ -91,7 +90,7 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/*
MacroAssembler _masm(&cbuf);
// Start the stub.
- address stub = __ start_a_stub(CompiledStaticCall::to_interp_stub_size());
+ address stub = __ start_a_stub(CompiledDirectCall::to_interp_stub_size());
if (stub == nullptr) {
return nullptr; // CodeCache is full
}
@@ -135,7 +134,7 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/*
// FIXME: Assert that the stub can be identified and patched.
// Java_to_interp_stub_size should be good.
- assert((__ offset() - stub_start_offset) <= CompiledStaticCall::to_interp_stub_size(),
+ assert((__ offset() - stub_start_offset) <= CompiledDirectCall::to_interp_stub_size(),
"should be good size");
assert(!is_NativeCallTrampolineStub_at(__ addr_at(stub_start_offset)),
"must not confuse java_to_interp with trampoline stubs");
@@ -153,27 +152,20 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/*
// Size of java_to_interp stub, this doesn't need to be accurate but it must
// be larger or equal to the real size of the stub.
// Used for optimization in Compile::Shorten_branches.
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return 12 * BytesPerInstWord;
}
// Relocation entries for call stub, compiled java to interpreter.
// Used for optimization in Compile::Shorten_branches.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 5;
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder = nativeMovConstReg_at(stub + IC_pos_in_java_to_interp_stub);
NativeJump* jump = nativeJump_at(method_holder->next_instruction_address());
@@ -188,7 +180,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
// Reset stub.
address stub = static_stub->addr();
assert(stub != nullptr, "stub not found");
@@ -204,7 +196,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/ppc/icBuffer_ppc.cpp b/src/hotspot/cpu/ppc/icBuffer_ppc.cpp
deleted file mode 100644
index 4157a5b0fd7..00000000000
--- a/src/hotspot/cpu/ppc/icBuffer_ppc.cpp
+++ /dev/null
@@ -1,69 +0,0 @@
-/*
- * Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2013 SAP SE. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/assembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_ppc.hpp"
-#include "oops/oop.inline.hpp"
-
-#define __ masm.
-
-int InlineCacheBuffer::ic_stub_code_size() {
- return MacroAssembler::load_const_size + MacroAssembler::b64_patchable_size;
-}
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point) {
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler masm(&code);
- // Note: even though the code contains an embedded metadata, we do not need reloc info
- // because
- // (1) the metadata is old (i.e., doesn't matter for scavenges)
- // (2) these ICStubs are removed *before* a GC happens, so the roots disappear.
-
- // Load the oop ...
- __ load_const(R19_method, (address) cached_value, R0);
- // ... and jump to entry point.
- __ b64_patchable((address) entry_point, relocInfo::none);
-
- __ flush();
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // creation also verifies the object
- NativeJump* jump = nativeJump_at(move->next_instruction_address());
- return jump->jump_destination();
-}
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // creation also verifies the object
- void* o = (void*)move->data();
- return o;
-}
-
diff --git a/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp b/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp
index b9d1cdb19ac..fe19cf03500 100644
--- a/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp
+++ b/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp
@@ -25,6 +25,7 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/disassembler.hpp"
#include "gc/shared/collectedHeap.inline.hpp"
#include "gc/shared/barrierSet.hpp"
@@ -1195,6 +1196,81 @@ void MacroAssembler::post_call_nop() {
assert(is_post_call_nop(*(int*)(pc() - 4)), "post call not not found");
}
+int MacroAssembler::ic_check_size() {
+ bool implicit_null_checks_available = ImplicitNullChecks && os::zero_page_read_protected(),
+ use_fast_receiver_null_check = implicit_null_checks_available || TrapBasedNullChecks,
+ use_trap_based_null_check = !implicit_null_checks_available && TrapBasedNullChecks;
+
+ int num_ins;
+ if (use_fast_receiver_null_check && TrapBasedICMissChecks) {
+ num_ins = 3;
+ if (use_trap_based_null_check) num_ins += 1;
+ } else {
+ num_ins = 7;
+ if (!implicit_null_checks_available) num_ins += 2;
+ }
+ return num_ins * BytesPerInstWord;
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ bool implicit_null_checks_available = ImplicitNullChecks && os::zero_page_read_protected(),
+ use_fast_receiver_null_check = implicit_null_checks_available || TrapBasedNullChecks,
+ use_trap_based_null_check = !implicit_null_checks_available && TrapBasedNullChecks;
+
+ Register receiver = R3_ARG1;
+ Register data = R19_inline_cache_reg;
+ Register tmp1 = R11_scratch1;
+ Register tmp2 = R12_scratch2;
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, end_alignment, end_alignment - ic_check_size());
+
+ int uep_offset = offset();
+
+ if (use_fast_receiver_null_check && TrapBasedICMissChecks) {
+ // Fast version which uses SIGTRAP
+
+ if (use_trap_based_null_check) {
+ trap_null_check(receiver);
+ }
+ if (UseCompressedClassPointers) {
+ lwz(tmp1, oopDesc::klass_offset_in_bytes(), receiver);
+ } else {
+ ld(tmp1, oopDesc::klass_offset_in_bytes(), receiver);
+ }
+ ld(tmp2, in_bytes(CompiledICData::speculated_klass_offset()), data);
+ trap_ic_miss_check(tmp1, tmp2);
+
+ } else {
+ // Slower version which doesn't use SIGTRAP
+
+ // Load stub address using toc (fixed instruction size, unlike load_const_optimized)
+ calculate_address_from_global_toc(tmp1, SharedRuntime::get_ic_miss_stub(),
+ true, true, false); // 2 instructions
+ mtctr(tmp1);
+
+ if (!implicit_null_checks_available) {
+ cmpdi(CCR0, receiver, 0);
+ beqctr(CCR0);
+ }
+ if (UseCompressedClassPointers) {
+ lwz(tmp1, oopDesc::klass_offset_in_bytes(), receiver);
+ } else {
+ ld(tmp1, oopDesc::klass_offset_in_bytes(), receiver);
+ }
+ ld(tmp2, in_bytes(CompiledICData::speculated_klass_offset()), data);
+ cmpd(CCR0, tmp1, tmp2);
+ bnectr(CCR0);
+ }
+
+ assert((offset() % end_alignment) == 0, "Misaligned verified entry point");
+
+ return uep_offset;
+}
+
void MacroAssembler::call_VM_base(Register oop_result,
Register last_java_sp,
address entry_point,
diff --git a/src/hotspot/cpu/ppc/macroAssembler_ppc.hpp b/src/hotspot/cpu/ppc/macroAssembler_ppc.hpp
index cddc8b92fa0..ec370a450ac 100644
--- a/src/hotspot/cpu/ppc/macroAssembler_ppc.hpp
+++ b/src/hotspot/cpu/ppc/macroAssembler_ppc.hpp
@@ -367,6 +367,9 @@ class MacroAssembler: public Assembler {
Register toc);
#endif
+ static int ic_check_size();
+ int ic_check(int end_alignment);
+
protected:
// It is imperative that all calls into the VM are handled via the
diff --git a/src/hotspot/cpu/ppc/ppc.ad b/src/hotspot/cpu/ppc/ppc.ad
index be37ff1785b..6f5e6dabec5 100644
--- a/src/hotspot/cpu/ppc/ppc.ad
+++ b/src/hotspot/cpu/ppc/ppc.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2012, 2023 SAP SE. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
@@ -1026,7 +1026,7 @@ bool followed_by_acquire(const Node *load) {
assert(load->is_Load(), "So far implemented only for loads.");
// Find MemBarAcquire.
- const Node *mba = NULL;
+ const Node *mba = nullptr;
for (DUIterator_Fast imax, i = load->fast_outs(imax); i < imax; i++) {
const Node *out = load->fast_out(i);
if (out->Opcode() == Op_MemBarAcquire) {
@@ -1043,7 +1043,7 @@ bool followed_by_acquire(const Node *load) {
// edge to assure no other operations are in between the two nodes.
//
// So first get the Proj node, mem_proj, to use it to iterate forward.
- Node *mem_proj = NULL;
+ Node *mem_proj = nullptr;
for (DUIterator_Fast imax, i = mba->fast_outs(imax); i < imax; i++) {
mem_proj = mba->fast_out(i); // Runs out of bounds and asserts if Proj not found.
assert(mem_proj->is_Proj(), "only projections here");
@@ -1270,7 +1270,7 @@ source %{
void CallStubImpl::emit_trampoline_stub(C2_MacroAssembler &_masm, int destination_toc_offset, int insts_call_instruction_offset) {
address stub = __ emit_trampoline_stub(destination_toc_offset, insts_call_instruction_offset);
- if (stub == NULL) {
+ if (stub == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
}
}
@@ -1305,11 +1305,11 @@ EmitCallOffsets emit_call_with_trampoline_stub(C2_MacroAssembler &_masm, address
offsets.insts_call_instruction_offset = __ offset();
// No entry point given, use the current pc.
- if (entry_point == NULL) entry_point = __ pc();
+ if (entry_point == nullptr) entry_point = __ pc();
// Put the entry point as a constant into the constant pool.
const address entry_point_toc_addr = __ address_constant(entry_point, RelocationHolder::none);
- if (entry_point_toc_addr == NULL) {
+ if (entry_point_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return offsets;
}
@@ -1355,8 +1355,8 @@ void MachConstantBaseNode::postalloc_expand(GrowableArray *nodes, Phase
MachNode *m1 = new loadToc_hiNode();
MachNode *m2 = new loadToc_loNode();
- m1->add_req(NULL);
- m2->add_req(NULL, m1);
+ m1->add_req(nullptr);
+ m2->add_req(nullptr, m1);
m1->_opnds[0] = op_dst;
m2->_opnds[0] = op_dst;
m2->_opnds[1] = op_dst;
@@ -1398,7 +1398,7 @@ void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
st->print("push frame %ld\n\t", -framesize);
}
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
st->print("nmethod entry barrier\n\t");
}
}
@@ -1554,7 +1554,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ std(return_pc, _abi0(lr), callers_sp);
}
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
bs->nmethod_entry_barrier(&_masm, push_frame_temp);
}
@@ -1724,7 +1724,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
if (src_lo == dst_lo && src_hi == dst_hi)
return size; // Self copy, no move.
- if (bottom_type()->isa_vect() != NULL && ideal_reg() == Op_VecX) {
+ if (bottom_type()->isa_vect() != nullptr && ideal_reg() == Op_VecX) {
// Memory->Memory Spill.
if (src_lo_rc == rc_stack && dst_lo_rc == rc_stack) {
int src_offset = ra_->reg2offset(src_lo);
@@ -1910,16 +1910,16 @@ void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
if (!ra_)
st->print("N%d = SpillCopy(N%d)", _idx, in(1)->_idx);
else
- implementation(NULL, ra_, false, st);
+ implementation(nullptr, ra_, false, st);
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation(&cbuf, ra_, false, NULL);
+ implementation(&cbuf, ra_, false, nullptr);
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
- return implementation(NULL, ra_, true, NULL);
+ return implementation(nullptr, ra_, true, nullptr);
}
#ifndef PRODUCT
@@ -1978,42 +1978,7 @@ void MachUEPNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
// This is the unverified entry point.
C2_MacroAssembler _masm(&cbuf);
- // Inline_cache contains a klass.
- Register ic_klass = as_Register(Matcher::inline_cache_reg_encode());
- Register receiver_klass = R12_scratch2; // tmp
-
- assert_different_registers(ic_klass, receiver_klass, R11_scratch1, R3_ARG1);
- assert(R11_scratch1 == R11, "need prologue scratch register");
-
- // Check for NULL argument if we don't have implicit null checks.
- if (!ImplicitNullChecks || !os::zero_page_read_protected()) {
- if (TrapBasedNullChecks) {
- __ trap_null_check(R3_ARG1);
- } else {
- Label valid;
- __ cmpdi(CCR0, R3_ARG1, 0);
- __ bne_predict_taken(CCR0, valid);
- // We have a null argument, branch to ic_miss_stub.
- __ b64_patchable((address)SharedRuntime::get_ic_miss_stub(),
- relocInfo::runtime_call_type);
- __ bind(valid);
- }
- }
- // Assume argument is not NULL, load klass from receiver.
- __ load_klass(receiver_klass, R3_ARG1);
-
- if (TrapBasedICMissChecks) {
- __ trap_ic_miss_check(receiver_klass, ic_klass);
- } else {
- Label valid;
- __ cmpd(CCR0, receiver_klass, ic_klass);
- __ beq_predict_taken(CCR0, valid);
- // We have an unexpected klass, branch to ic_miss_stub.
- __ b64_patchable((address)SharedRuntime::get_ic_miss_stub(),
- relocInfo::runtime_call_type);
- __ bind(valid);
- }
-
+ __ ic_check(CodeEntryAlignment);
// Argument is valid and klass is as expected, continue.
}
@@ -2062,7 +2027,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer &cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) return 0; // CodeBuffer::expand failed
+ if (base == nullptr) return 0; // CodeBuffer::expand failed
int offset = __ offset();
__ b64_patchable((address)OptoRuntime::exception_blob()->content_begin(),
@@ -2079,7 +2044,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) return 0; // CodeBuffer::expand failed
+ if (base == nullptr) return 0; // CodeBuffer::expand failed
int offset = __ offset();
__ bl64_patchable((address)SharedRuntime::deopt_blob()->unpack(),
@@ -2173,7 +2138,7 @@ bool Matcher::match_rule_supported(int opcode) {
return true; // Per default match rules are supported.
}
-bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
return match_rule_supported_vector(opcode, vlen, bt);
}
@@ -2193,11 +2158,11 @@ bool Matcher::vector_needs_partial_operations(Node* node, const TypeVect* vt) {
}
const RegMask* Matcher::predicate_reg_mask(void) {
- return NULL;
+ return nullptr;
}
const TypeVectMask* Matcher::predicate_reg_type(const Type* elemTy, int length) {
- return NULL;
+ return nullptr;
}
// Vector calling convention not yet implemented.
@@ -2242,7 +2207,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return max_vector_size(bt); // Same as max.
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
return Matcher::max_vector_size(bt);
}
@@ -2275,7 +2240,7 @@ bool Matcher::is_short_branch_offset(int rule, int br_size, int offset) {
/* TODO: PPC port
// Make a new machine dependent decode node (with its operands).
MachTypeNode *Matcher::make_decode_node() {
- assert(CompressedOops::base() == NULL && CompressedOops::shift() == 0,
+ assert(CompressedOops::base() == nullptr && CompressedOops::shift() == 0,
"This method is only implemented for unscaled cOops mode so far");
MachTypeNode *decode = new decodeN_unscaledNode();
decode->set_opnd_array(0, new iRegPdstOper());
@@ -2286,7 +2251,7 @@ MachTypeNode *Matcher::make_decode_node() {
MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* original_opnd, uint ideal_reg, bool is_temp) {
ShouldNotReachHere(); // generic vector operands not supported
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -2545,7 +2510,7 @@ encode %{
// Create a non-oop constant, no relocation needed.
// If it is an IC, it has a virtual_call_Relocation.
const_toc_addr = __ long_constant((jlong)$src$$constant);
- if (const_toc_addr == NULL) {
+ if (const_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -2568,7 +2533,7 @@ encode %{
// Create a non-oop constant, no relocation needed.
// If it is an IC, it has a virtual_call_Relocation.
const_toc_addr = __ long_constant((jlong)$src$$constant);
- if (const_toc_addr == NULL) {
+ if (const_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -2607,8 +2572,8 @@ loadConLNodesTuple loadConLNodesTuple_create(PhaseRegAlloc *ra_, Node *toc, immL
loadConL_loNode *m2 = new loadConL_loNode();
// inputs for new nodes
- m1->add_req(NULL, toc);
- m2->add_req(NULL, m1);
+ m1->add_req(nullptr, toc);
+ m2->add_req(nullptr, m1);
// operands for new nodes
m1->_opnds[0] = new iRegLdstOper(); // dst
@@ -2632,14 +2597,14 @@ loadConLNodesTuple loadConLNodesTuple_create(PhaseRegAlloc *ra_, Node *toc, immL
// Create result.
nodes._large_hi = m1;
nodes._large_lo = m2;
- nodes._small = NULL;
+ nodes._small = nullptr;
nodes._last = nodes._large_lo;
assert(m2->bottom_type()->isa_long(), "must be long");
} else {
loadConLNode *m2 = new loadConLNode();
// inputs for new nodes
- m2->add_req(NULL, toc);
+ m2->add_req(nullptr, toc);
// operands for new nodes
m2->_opnds[0] = new iRegLdstOper(); // dst
@@ -2653,8 +2618,8 @@ loadConLNodesTuple loadConLNodesTuple_create(PhaseRegAlloc *ra_, Node *toc, immL
ra_->set_pair(m2->_idx, reg_second, reg_first);
// Create result.
- nodes._large_hi = NULL;
- nodes._large_lo = NULL;
+ nodes._large_hi = nullptr;
+ nodes._large_lo = nullptr;
nodes._small = m2;
nodes._last = nodes._small;
assert(m2->bottom_type()->isa_long(), "must be long");
@@ -2687,10 +2652,10 @@ loadConLReplicatedNodesTuple loadConLReplicatedNodesTuple_create(Compile *C, Pha
xxspltdNode *m4 = new xxspltdNode();
// inputs for new nodes
- m1->add_req(NULL, toc);
- m2->add_req(NULL, m1);
- m3->add_req(NULL, m2);
- m4->add_req(NULL, m3);
+ m1->add_req(nullptr, toc);
+ m2->add_req(nullptr, m1);
+ m3->add_req(nullptr, m2);
+ m4->add_req(nullptr, m3);
// operands for new nodes
m1->_opnds[0] = new iRegLdstOper(); // dst
@@ -2727,7 +2692,7 @@ loadConLReplicatedNodesTuple loadConLReplicatedNodesTuple_create(Compile *C, Pha
nodes._large_lo = m2;
nodes._moved = m3;
nodes._replicated = m4;
- nodes._small = NULL;
+ nodes._small = nullptr;
nodes._last = nodes._replicated;
assert(m2->bottom_type()->isa_long(), "must be long");
} else {
@@ -2736,7 +2701,7 @@ loadConLReplicatedNodesTuple loadConLReplicatedNodesTuple_create(Compile *C, Pha
xxspltdNode *m4 = new xxspltdNode();
// inputs for new nodes
- m2->add_req(NULL, toc);
+ m2->add_req(nullptr, toc);
// operands for new nodes
m2->_opnds[0] = new iRegLdstOper(); // dst
@@ -2760,8 +2725,8 @@ loadConLReplicatedNodesTuple loadConLReplicatedNodesTuple_create(Compile *C, Pha
ra_->set_pair(m2->_idx, reg_second, reg_first);
// Create result.
- nodes._large_hi = NULL;
- nodes._large_lo = NULL;
+ nodes._large_hi = nullptr;
+ nodes._large_lo = nullptr;
nodes._small = m2;
nodes._moved = m3;
nodes._replicated = m4;
@@ -2815,7 +2780,7 @@ encode %{
const_toc_addr = __ long_constant((jlong)$src$$constant);
}
- if (const_toc_addr == NULL) {
+ if (const_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -2846,7 +2811,7 @@ encode %{
const_toc_addr = __ long_constant((jlong)$src$$constant);
}
- if (const_toc_addr == NULL) {
+ if (const_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -2870,8 +2835,8 @@ encode %{
loadConP_loNode *m2 = new loadConP_loNode();
// inputs for new nodes
- m1->add_req(NULL, n_toc);
- m2->add_req(NULL, m1);
+ m1->add_req(nullptr, n_toc);
+ m2->add_req(nullptr, m1);
// operands for new nodes
m1->_opnds[0] = new iRegPdstOper(); // dst
@@ -2896,7 +2861,7 @@ encode %{
loadConPNode *m2 = new loadConPNode();
// inputs for new nodes
- m2->add_req(NULL, n_toc);
+ m2->add_req(nullptr, n_toc);
// operands for new nodes
m2->_opnds[0] = new iRegPdstOper(); // dst
@@ -2923,7 +2888,7 @@ encode %{
m2 = new loadConFNode();
}
// inputs for new nodes
- m2->add_req(NULL, n_toc);
+ m2->add_req(nullptr, n_toc);
// operands for new nodes
m2->_opnds[0] = op_dst;
@@ -2947,7 +2912,7 @@ encode %{
m2 = new loadConDNode();
}
// inputs for new nodes
- m2->add_req(NULL, n_toc);
+ m2->add_req(nullptr, n_toc);
// operands for new nodes
m2->_opnds[0] = op_dst;
@@ -3258,9 +3223,9 @@ encode %{
Label d; // dummy
__ bind(d);
Label* p = ($lbl$$label);
- // `p' is `NULL' when this encoding class is used only to
+ // `p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
- Label& l = (NULL == p)? d : *(p);
+ Label& l = (nullptr == p)? d : *(p);
int cc = $cmp$$cmpcode;
int flags_reg = $crx$$reg;
assert((Assembler::bcondCRbiIs1 & ~Assembler::bcondCRbiIs0) == 8, "check encoding");
@@ -3287,9 +3252,9 @@ encode %{
Label d; // dummy
__ bind(d);
Label* p = ($lbl$$label);
- // `p' is `NULL' when this encoding class is used only to
+ // `p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
- Label& l = (NULL == p)? d : *(p);
+ Label& l = (nullptr == p)? d : *(p);
int cc = $cmp$$cmpcode;
int flags_reg = $crx$$reg;
int bhint = Assembler::bhintNoHint;
@@ -3431,7 +3396,7 @@ encode %{
// Put the entry point as a constant into the constant pool.
const address entry_point_toc_addr = __ address_constant(entry_point, RelocationHolder::none);
- if (entry_point_toc_addr == NULL) {
+ if (entry_point_toc_addr == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -3452,8 +3417,8 @@ encode %{
__ bl(__ pc()); // Emits a relocation.
// The stub for call to interpreter.
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3470,7 +3435,7 @@ encode %{
// Create a call trampoline stub for the given method.
const address entry_point = !($meth$$method) ? 0 : (address)$meth$$method;
const address entry_point_const = __ address_constant(entry_point, RelocationHolder::none);
- if (entry_point_const == NULL) {
+ if (entry_point_const == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -3479,13 +3444,13 @@ encode %{
if (ra_->C->env()->failing()) { return; } // Code cache may be full.
// Build relocation at call site with ic position as data.
- assert((_load_ic_hi_node != NULL && _load_ic_node == NULL) ||
- (_load_ic_hi_node == NULL && _load_ic_node != NULL),
+ assert((_load_ic_hi_node != nullptr && _load_ic_node == nullptr) ||
+ (_load_ic_hi_node == nullptr && _load_ic_node != nullptr),
"must have one, but can't have both");
- assert((_load_ic_hi_node != NULL && _load_ic_hi_node->_cbuf_insts_offset != -1) ||
- (_load_ic_node != NULL && _load_ic_node->_cbuf_insts_offset != -1),
+ assert((_load_ic_hi_node != nullptr && _load_ic_hi_node->_cbuf_insts_offset != -1) ||
+ (_load_ic_node != nullptr && _load_ic_node->_cbuf_insts_offset != -1),
"must contain instruction offset");
- const int virtual_call_oop_addr_offset = _load_ic_hi_node != NULL
+ const int virtual_call_oop_addr_offset = _load_ic_hi_node != nullptr
? _load_ic_hi_node->_cbuf_insts_offset
: _load_ic_node->_cbuf_insts_offset;
const address virtual_call_oop_addr = __ addr_at(virtual_call_oop_addr_offset);
@@ -3507,7 +3472,7 @@ encode %{
// Create the nodes for loading the IC from the TOC.
loadConLNodesTuple loadConLNodes_IC =
- loadConLNodesTuple_create(ra_, n_toc, new immLOper((jlong)Universe::non_oop_word()),
+ loadConLNodesTuple_create(ra_, n_toc, new immLOper((jlong) Universe::non_oop_word()),
OptoReg::Name(R19_H_num), OptoReg::Name(R19_num));
// Create the call node.
@@ -3625,7 +3590,7 @@ encode %{
const address start_pc = __ pc();
#if defined(ABI_ELFv2)
- address entry= !($meth$$method) ? NULL : (address)$meth$$method;
+ address entry= !($meth$$method) ? nullptr : (address)$meth$$method;
__ call_c(entry, relocInfo::runtime_call_type);
__ post_call_nop();
#else
@@ -3682,13 +3647,13 @@ encode %{
// Create nodes and operands for loading the env pointer.
- if (fd->env() != NULL) {
+ if (fd->env() != nullptr) {
loadConLNodes_Env = loadConLNodesTuple_create(ra_, n_toc, new immLOper((jlong) fd->env()),
OptoReg::Name(R11_H_num), OptoReg::Name(R11_num));
} else {
- loadConLNodes_Env._large_hi = NULL;
- loadConLNodes_Env._large_lo = NULL;
- loadConLNodes_Env._small = NULL;
+ loadConLNodes_Env._large_hi = nullptr;
+ loadConLNodes_Env._large_lo = nullptr;
+ loadConLNodes_Env._small = nullptr;
loadConLNodes_Env._last = new loadConL16Node();
loadConLNodes_Env._last->_opnds[0] = new iRegLdstOper();
loadConLNodes_Env._last->_opnds[1] = new immL16Oper(0);
@@ -3702,7 +3667,7 @@ encode %{
// mtctr node
MachNode *mtctr = new CallLeafDirect_mtctrNode();
- assert(loadConLNodes_Entry._last != NULL, "entry must exist");
+ assert(loadConLNodes_Entry._last != nullptr, "entry must exist");
mtctr->add_req(0, loadConLNodes_Entry._last);
mtctr->_opnds[0] = new iRegLdstOper();
@@ -3722,7 +3687,7 @@ encode %{
call->_guaranteed_safepoint = false;
call->_oop_map = _oop_map;
guarantee(!_jvms, "You must clone the jvms and adapt the offsets by fix_jvms().");
- call->_jvms = NULL;
+ call->_jvms = nullptr;
call->_jvmadj = _jvmadj;
call->_in_rms = _in_rms;
call->_nesting = _nesting;
@@ -3826,7 +3791,7 @@ frame %{
// opcodes. This simplifies the register allocator.
c_return_value %{
assert((ideal_reg >= Op_RegI && ideal_reg <= Op_RegL) ||
- (ideal_reg == Op_RegN && CompressedOops::base() == NULL && CompressedOops::shift() == 0),
+ (ideal_reg == Op_RegN && CompressedOops::base() == nullptr && CompressedOops::shift() == 0),
"only return normal values");
// enum names from opcodes.hpp: Op_Node Op_Set Op_RegN Op_RegI Op_RegP Op_RegF Op_RegD Op_RegL
static int typeToRegLo[Op_RegL+1] = { 0, 0, R3_num, R3_num, R3_num, F1_num, F1_num, R3_num };
@@ -3837,7 +3802,7 @@ frame %{
// Location of compiled Java return values. Same as C
return_value %{
assert((ideal_reg >= Op_RegI && ideal_reg <= Op_RegL) ||
- (ideal_reg == Op_RegN && CompressedOops::base() == NULL && CompressedOops::shift() == 0),
+ (ideal_reg == Op_RegN && CompressedOops::base() == nullptr && CompressedOops::shift() == 0),
"only return normal values");
// enum names from opcodes.hpp: Op_Node Op_Set Op_RegN Op_RegI Op_RegP Op_RegF Op_RegD Op_RegL
static int typeToRegLo[Op_RegL+1] = { 0, 0, R3_num, R3_num, R3_num, F1_num, F1_num, R3_num };
@@ -4110,7 +4075,7 @@ operand immN() %{
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immN_0() %{
predicate(n->get_narrowcon() == 0);
match(ConN);
@@ -4682,7 +4647,7 @@ operand iRegN2P(iRegNsrc reg) %{
%}
operand iRegN2P_klass(iRegNsrc reg) %{
- predicate(CompressedKlassPointers::base() == NULL && CompressedKlassPointers::shift() == 0);
+ predicate(CompressedKlassPointers::base() == nullptr && CompressedKlassPointers::shift() == 0);
constraint(ALLOC_IN_RC(bits32_reg_ro));
match(DecodeNKlass reg);
format %{ "$reg" %}
@@ -4751,7 +4716,7 @@ operand indirectNarrow(iRegNsrc reg) %{
%}
operand indirectNarrow_klass(iRegNsrc reg) %{
- predicate(CompressedKlassPointers::base() == NULL && CompressedKlassPointers::shift() == 0);
+ predicate(CompressedKlassPointers::base() == nullptr && CompressedKlassPointers::shift() == 0);
constraint(ALLOC_IN_RC(bits64_reg_ro));
match(DecodeNKlass reg);
op_cost(100);
@@ -4780,7 +4745,7 @@ operand indOffset16Narrow(iRegNsrc reg, immL16 offset) %{
%}
operand indOffset16Narrow_klass(iRegNsrc reg, immL16 offset) %{
- predicate(CompressedKlassPointers::base() == NULL && CompressedKlassPointers::shift() == 0);
+ predicate(CompressedKlassPointers::base() == nullptr && CompressedKlassPointers::shift() == 0);
constraint(ALLOC_IN_RC(bits64_reg_ro));
match(AddP (DecodeNKlass reg) offset);
op_cost(100);
@@ -4809,7 +4774,7 @@ operand indOffset16NarrowAlg4(iRegNsrc reg, immL16Alg4 offset) %{
%}
operand indOffset16NarrowAlg4_klass(iRegNsrc reg, immL16Alg4 offset) %{
- predicate(CompressedKlassPointers::base() == NULL && CompressedKlassPointers::shift() == 0);
+ predicate(CompressedKlassPointers::base() == nullptr && CompressedKlassPointers::shift() == 0);
constraint(ALLOC_IN_RC(bits64_reg_ro));
match(AddP (DecodeNKlass reg) offset);
op_cost(100);
@@ -5544,7 +5509,7 @@ instruct loadN2P_unscaled(iRegPdst dst, memory mem) %{
instruct loadN2P_klass_unscaled(iRegPdst dst, memory mem) %{
match(Set dst (DecodeNKlass (LoadNKlass mem)));
- predicate(CompressedKlassPointers::base() == NULL && CompressedKlassPointers::shift() == 0 &&
+ predicate(CompressedKlassPointers::base() == nullptr && CompressedKlassPointers::shift() == 0 &&
_kids[0]->_leaf->as_Load()->is_unordered());
ins_cost(MEMORY_REF_COST);
@@ -5954,7 +5919,7 @@ instruct loadConL_Ex(iRegLdst dst, immL src) %{
postalloc_expand( postalloc_expand_load_long_constant(dst, src, constanttablebase) );
%}
-// Load NULL as compressed oop.
+// Load nullptr as compressed oop.
instruct loadConN0(iRegNdst dst, immN_0 src) %{
match(Set dst src);
ins_cost(DEFAULT_COST);
@@ -6045,9 +6010,9 @@ instruct loadConN_Ex(iRegNdst dst, immN src) %{
MachNode *m1 = new loadConN_hiNode();
MachNode *m2 = new loadConN_loNode();
MachNode *m3 = new clearMs32bNode();
- m1->add_req(NULL);
- m2->add_req(NULL, m1);
- m3->add_req(NULL, m2);
+ m1->add_req(nullptr);
+ m2->add_req(nullptr, m1);
+ m3->add_req(nullptr, m2);
m1->_opnds[0] = op_dst;
m1->_opnds[1] = op_src;
m2->_opnds[0] = op_dst;
@@ -6123,7 +6088,7 @@ instruct loadConNKlass_Ex(iRegNdst dst, immNKlass src) %{
postalloc_expand %{
// Load high bits into register. Sign extended.
MachNode *m1 = new loadConNKlass_hiNode();
- m1->add_req(NULL);
+ m1->add_req(nullptr);
m1->_opnds[0] = op_dst;
m1->_opnds[1] = op_src;
ra_->set_pair(m1->_idx, ra_->get_reg_second(this), ra_->get_reg_first(this));
@@ -6133,7 +6098,7 @@ instruct loadConNKlass_Ex(iRegNdst dst, immNKlass src) %{
if (!Assembler::is_uimm((jlong)CompressedKlassPointers::encode((Klass *)op_src->constant()), 31)) {
// Value might be 1-extended. Mask out these bits.
m2 = new loadConNKlass_maskNode();
- m2->add_req(NULL, m1);
+ m2->add_req(nullptr, m1);
m2->_opnds[0] = op_dst;
m2->_opnds[1] = op_src;
m2->_opnds[2] = op_dst;
@@ -6142,7 +6107,7 @@ instruct loadConNKlass_Ex(iRegNdst dst, immNKlass src) %{
}
MachNode *m3 = new loadConNKlass_loNode();
- m3->add_req(NULL, m2);
+ m3->add_req(nullptr, m2);
m3->_opnds[0] = op_dst;
m3->_opnds[1] = op_src;
m3->_opnds[2] = op_dst;
@@ -6243,7 +6208,7 @@ instruct loadConF(regF dst, immF src, iRegLdst toc) %{
size(4);
ins_encode %{
address float_address = __ float_constant($src$$constant);
- if (float_address == NULL) {
+ if (float_address == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -6267,7 +6232,7 @@ instruct loadConFComp(regF dst, immF src, iRegLdst toc) %{
FloatRegister Rdst = $dst$$FloatRegister;
Register Rtoc = $toc$$Register;
address float_address = __ float_constant($src$$constant);
- if (float_address == NULL) {
+ if (float_address == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -6305,7 +6270,7 @@ instruct loadConD(regD dst, immD src, iRegLdst toc) %{
size(4);
ins_encode %{
address float_address = __ double_constant($src$$constant);
- if (float_address == NULL) {
+ if (float_address == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -6330,7 +6295,7 @@ instruct loadConDComp(regD dst, immD src, iRegLdst toc) %{
FloatRegister Rdst = $dst$$FloatRegister;
Register Rtoc = $toc$$Register;
address float_address = __ double_constant($src$$constant);
- if (float_address == NULL) {
+ if (float_address == nullptr) {
ciEnv::current()->record_out_of_memory_failure();
return;
}
@@ -6632,7 +6597,7 @@ instruct cond_sub_base(iRegNdst dst, flagsRegSrc crx, iRegPsrc src1) %{
predicate(false);
format %{ "BEQ $crx, done\n\t"
- "SUB $dst, $src1, heapbase \t// encode: subtract base if != NULL\n"
+ "SUB $dst, $src1, heapbase \t// encode: subtract base if != nullptr\n"
"done:" %}
ins_encode %{
Label done;
@@ -6701,7 +6666,7 @@ instruct encodeP_not_null_base_null(iRegNdst dst, iRegPsrc src) %{
predicate(CompressedOops::shift() != 0 &&
CompressedOops::base() ==0);
- format %{ "SRDI $dst, $src, #3 \t// encodeP, $src != NULL" %}
+ format %{ "SRDI $dst, $src, #3 \t// encodeP, $src != nullptr" %}
size(4);
ins_encode %{
__ srdi($dst$$Register, $src$$Register, CompressedOops::shift() & 0x3f);
@@ -6762,7 +6727,7 @@ instruct cond_add_base(iRegPdst dst, flagsRegSrc crx, iRegPsrc src) %{
predicate(false);
format %{ "BEQ $crx, done\n\t"
- "ADD $dst, $src, heapbase \t// DecodeN: add oop base if $src != NULL\n"
+ "ADD $dst, $src, heapbase \t// DecodeN: add oop base if $src != nullptr\n"
"done:" %}
ins_encode %{
Label done;
@@ -6850,7 +6815,7 @@ instruct decodeN_Disjoint_notNull_Ex(iRegPdst dst, iRegNsrc src) %{
"RLDIMI $dst, $src, shift, 32-shift \t// decode with disjoint base" %}
postalloc_expand %{
loadBaseNode *n1 = new loadBaseNode();
- n1->add_req(NULL);
+ n1->add_req(nullptr);
n1->_opnds[0] = op_dst;
decodeN_mergeDisjointNode *n2 = new decodeN_mergeDisjointNode();
@@ -6882,7 +6847,7 @@ instruct decodeN_Disjoint_isel_Ex(iRegPdst dst, iRegNsrc src, flagsReg crx) %{
format %{ "DecodeN $dst, $src \t// decode with disjoint base using isel" %}
postalloc_expand %{
loadBaseNode *n1 = new loadBaseNode();
- n1->add_req(NULL);
+ n1->add_req(nullptr);
n1->_opnds[0] = op_dst;
cmpN_reg_imm0Node *n_compare = new cmpN_reg_imm0Node();
@@ -6929,7 +6894,7 @@ instruct decodeN_notNull_addBase_Ex(iRegPdst dst, iRegNsrc src) %{
CompressedOops::base() != 0);
ins_cost(2 * DEFAULT_COST);
- format %{ "DecodeN $dst, $src \t// $src != NULL, postalloc expanded" %}
+ format %{ "DecodeN $dst, $src \t// $src != nullptr, postalloc expanded" %}
postalloc_expand( postalloc_expand_decode_oop_not_null(dst, src));
%}
@@ -7086,7 +7051,7 @@ instruct decodeNKlass_notNull_addBase_Ex(iRegPdst dst, iRegLsrc base, iRegNsrc s
//effect(kill src); // We need a register for the immediate result after shifting.
predicate(false);
- format %{ "DecodeNKlass $dst = $base + ($src << 3) \t// $src != NULL, postalloc expanded" %}
+ format %{ "DecodeNKlass $dst = $base + ($src << 3) \t// $src != nullptr, postalloc expanded" %}
postalloc_expand %{
decodeNKlass_add_baseNode *n1 = new decodeNKlass_add_baseNode();
n1->add_req(n_region, n_base, n_src);
@@ -7115,7 +7080,7 @@ instruct decodeNKlass_notNull_addBase_ExEx(iRegPdst dst, iRegNsrc src) %{
// predicate(CompressedKlassPointers::shift() != 0 &&
// CompressedKlassPointers::base() != 0);
- //format %{ "DecodeNKlass $dst, $src \t// $src != NULL, expanded" %}
+ //format %{ "DecodeNKlass $dst, $src \t// $src != nullptr, expanded" %}
ins_cost(DEFAULT_COST*2); // Don't count constant.
expand %{
@@ -7602,7 +7567,7 @@ instruct compareAndSwapL_regP_regL_regL(iRegIdst res, iRegPdst mem_ptr, iRegLsrc
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- $res$$Register, NULL, true);
+ $res$$Register, nullptr, true);
if (support_IRIW_for_not_multiple_copy_atomic_cpu) {
__ isync();
} else {
@@ -7621,7 +7586,7 @@ instruct compareAndSwapP_regP_regP_regP(iRegIdst res, iRegPdst mem_ptr, iRegPsrc
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- $res$$Register, NULL, true);
+ $res$$Register, nullptr, true);
if (support_IRIW_for_not_multiple_copy_atomic_cpu) {
__ isync();
} else {
@@ -7815,7 +7780,7 @@ instruct weakCompareAndSwapL_regP_regL_regL(iRegIdst res, iRegPdst mem_ptr, iReg
// value is never passed to caller.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone,
- MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, NULL, true, /*weak*/ true);
+ MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, nullptr, true, /*weak*/ true);
%}
ins_pipe(pipe_class_default);
%}
@@ -7831,7 +7796,7 @@ instruct weakCompareAndSwapL_acq_regP_regL_regL(iRegIdst res, iRegPdst mem_ptr,
// value is never passed to caller.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
support_IRIW_for_not_multiple_copy_atomic_cpu ? MacroAssembler::MemBarAcq : MacroAssembler::MemBarFenceAfter,
- MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, NULL, true, /*weak*/ true);
+ MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, nullptr, true, /*weak*/ true);
%}
ins_pipe(pipe_class_default);
%}
@@ -7845,7 +7810,7 @@ instruct weakCompareAndSwapP_regP_regP_regP(iRegIdst res, iRegPdst mem_ptr, iReg
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone,
- MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, NULL, true, /*weak*/ true);
+ MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, nullptr, true, /*weak*/ true);
%}
ins_pipe(pipe_class_default);
%}
@@ -7861,7 +7826,7 @@ instruct weakCompareAndSwapP_acq_regP_regP_regP(iRegIdst res, iRegPdst mem_ptr,
// value is never passed to caller.
__ cmpxchgd(CCR0, R0, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
support_IRIW_for_not_multiple_copy_atomic_cpu ? MacroAssembler::MemBarAcq : MacroAssembler::MemBarFenceAfter,
- MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, NULL, true, /*weak*/ true);
+ MacroAssembler::cmpxchgx_hint_atomic_update(), $res$$Register, nullptr, true, /*weak*/ true);
%}
ins_pipe(pipe_class_default);
%}
@@ -8081,7 +8046,7 @@ instruct compareAndExchangeL_regP_regL_regL(iRegLdst res, iRegPdst mem_ptr, iReg
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, $res$$Register, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- noreg, NULL, true);
+ noreg, nullptr, true);
%}
ins_pipe(pipe_class_default);
%}
@@ -8095,7 +8060,7 @@ instruct compareAndExchangeL_acq_regP_regL_regL(iRegLdst res, iRegPdst mem_ptr,
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, $res$$Register, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- noreg, NULL, true);
+ noreg, nullptr, true);
if (support_IRIW_for_not_multiple_copy_atomic_cpu) {
__ isync();
} else {
@@ -8116,7 +8081,7 @@ instruct compareAndExchangeP_regP_regP_regP(iRegPdst res, iRegPdst mem_ptr, iReg
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, $res$$Register, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- noreg, NULL, true);
+ noreg, nullptr, true);
%}
ins_pipe(pipe_class_default);
%}
@@ -8131,7 +8096,7 @@ instruct compareAndExchangeP_acq_regP_regP_regP(iRegPdst res, iRegPdst mem_ptr,
// CmpxchgX sets CCR0 to cmpX(src1, src2) and Rres to 'true'/'false'.
__ cmpxchgd(CCR0, $res$$Register, $src1$$Register, $src2$$Register, $mem_ptr$$Register,
MacroAssembler::MemBarNone, MacroAssembler::cmpxchgx_hint_atomic_update(),
- noreg, NULL, true);
+ noreg, nullptr, true);
if (support_IRIW_for_not_multiple_copy_atomic_cpu) {
__ isync();
} else {
@@ -12043,9 +12008,9 @@ instruct branch(label labl) %{
Label d; // dummy
__ bind(d);
Label* p = $labl$$label;
- // `p' is `NULL' when this encoding class is used only to
+ // `p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
- Label& l = (NULL == p)? d : *(p);
+ Label& l = (nullptr == p)? d : *(p);
__ b(l);
%}
ins_pipe(pipe_class_default);
@@ -12139,7 +12104,7 @@ instruct partialSubtypeCheck(iRegPdst result, iRegP_N2P subklass, iRegP_N2P supe
format %{ "PartialSubtypeCheck $result = ($subklass instanceOf $superklass) tmp: $tmp_klass, $tmp_arrayptr" %}
ins_encode %{
__ check_klass_subtype_slow_path($subklass$$Register, $superklass$$Register, $tmp_arrayptr$$Register,
- $tmp_klass$$Register, NULL, $result$$Register);
+ $tmp_klass$$Register, nullptr, $result$$Register);
%}
ins_pipe(pipe_class_default);
%}
@@ -12318,21 +12283,6 @@ instruct string_equalsL(rarg1RegP str1, rarg2RegP str2, rarg3RegI cnt, iRegIdst
ins_pipe(pipe_class_default);
%}
-instruct string_equalsU(rarg1RegP str1, rarg2RegP str2, rarg3RegI cnt, iRegIdst result,
- iRegIdst tmp, regCTR ctr, flagsRegCR0 cr0) %{
- predicate(((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU);
- match(Set result (StrEquals (Binary str1 str2) cnt));
- effect(TEMP_DEF result, USE_KILL str1, USE_KILL str2, USE_KILL cnt, TEMP tmp, KILL ctr, KILL cr0);
- ins_cost(300);
- format %{ "String Equals char[] $str1,$str2,$cnt -> $result \t// KILL $tmp" %}
- ins_encode %{
- __ array_equals(false, $str1$$Register, $str2$$Register,
- $cnt$$Register, $tmp$$Register,
- $result$$Register, false /* byte */);
- %}
- ins_pipe(pipe_class_default);
-%}
-
instruct array_equalsB(rarg1RegP ary1, rarg2RegP ary2, iRegIdst result,
iRegIdst tmp1, iRegIdst tmp2, regCTR ctr, flagsRegCR0 cr0, flagsRegCR0 cr1) %{
predicate(((AryEqNode*)n)->encoding() == StrIntrinsicNode::LL);
@@ -12670,7 +12620,7 @@ instruct indexOf_U(iRegIdst result, iRegPsrc haystack, rscratch1RegI haycnt, iRe
ins_encode %{
__ string_indexof($result$$Register,
$haystack$$Register, $haycnt$$Register,
- $needle$$Register, NULL, $needlecnt$$Register, 0, // needlecnt not constant.
+ $needle$$Register, nullptr, $needlecnt$$Register, 0, // needlecnt not constant.
$tmp1$$Register, $tmp2$$Register, $tmp3$$Register, $tmp4$$Register, StrIntrinsicNode::UU);
%}
ins_pipe(pipe_class_compare);
@@ -12691,7 +12641,7 @@ instruct indexOf_L(iRegIdst result, iRegPsrc haystack, rscratch1RegI haycnt, iRe
ins_encode %{
__ string_indexof($result$$Register,
$haystack$$Register, $haycnt$$Register,
- $needle$$Register, NULL, $needlecnt$$Register, 0, // needlecnt not constant.
+ $needle$$Register, nullptr, $needlecnt$$Register, 0, // needlecnt not constant.
$tmp1$$Register, $tmp2$$Register, $tmp3$$Register, $tmp4$$Register, StrIntrinsicNode::LL);
%}
ins_pipe(pipe_class_compare);
@@ -12712,7 +12662,7 @@ instruct indexOf_UL(iRegIdst result, iRegPsrc haystack, rscratch1RegI haycnt, iR
ins_encode %{
__ string_indexof($result$$Register,
$haystack$$Register, $haycnt$$Register,
- $needle$$Register, NULL, $needlecnt$$Register, 0, // needlecnt not constant.
+ $needle$$Register, nullptr, $needlecnt$$Register, 0, // needlecnt not constant.
$tmp1$$Register, $tmp2$$Register, $tmp3$$Register, $tmp4$$Register, StrIntrinsicNode::UL);
%}
ins_pipe(pipe_class_compare);
diff --git a/src/hotspot/cpu/ppc/sharedRuntime_ppc.cpp b/src/hotspot/cpu/ppc/sharedRuntime_ppc.cpp
index 3382df355da..5a080adc7a9 100644
--- a/src/hotspot/cpu/ppc/sharedRuntime_ppc.cpp
+++ b/src/hotspot/cpu/ppc/sharedRuntime_ppc.cpp
@@ -27,7 +27,6 @@
#include "asm/macroAssembler.inline.hpp"
#include "code/debugInfoRec.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "frame_ppc.hpp"
#include "compiler/oopMap.hpp"
@@ -35,7 +34,6 @@
#include "interpreter/interpreter.hpp"
#include "interpreter/interp_masm.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "prims/methodHandles.hpp"
#include "runtime/continuation.hpp"
@@ -1174,8 +1172,8 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
BLOCK_COMMENT("c2i unverified entry");
c2i_unverified_entry = __ pc();
- // inline_cache contains a compiledICHolder
- const Register ic = R19_method;
+ // inline_cache contains a CompiledICData
+ const Register ic = R19_inline_cache_reg;
const Register ic_klass = R11_scratch1;
const Register receiver_klass = R12_scratch2;
const Register code = R21_tmp1;
@@ -1186,45 +1184,10 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
Label call_interpreter;
- assert(!MacroAssembler::needs_explicit_null_check(oopDesc::klass_offset_in_bytes()),
- "klass offset should reach into any page");
- // Check for null argument if we don't have implicit null checks.
- if (!ImplicitNullChecks || !os::zero_page_read_protected()) {
- if (TrapBasedNullChecks) {
- __ trap_null_check(R3_ARG1);
- } else {
- Label valid;
- __ cmpdi(CCR0, R3_ARG1, 0);
- __ bne_predict_taken(CCR0, valid);
- // We have a null argument, branch to ic_miss_stub.
- __ b64_patchable((address)SharedRuntime::get_ic_miss_stub(),
- relocInfo::runtime_call_type);
- __ BIND(valid);
- }
- }
- // Assume argument is not null, load klass from receiver.
- __ load_klass(receiver_klass, R3_ARG1);
-
- __ ld(ic_klass, CompiledICHolder::holder_klass_offset(), ic);
-
- if (TrapBasedICMissChecks) {
- __ trap_ic_miss_check(receiver_klass, ic_klass);
- } else {
- Label valid;
- __ cmpd(CCR0, receiver_klass, ic_klass);
- __ beq_predict_taken(CCR0, valid);
- // We have an unexpected klass, branch to ic_miss_stub.
- __ b64_patchable((address)SharedRuntime::get_ic_miss_stub(),
- relocInfo::runtime_call_type);
- __ BIND(valid);
- }
-
+ __ ic_check(4 /* end_alignment */);
+ __ ld(R19_method, CompiledICData::speculated_method_offset(), ic);
// Argument is valid and klass is as expected, continue.
- // Extract method from inline cache, verified entry point needs it.
- __ ld(R19_method, CompiledICHolder::holder_metadata_offset(), ic);
- assert(R19_method == ic, "the inline cache register is dead here");
-
__ ld(code, method_(code));
__ cmpdi(CCR0, code, 0);
__ ld(ientry, method_(interpreter_entry)); // preloaded
@@ -1798,7 +1761,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
// static stub for the call above
CodeBuffer* cbuf = masm->code_section()->outer();
- stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, c2i_call_pc);
+ stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, c2i_call_pc);
guarantee(stub != nullptr, "no space for static stub");
}
@@ -1891,7 +1854,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
// static stub for the call above
CodeBuffer* cbuf = masm->code_section()->outer();
- stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, call_pc);
+ stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, call_pc);
guarantee(stub != nullptr, "no space for static stub");
}
@@ -2039,6 +2002,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
in_ByteSize(-1),
oop_maps,
exception_offset);
+ if (nm == nullptr) return nm;
if (method->is_continuation_enter_intrinsic()) {
ContinuationEntry::set_enter_code(nm, interpreted_entry_offset);
} else if (method->is_continuation_yield_intrinsic()) {
@@ -2187,7 +2151,6 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
intptr_t frame_done_pc;
intptr_t oopmap_pc;
- Label ic_miss;
Label handle_pending_exception;
Register r_callers_sp = R21;
@@ -2211,19 +2174,9 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
// Check ic: object class == cached class?
if (!method_is_static) {
- Register ic = R19_inline_cache_reg;
- Register receiver_klass = r_temp_1;
-
- __ cmpdi(CCR0, R3_ARG1, 0);
- __ beq(CCR0, ic_miss);
- __ verify_oop(R3_ARG1, FILE_AND_LINE);
- __ load_klass(receiver_klass, R3_ARG1);
-
- __ cmpd(CCR0, receiver_klass, ic);
- __ bne(CCR0, ic_miss);
+ __ ic_check(4 /* end_alignment */);
}
-
// Generate the Verified Entry Point (VEP).
// --------------------------------------------------------------------------
vep_start_pc = (intptr_t)__ pc();
@@ -2703,16 +2656,6 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
__ b64_patchable((address)StubRoutines::forward_exception_entry(),
relocInfo::runtime_call_type);
- // Handler for a cache miss (out-of-line).
- // --------------------------------------------------------------------------
-
- if (!method_is_static) {
- __ bind(ic_miss);
-
- __ b64_patchable((address)SharedRuntime::get_ic_miss_stub(),
- relocInfo::runtime_call_type);
- }
-
// Done.
// --------------------------------------------------------------------------
diff --git a/src/hotspot/cpu/ppc/stubGenerator_ppc.cpp b/src/hotspot/cpu/ppc/stubGenerator_ppc.cpp
index e26f03f52d8..094757ad3e1 100644
--- a/src/hotspot/cpu/ppc/stubGenerator_ppc.cpp
+++ b/src/hotspot/cpu/ppc/stubGenerator_ppc.cpp
@@ -3643,8 +3643,6 @@ class StubGenerator: public StubCodeGenerator {
#define VALID_B64 0x80
#define VB64(x) (VALID_B64 | x)
-#define VEC_ALIGN __attribute__ ((aligned(16)))
-
#define BLK_OFFSETOF(x) (offsetof(constant_block, x))
// In little-endian mode, the lxv instruction loads the element at EA into
@@ -3681,7 +3679,7 @@ class StubGenerator: public StubCodeGenerator {
unsigned char pack_permute_val[16];
} constant_block;
- static const constant_block VEC_ALIGN const_block = {
+ alignas(16) static const constant_block const_block = {
.offsetLUT_val = {
ARRAY_TO_LXV_ORDER(
@@ -4263,7 +4261,7 @@ class StubGenerator: public StubCodeGenerator {
unsigned char base64_48_63_URL_val[16];
} constant_block;
- static const constant_block VEC_ALIGN const_block = {
+ alignas(16) static const constant_block const_block = {
.expand_permute_val = {
ARRAY_TO_LXV_ORDER(
0, 4, 5, 6,
diff --git a/src/hotspot/cpu/ppc/templateTable_ppc_64.cpp b/src/hotspot/cpu/ppc/templateTable_ppc_64.cpp
index 84ecfc4f934..86353aefb36 100644
--- a/src/hotspot/cpu/ppc/templateTable_ppc_64.cpp
+++ b/src/hotspot/cpu/ppc/templateTable_ppc_64.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2013, 2023 SAP SE. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2013, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -3803,16 +3803,15 @@ void TemplateTable::_new() {
__ sldi(Roffset, Rindex, LogBytesPerWord);
__ load_resolved_klass_at_offset(Rcpool, Roffset, RinstanceKlass);
- // Make sure klass is fully initialized and get instance_size.
- __ lbz(Rscratch, in_bytes(InstanceKlass::init_state_offset()), RinstanceKlass);
+ // Make sure klass is initialized.
+ assert(VM_Version::supports_fast_class_init_checks(), "Optimization requires support for fast class initialization checks");
+ __ clinit_barrier(RinstanceKlass, R16_thread, nullptr /*L_fast_path*/, &Lslow_case);
+
__ lwz(Rinstance_size, in_bytes(Klass::layout_helper_offset()), RinstanceKlass);
- __ cmpdi(CCR1, Rscratch, InstanceKlass::fully_initialized);
// Make sure klass does not have has_finalizer, or is abstract, or interface or java/lang/Class.
__ andi_(R0, Rinstance_size, Klass::_lh_instance_slow_path_bit); // slow path bit equals 0?
-
- __ crnand(CCR0, Assembler::equal, CCR1, Assembler::equal); // slow path bit set or not fully initialized?
- __ beq(CCR0, Lslow_case);
+ __ bne(CCR0, Lslow_case);
// --------------------------------------------------------------------------
// Fast case:
diff --git a/src/hotspot/cpu/ppc/vtableStubs_ppc_64.cpp b/src/hotspot/cpu/ppc/vtableStubs_ppc_64.cpp
index fe4eb3df8f1..28ba04d833b 100644
--- a/src/hotspot/cpu/ppc/vtableStubs_ppc_64.cpp
+++ b/src/hotspot/cpu/ppc/vtableStubs_ppc_64.cpp
@@ -25,10 +25,10 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_ppc.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klass.inline.hpp"
#include "oops/klassVtable.hpp"
@@ -181,13 +181,13 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
__ load_klass_check_null(rcvr_klass, R3_ARG1);
// Receiver subtype check against REFC.
- __ ld(interface, CompiledICHolder::holder_klass_offset(), R19_method);
+ __ ld(interface, CompiledICData::itable_refc_klass_offset(), R19_method);
__ lookup_interface_method(rcvr_klass, interface, noreg,
R0, tmp1, tmp2,
L_no_such_interface, /*return_method=*/ false);
// Get Method* and entrypoint for compiler
- __ ld(interface, CompiledICHolder::holder_metadata_offset(), R19_method);
+ __ ld(interface, CompiledICData::itable_defc_klass_offset(), R19_method);
__ lookup_interface_method(rcvr_klass, interface, itable_index,
R19_method, tmp1, tmp2,
L_no_such_interface, /*return_method=*/ true);
diff --git a/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.cpp b/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.cpp
index 0bbf3771a04..e3ec023aef2 100644
--- a/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.cpp
+++ b/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.cpp
@@ -51,7 +51,6 @@
#endif
NEEDS_CLEANUP // remove this definitions ?
-const Register IC_Klass = t1; // where the IC klass is cached
const Register SYNC_header = x10; // synchronization header
const Register SHIFT_count = x10; // where count for shift operations must be
@@ -265,26 +264,7 @@ void LIR_Assembler::osr_entry() {
// inline cache check; done before the frame is built.
int LIR_Assembler::check_icache() {
- Register receiver = FrameMap::receiver_opr->as_register();
- Register ic_klass = IC_Klass;
- int start_offset = __ offset();
- Label dont;
- __ inline_cache_check(receiver, ic_klass, dont);
-
- // if icache check fails, then jump to runtime routine
- // Note: RECEIVER must still contain the receiver!
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- // We align the verified entry point unless the method body
- // (including its inline cache check) will fit in a single 64-byte
- // icache line.
- if (!method()->is_accessor() || __ offset() - start_offset > 4 * 4) {
- // force alignment after the cache check.
- __ align(CodeEntryAlignment);
- }
-
- __ bind(dont);
- return start_offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::jobject2reg(jobject o, Register reg) {
@@ -1398,7 +1378,7 @@ void LIR_Assembler::emit_static_call_stub() {
__ relocate(static_stub_Relocation::spec(call_pc));
__ emit_static_call_stub();
- assert(__ offset() - start + CompiledStaticCall::to_trampoline_stub_size()
+ assert(__ offset() - start + CompiledDirectCall::to_trampoline_stub_size()
<= call_stub_size(), "stub too big");
__ end_a_stub();
}
diff --git a/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.hpp b/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.hpp
index b088498e6fc..ce23213776c 100644
--- a/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.hpp
+++ b/src/hotspot/cpu/riscv/c1_LIRAssembler_riscv.hpp
@@ -68,7 +68,7 @@ friend class ArrayCopyStub;
enum {
// See emit_static_call_stub for detail
- // CompiledStaticCall::to_interp_stub_size() (14) + CompiledStaticCall::to_trampoline_stub_size() (1 + 3 + address)
+ // CompiledDirectCall::to_interp_stub_size() (14) + CompiledDirectCall::to_trampoline_stub_size() (1 + 3 + address)
_call_stub_size = 14 * NativeInstruction::instruction_size +
(NativeInstruction::instruction_size + NativeCallTrampolineStub::instruction_size),
// See emit_exception_handler for detail
diff --git a/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp
index 6c1dce0de15..2961b1a91ce 100644
--- a/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp
+++ b/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp
@@ -314,15 +314,6 @@ void C1_MacroAssembler::allocate_array(Register obj, Register len, Register tmp1
verify_oop(obj);
}
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache, Label &L) {
- verify_oop(receiver);
- // explicit null check not needed since load from [klass_offset] causes a trap
- // check against inline cache
- assert(!MacroAssembler::needs_explicit_null_check(oopDesc::klass_offset_in_bytes()), "must add explicit null check");
- assert_different_registers(receiver, iCache, t0, t2);
- cmp_klass(receiver, iCache, t0, t2 /* call-clobbered t2 as a tmp */, L);
-}
-
void C1_MacroAssembler::build_frame(int framesize, int bang_size_in_bytes) {
assert(bang_size_in_bytes >= framesize, "stack bang size incorrect");
// Make sure there is enough stack space for this method's activation.
diff --git a/src/hotspot/cpu/riscv/c1_Runtime1_riscv.cpp b/src/hotspot/cpu/riscv/c1_Runtime1_riscv.cpp
index b76163a3084..9fa8939837a 100644
--- a/src/hotspot/cpu/riscv/c1_Runtime1_riscv.cpp
+++ b/src/hotspot/cpu/riscv/c1_Runtime1_riscv.cpp
@@ -37,7 +37,6 @@
#include "interpreter/interpreter.hpp"
#include "memory/universe.hpp"
#include "nativeInst_riscv.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_riscv.hpp"
diff --git a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp
index 0617c37687f..9670bc987a3 100644
--- a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp
+++ b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2020, 2022, Huawei Technologies Co., Ltd. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -121,10 +121,10 @@ void C2_MacroAssembler::fast_lock(Register objectReg, Register boxReg,
// Handle existing monitor.
bind(object_has_monitor);
- // The object's monitor m is unlocked iff m->owner == NULL,
+ // The object's monitor m is unlocked iff m->owner == nullptr,
// otherwise m->owner may contain a thread or a stack address.
//
- // Try to CAS m->owner from NULL to current thread.
+ // Try to CAS m->owner from null to current thread.
add(tmp, disp_hdr, (in_bytes(ObjectMonitor::owner_offset()) - markWord::monitor_value));
cmpxchg(/*memory address*/tmp, /*expected value*/zr, /*new value*/xthread, Assembler::int64, Assembler::aq,
Assembler::rl, /*result*/flag); // cas succeeds if flag == zr(expected)
@@ -1358,7 +1358,6 @@ void C2_MacroAssembler::arrays_equals(Register a1, Register a2, Register tmp3,
// For Strings we're passed the address of the first characters in a1
// and a2 and the length in cnt1.
-// elem_size is the element size in bytes: either 1 or 2.
// There are two implementations. For arrays >= 8 bytes, all
// comparisons (for hw supporting unaligned access: including the final one,
// which may overlap) are performed 8 bytes at a time.
@@ -1367,13 +1366,12 @@ void C2_MacroAssembler::arrays_equals(Register a1, Register a2, Register tmp3,
// halfword, then a short, and then a byte.
void C2_MacroAssembler::string_equals(Register a1, Register a2,
- Register result, Register cnt1, int elem_size)
+ Register result, Register cnt1)
{
Label SAME, DONE, SHORT, NEXT_WORD;
Register tmp1 = t0;
Register tmp2 = t1;
- assert(elem_size == 1 || elem_size == 2, "must be 2 or 1 byte");
assert_different_registers(a1, a2, result, cnt1, tmp1, tmp2);
BLOCK_COMMENT("string_equals {");
@@ -1439,15 +1437,13 @@ void C2_MacroAssembler::string_equals(Register a1, Register a2,
}
bind(TAIL01);
- if (elem_size == 1) { // Only needed when comparing 1-byte elements
- // 0-1 bytes left.
- test_bit(tmp1, cnt1, 0);
- beqz(tmp1, SAME);
- {
- lbu(tmp1, Address(a1, 0));
- lbu(tmp2, Address(a2, 0));
- bne(tmp1, tmp2, DONE);
- }
+ // 0-1 bytes left.
+ test_bit(tmp1, cnt1, 0);
+ beqz(tmp1, SAME);
+ {
+ lbu(tmp1, Address(a1, 0));
+ lbu(tmp2, Address(a2, 0));
+ bne(tmp1, tmp2, DONE);
}
// Arrays are equal.
@@ -1829,6 +1825,49 @@ void C2_MacroAssembler::float16_to_float(FloatRegister dst, Register src, Regist
bind(stub->continuation());
}
+static void float_to_float16_slow_path(C2_MacroAssembler& masm, C2GeneralStub& stub) {
+#define __ masm.
+ Register dst = stub.data<0>();
+ FloatRegister src = stub.data<1>();
+ Register tmp = stub.data<2>();
+ __ bind(stub.entry());
+
+ __ fmv_x_w(dst, src);
+
+ // preserve the payloads of non-canonical NaNs.
+ __ srai(dst, dst, 13);
+ // preserve the sign bit.
+ __ srai(tmp, dst, 13);
+ __ slli(tmp, tmp, 10);
+ __ mv(t0, 0x3ff);
+ __ orr(tmp, tmp, t0);
+
+ // get the result by merging sign bit and payloads of preserved non-canonical NaNs.
+ __ andr(dst, dst, tmp);
+
+ __ j(stub.continuation());
+#undef __
+}
+
+// j.l.Float.floatToFloat16
+void C2_MacroAssembler::float_to_float16(Register dst, FloatRegister src, FloatRegister ftmp, Register xtmp) {
+ auto stub = C2CodeStub::make(dst, src, xtmp, 130, float_to_float16_slow_path);
+
+ // in riscv, NaN needs a special process as fcvt does not work in that case.
+
+ // check whether it's a NaN.
+ // replace fclass with feq as performance optimization.
+ feq_s(t0, src, src);
+ // jump to stub processing NaN cases.
+ beqz(t0, stub->entry());
+
+ // non-NaN cases, just use built-in instructions.
+ fcvt_h_s(ftmp, src);
+ fmv_x_h(dst, ftmp);
+
+ bind(stub->continuation());
+}
+
void C2_MacroAssembler::signum_fp_v(VectorRegister dst, VectorRegister one, BasicType bt, int vlen) {
vsetvli_helper(bt, vlen);
@@ -1942,7 +1981,7 @@ void C2_MacroAssembler::element_compare(Register a1, Register a2, Register resul
mv(result, true);
}
-void C2_MacroAssembler::string_equals_v(Register a1, Register a2, Register result, Register cnt, int elem_size) {
+void C2_MacroAssembler::string_equals_v(Register a1, Register a2, Register result, Register cnt) {
Label DONE;
Register tmp1 = t0;
Register tmp2 = t1;
@@ -1951,11 +1990,7 @@ void C2_MacroAssembler::string_equals_v(Register a1, Register a2, Register resul
mv(result, false);
- if (elem_size == 2) {
- srli(cnt, cnt, 1);
- }
-
- element_compare(a1, a2, result, cnt, tmp1, tmp2, v2, v4, v2, elem_size == 1, DONE);
+ element_compare(a1, a2, result, cnt, tmp1, tmp2, v2, v4, v2, true, DONE);
bind(DONE);
BLOCK_COMMENT("} string_equals_v");
diff --git a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp
index 7309c59110a..9fe4dc002c9 100644
--- a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp
+++ b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp
@@ -92,8 +92,7 @@
void arrays_hashcode_elload(Register dst, Address src, BasicType eltype);
void string_equals(Register r1, Register r2,
- Register result, Register cnt1,
- int elem_size);
+ Register result, Register cnt1);
// refer to conditional_branches and float_conditional_branches
static const int bool_test_bits = 3;
@@ -173,6 +172,7 @@
void signum_fp(FloatRegister dst, FloatRegister one, bool is_double);
void float16_to_float(FloatRegister dst, Register src, Register tmp);
+ void float_to_float16(Register dst, FloatRegister src, FloatRegister ftmp, Register xtmp);
void signum_fp_v(VectorRegister dst, VectorRegister one, BasicType bt, int vlen);
@@ -187,8 +187,7 @@
void expand_bits_l_v(Register dst, Register src, Register mask);
void string_equals_v(Register r1, Register r2,
- Register result, Register cnt1,
- int elem_size);
+ Register result, Register cnt1);
void arrays_equals_v(Register r1, Register r2,
Register result, Register cnt1,
diff --git a/src/hotspot/cpu/riscv/compiledIC_riscv.cpp b/src/hotspot/cpu/riscv/compiledIC_riscv.cpp
index e29dee56de8..fdb2bcb06ff 100644
--- a/src/hotspot/cpu/riscv/compiledIC_riscv.cpp
+++ b/src/hotspot/cpu/riscv/compiledIC_riscv.cpp
@@ -27,7 +27,6 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
@@ -37,7 +36,7 @@
// ----------------------------------------------------------------------------
#define __ _masm.
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
precond(cbuf.stubs()->start() != badAddress);
precond(cbuf.stubs()->end() != badAddress);
// Stub is fixed up when the corresponding call is converted from
@@ -69,11 +68,11 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark)
}
#undef __
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return MacroAssembler::static_call_stub_size();
}
-int CompiledStaticCall::to_trampoline_stub_size() {
+int CompiledDirectCall::to_trampoline_stub_size() {
// Somewhat pessimistically, we count 4 instructions here (although
// there are only 3) because we sometimes emit an alignment nop.
// Trampoline stubs are always word aligned.
@@ -81,21 +80,14 @@ int CompiledStaticCall::to_trampoline_stub_size() {
}
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 4; // 3 in emit_to_interp_stub + 1 in emit_call
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder
= nativeMovConstReg_at(stub);
@@ -112,7 +104,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
// Reset stub.
address stub = static_stub->addr();
assert(stub != nullptr, "stub not found");
@@ -129,7 +121,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/riscv/icBuffer_riscv.cpp b/src/hotspot/cpu/riscv/icBuffer_riscv.cpp
deleted file mode 100644
index ab904817816..00000000000
--- a/src/hotspot/cpu/riscv/icBuffer_riscv.cpp
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2014, Red Hat Inc. All rights reserved.
- * Copyright (c) 2020, 2023, Huawei Technologies Co., Ltd. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/macroAssembler.hpp"
-#include "asm/macroAssembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_riscv.hpp"
-#include "oops/oop.inline.hpp"
-
-int InlineCacheBuffer::ic_stub_code_size() {
- // 6: auipc + ld + auipc + jalr + address(2 * instruction_size)
- return 6 * NativeInstruction::instruction_size;
-}
-
-#define __ masm->
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point) {
- assert_cond(code_begin != nullptr && entry_point != nullptr);
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler* masm = new MacroAssembler(&code);
- // Note: even though the code contains an embedded value, we do not need reloc info
- // because
- // (1) the value is old (i.e., doesn't matter for scavenges)
- // (2) these ICStubs are removed *before* a GC happens, so the roots disappear
-
- address start = __ pc();
- Label l;
- __ ld(t1, l);
- __ far_jump(ExternalAddress(entry_point));
- __ align(wordSize);
- __ bind(l);
- __ emit_int64((intptr_t)cached_value);
- // Only need to invalidate the 1st two instructions - not the whole ic stub
- ICache::invalidate_range(code_begin, InlineCacheBuffer::ic_stub_code_size());
- assert(__ pc() - start == ic_stub_code_size(), "must be");
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // creation also verifies the object
- NativeJump* jump = nativeJump_at(move->next_instruction_address());
- return jump->jump_destination();
-}
-
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- // The word containing the cached value is at the end of this IC buffer
- uintptr_t *p = (uintptr_t *)(code_begin + ic_stub_code_size() - wordSize);
- void* o = (void*)*p;
- return o;
-}
diff --git a/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp
index ce336c16aa7..96e07319e84 100644
--- a/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp
+++ b/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp
@@ -27,6 +27,7 @@
#include "precompiled.hpp"
#include "asm/assembler.hpp"
#include "asm/assembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/disassembler.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
@@ -634,8 +635,8 @@ void MacroAssembler::unimplemented(const char* what) {
}
void MacroAssembler::emit_static_call_stub() {
- IncompressibleRegion ir(this); // Fixed length: see CompiledStaticCall::to_interp_stub_size().
- // CompiledDirectStaticCall::set_to_interpreted knows the
+ IncompressibleRegion ir(this); // Fixed length: see CompiledDirectCall::to_interp_stub_size().
+ // CompiledDirectCall::set_to_interpreted knows the
// exact layout of this stub.
mov_metadata(xmethod, (Metadata*)nullptr);
@@ -2542,7 +2543,7 @@ void MacroAssembler::lookup_interface_method(Register recv_klass,
}
// Look up the method for a megamorphic invokeinterface call in a single pass over itable:
-// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICHolder
+// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICData
// - find a holder_klass (class that implements the method) vtable offset and get the method from vtable by index
// The target method is determined by .
// The receiver klass is in recv_klass.
@@ -3542,6 +3543,48 @@ address MacroAssembler::ic_call(address entry, jint method_index) {
return trampoline_call(Address(entry, rh));
}
+int MacroAssembler::ic_check_size() {
+ // No compressed
+ return (NativeInstruction::instruction_size * (2 /* 2 loads */ + 1 /* branch */)) +
+ far_branch_size();
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ IncompressibleRegion ir(this);
+ Register receiver = j_rarg0;
+ Register data = t1;
+
+ Register tmp1 = t0; // t0 always scratch
+ // t2 is saved on call, thus should have been saved before this check.
+ // Hence we can clobber it.
+ Register tmp2 = t2;
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, ic_check_size());
+ int uep_offset = offset();
+
+ if (UseCompressedClassPointers) {
+ lwu(tmp1, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ lwu(tmp2, Address(data, CompiledICData::speculated_klass_offset()));
+ } else {
+ ld(tmp1, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ ld(tmp2, Address(data, CompiledICData::speculated_klass_offset()));
+ }
+
+ Label ic_hit;
+ beq(tmp1, tmp2, ic_hit);
+ // Note, far_jump is not fixed size.
+ // Is this ever generates a movptr alignment/size will be off.
+ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
+ bind(ic_hit);
+
+ assert((offset() % end_alignment) == 0, "Misaligned verified entry point.");
+ return uep_offset;
+}
+
// Emit a trampoline stub for a call to a target which is too far away.
//
// code sequences:
diff --git a/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp b/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp
index d283654e6e1..63cfb228551 100644
--- a/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp
+++ b/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp
@@ -1193,7 +1193,10 @@ class MacroAssembler: public Assembler {
//
// Return: the call PC or null if CodeCache is full.
address trampoline_call(Address entry);
+
address ic_call(address entry, jint method_index = 0);
+ static int ic_check_size();
+ int ic_check(int end_alignment = NativeInstruction::instruction_size);
// Support for memory inc/dec
// n.b. increment/decrement calls with an Address destination will
diff --git a/src/hotspot/cpu/riscv/riscv.ad b/src/hotspot/cpu/riscv/riscv.ad
index 550e7947cc5..10a80cd0940 100644
--- a/src/hotspot/cpu/riscv/riscv.ad
+++ b/src/hotspot/cpu/riscv/riscv.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved.
// Copyright (c) 2020, 2023, Huawei Technologies Co., Ltd. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
@@ -1196,13 +1196,13 @@ bool is_CAS(int opcode, bool maybe_volatile)
// returns true if CAS needs to use an acquiring load otherwise false
bool needs_acquiring_load_reserved(const Node *n)
{
- assert(n != NULL && is_CAS(n->Opcode(), true), "expecting a compare and swap");
+ assert(n != nullptr && is_CAS(n->Opcode(), true), "expecting a compare and swap");
LoadStoreNode* ldst = n->as_LoadStore();
- if (n != NULL && is_CAS(n->Opcode(), false)) {
- assert(ldst != NULL && ldst->trailing_membar() != NULL, "expected trailing membar");
+ if (n != nullptr && is_CAS(n->Opcode(), false)) {
+ assert(ldst != nullptr && ldst->trailing_membar() != nullptr, "expected trailing membar");
} else {
- return ldst != NULL && ldst->trailing_membar() != NULL;
+ return ldst != nullptr && ldst->trailing_membar() != nullptr;
}
// so we can just return true here
return true;
@@ -1247,7 +1247,7 @@ int MachCallRuntimeNode::ret_addr_offset() {
// sd(t1, Address(sp, wordSize)) -> sd
// jalr(t0) -> jalr
CodeBlob *cb = CodeCache::find_blob(_entry_point);
- if (cb != NULL) {
+ if (cb != nullptr) {
return 1 * NativeInstruction::instruction_size;
} else {
return 11 * NativeInstruction::instruction_size;
@@ -1286,7 +1286,7 @@ int CallDynamicJavaDirectNode::compute_padding(int current_offset) const
#ifndef PRODUCT
void MachBreakpointNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
- assert_cond(st != NULL);
+ assert_cond(st != nullptr);
st->print("BREAKPOINT");
}
#endif
@@ -1342,14 +1342,14 @@ uint MachConstantBaseNode::size(PhaseRegAlloc* ra_) const {
#ifndef PRODUCT
void MachConstantBaseNode::format(PhaseRegAlloc* ra_, outputStream* st) const {
- assert_cond(st != NULL);
+ assert_cond(st != nullptr);
st->print("-- \t// MachConstantBaseNode (empty encoding)");
}
#endif
#ifndef PRODUCT
void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
- assert_cond(st != NULL && ra_ != NULL);
+ assert_cond(st != nullptr && ra_ != nullptr);
Compile* C = ra_->C;
int framesize = C->output()->frame_slots() << LogBytesPerInt;
@@ -1363,7 +1363,7 @@ void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
if (PreserveFramePointer) { st->print("sub fp, sp, #%d\n\t", 2 * wordSize); }
st->print("sub sp, sp, #%d\n\t", framesize);
- if (C->stub_function() == NULL && BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (C->stub_function() == nullptr && BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
st->print("ld t0, [guard]\n\t");
st->print("membar LoadLoad\n\t");
st->print("ld t1, [xthread, #thread_disarmed_guard_value_offset]\n\t");
@@ -1377,7 +1377,7 @@ void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
#endif
void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
Compile* C = ra_->C;
C2_MacroAssembler _masm(&cbuf);
@@ -1392,7 +1392,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ nop(); // 4 bytes
}
- assert_cond(C != NULL);
+ assert_cond(C != nullptr);
if (C->clinit_barrier_on_entry()) {
assert(!C->method()->holder()->is_not_initialized(), "initialization should have been started");
@@ -1412,9 +1412,9 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ build_frame(framesize);
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
- if (BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
// Dummy labels for just measuring the code size
Label dummy_slow_path;
Label dummy_continuation;
@@ -1451,7 +1451,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
uint MachPrologNode::size(PhaseRegAlloc* ra_) const
{
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
return MachNode::size(ra_); // too many variables; just compute it
// the hard way
}
@@ -1465,9 +1465,9 @@ int MachPrologNode::reloc() const
#ifndef PRODUCT
void MachEpilogNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
- assert_cond(st != NULL && ra_ != NULL);
+ assert_cond(st != nullptr && ra_ != nullptr);
Compile* C = ra_->C;
- assert_cond(C != NULL);
+ assert_cond(C != nullptr);
int framesize = C->output()->frame_size_in_bytes();
st->print("# pop frame %d\n\t", framesize);
@@ -1491,10 +1491,10 @@ void MachEpilogNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
#endif
void MachEpilogNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
Compile* C = ra_->C;
C2_MacroAssembler _masm(&cbuf);
- assert_cond(C != NULL);
+ assert_cond(C != nullptr);
int framesize = C->output()->frame_size_in_bytes();
__ remove_frame(framesize);
@@ -1517,7 +1517,7 @@ void MachEpilogNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
}
uint MachEpilogNode::size(PhaseRegAlloc *ra_) const {
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
// Variable size. Determine dynamically.
return MachNode::size(ra_);
}
@@ -1568,7 +1568,7 @@ static enum RC rc_class(OptoReg::Name reg) {
}
uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, bool do_size, outputStream *st) const {
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
Compile* C = ra_->C;
// Get registers to move.
@@ -1599,7 +1599,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
int src_offset = ra_->reg2offset(src_lo);
int dst_offset = ra_->reg2offset(dst_lo);
- if (bottom_type()->isa_vect() != NULL) {
+ if (bottom_type()->isa_vect() != nullptr) {
uint ireg = ideal_reg();
if (ireg == Op_VecA && cbuf) {
C2_MacroAssembler _masm(cbuf);
@@ -1640,7 +1640,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
ShouldNotReachHere();
}
}
- } else if (cbuf != NULL) {
+ } else if (cbuf != nullptr) {
C2_MacroAssembler _masm(cbuf);
switch (src_lo_rc) {
case rc_int:
@@ -1711,7 +1711,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
}
}
- if (st != NULL) {
+ if (st != nullptr) {
st->print("spill ");
if (src_lo_rc == rc_stack) {
st->print("[sp, #%d] -> ", src_offset);
@@ -1745,16 +1745,16 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
#ifndef PRODUCT
void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
- if (ra_ == NULL) {
+ if (ra_ == nullptr) {
st->print("N%d = SpillCopy(N%d)", _idx, in(1)->_idx);
} else {
- implementation(NULL, ra_, false, st);
+ implementation(nullptr, ra_, false, st);
}
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation(&cbuf, ra_, false, NULL);
+ implementation(&cbuf, ra_, false, nullptr);
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
@@ -1765,7 +1765,7 @@ uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
#ifndef PRODUCT
void BoxLockNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
- assert_cond(ra_ != NULL && st != NULL);
+ assert_cond(ra_ != nullptr && st != nullptr);
int offset = ra_->reg2offset(in_RegMask(0).find_first_elem());
int reg = ra_->get_reg_first(this);
st->print("add %s, sp, #%d\t# box lock",
@@ -1777,7 +1777,7 @@ void BoxLockNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
C2_MacroAssembler _masm(&cbuf);
Assembler::IncompressibleRegion ir(&_masm); // Fixed length: see BoxLockNode::size()
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
int offset = ra_->reg2offset(in_RegMask(0).find_first_elem());
int reg = ra_->get_encode(this);
@@ -1805,17 +1805,16 @@ uint BoxLockNode::size(PhaseRegAlloc *ra_) const {
#ifndef PRODUCT
void MachUEPNode::format(PhaseRegAlloc* ra_, outputStream* st) const
{
- assert_cond(st != NULL);
+ assert_cond(st != nullptr);
st->print_cr("# MachUEPNode");
if (UseCompressedClassPointers) {
- st->print_cr("\tlwu t0, [j_rarg0, oopDesc::klass_offset_in_bytes()]\t# compressed klass");
- if (CompressedKlassPointers::shift() != 0) {
- st->print_cr("\tdecode_klass_not_null t0, t0");
- }
+ st->print_cr("\tlwu t0, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tlwu t2, [t1 + CompiledICData::speculated_klass_offset()]\t# compressed klass");
} else {
- st->print_cr("\tld t0, [j_rarg0, oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tld t0, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tld t2, [t1 + CompiledICData::speculated_klass_offset()]\t# compressed klass");
}
- st->print_cr("\tbeq t0, t1, ic_hit");
+ st->print_cr("\tbeq t0, t2, ic_hit");
st->print_cr("\tj, SharedRuntime::_ic_miss_stub\t # Inline cache check");
st->print_cr("\tic_hit:");
}
@@ -1825,20 +1824,16 @@ void MachUEPNode::emit(CodeBuffer& cbuf, PhaseRegAlloc* ra_) const
{
// This is the unverified entry point.
C2_MacroAssembler _masm(&cbuf);
+ __ ic_check(CodeEntryAlignment);
- Label skip;
- __ cmp_klass(j_rarg0, t1, t0, t2 /* call-clobbered t2 as a tmp */, skip);
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
- __ bind(skip);
-
- // These NOPs are critical so that verified entry point is properly
- // 4 bytes aligned for patching by NativeJump::patch_verified_entry()
- __ align(NativeInstruction::instruction_size);
+ // Verified entry point must be properly 4 bytes aligned for patching by NativeJump::patch_verified_entry().
+ // ic_check() aligns to CodeEntryAlignment >= InteriorEntryAlignment(min 16) > NativeInstruction::instruction_size(4).
+ assert(((__ offset()) % CodeEntryAlignment) == 0, "Misaligned verified entry point");
}
uint MachUEPNode::size(PhaseRegAlloc* ra_) const
{
- assert_cond(ra_ != NULL);
+ assert_cond(ra_ != nullptr);
return MachNode::size(ra_);
}
@@ -1855,7 +1850,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer& cbuf)
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -1873,7 +1868,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf)
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -1934,6 +1929,7 @@ bool Matcher::match_rule_supported(int opcode) {
return UseFMA;
case Op_ConvHF2F:
+ case Op_ConvF2HF:
return UseZfh;
}
@@ -2016,7 +2012,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return MIN2(size, max_size);
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
return Matcher::max_vector_size(bt);
}
@@ -2037,7 +2033,7 @@ int Matcher::scalable_vector_reg_size(const BasicType bt) {
MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* original_opnd, uint ideal_reg, bool is_temp) {
ShouldNotReachHere(); // generic vector operands not supported
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -2140,10 +2136,10 @@ const RegMask Matcher::method_handle_invoke_SP_save_mask() {
}
bool size_fits_all_mem_uses(AddPNode* addp, int shift) {
- assert_cond(addp != NULL);
+ assert_cond(addp != nullptr);
for (DUIterator_Fast imax, i = addp->fast_outs(imax); i < imax; i++) {
Node* u = addp->fast_out(i);
- if (u != NULL && u->is_Mem()) {
+ if (u != nullptr && u->is_Mem()) {
int opsize = u->as_Mem()->memory_size();
assert(opsize > 0, "unexpected memory operand size");
if (u->as_Mem()->memory_size() != (1 << shift)) {
@@ -2156,7 +2152,7 @@ bool size_fits_all_mem_uses(AddPNode* addp, int shift) {
// Should the Matcher clone input 'm' of node 'n'?
bool Matcher::pd_clone_node(Node* n, Node* m, Matcher::MStack& mstack) {
- assert_cond(m != NULL);
+ assert_cond(m != nullptr);
if (is_vshift_con_pattern(n, m)) { // ShiftV src (ShiftCntV con)
mstack.push(m, Visit); // m = ShiftCntV
return true;
@@ -2215,7 +2211,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL || con == (address)1) {
+ if (con == nullptr || con == (address)1) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -2245,7 +2241,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -2264,7 +2260,7 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Register dst_reg = as_Register($dst$$reg);
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
relocInfo::relocType rtype = $src->constant_reloc();
@@ -2350,7 +2346,7 @@ encode %{
Label done;
C2_MacroAssembler _masm(&cbuf);
__ check_klass_subtype_slow_path(sub_reg, super_reg, temp_reg, result_reg,
- NULL, &miss);
+ nullptr, &miss);
if ($primary) {
__ mv(result_reg, zr);
} else {
@@ -2371,12 +2367,12 @@ encode %{
Assembler::IncompressibleRegion ir(&_masm); // Fixed length: see ret_addr_offset
address addr = (address)$meth$$method;
- address call = NULL;
- assert_cond(addr != NULL);
+ address call = nullptr;
+ assert_cond(addr != nullptr);
if (!_method) {
// A call to a runtime wrapper, e.g. new, new_typeArray_Java, uncommon_trap.
call = __ trampoline_call(Address(addr, relocInfo::runtime_call_type));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2390,7 +2386,7 @@ encode %{
RelocationHolder rspec = _optimized_virtual ? opt_virtual_call_Relocation::spec(method_index)
: static_call_Relocation::spec(method_index);
call = __ trampoline_call(Address(addr, rspec));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2401,8 +2397,8 @@ encode %{
cbuf.shared_stub_to_interp_for(_method, call - cbuf.insts_begin());
} else {
// Emit stub for static call
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf, call);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf, call);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2417,7 +2413,7 @@ encode %{
Assembler::IncompressibleRegion ir(&_masm); // Fixed length: see ret_addr_offset
int method_index = resolved_method_index(cbuf);
address call = __ ic_call((address)$meth$$method, method_index);
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2443,9 +2439,9 @@ encode %{
// which loads the absolute address into a register.
address entry = (address)$meth$$method;
CodeBlob *cb = CodeCache::find_blob(entry);
- if (cb != NULL) {
+ if (cb != nullptr) {
address call = __ trampoline_call(Address(entry, relocInfo::runtime_call_type));
- if (call == NULL) {
+ if (call == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2893,7 +2889,7 @@ operand immP()
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immP0()
%{
predicate(n->get_ptr() == 0);
@@ -3014,7 +3010,7 @@ operand immN()
interface(CONST_INTER);
%}
-// Narrow NULL Pointer Immediate
+// Narrow Null Pointer Immediate
operand immN0()
%{
predicate(n->get_narrowcon() == 0);
@@ -4892,7 +4888,7 @@ instruct loadConP0(iRegPNoSp dst, immP0 con)
match(Set dst con);
ins_cost(ALU_COST);
- format %{ "mv $dst, $con\t# NULL ptr, #@loadConP0" %}
+ format %{ "mv $dst, $con\t# null pointer, #@loadConP0" %}
ins_encode(riscv_enc_mov_zero(dst));
@@ -4943,7 +4939,7 @@ instruct loadConN0(iRegNNoSp dst, immN0 con)
match(Set dst con);
ins_cost(ALU_COST);
- format %{ "mv $dst, $con\t# compressed NULL ptr, #@loadConN0" %}
+ format %{ "mv $dst, $con\t# compressed null pointer, #@loadConN0" %}
ins_encode(riscv_enc_mov_zero(dst));
@@ -8292,6 +8288,18 @@ instruct convHF2F_reg_reg(fRegF dst, iRegINoSp src, iRegINoSp tmp) %{
ins_pipe(pipe_slow);
%}
+instruct convF2HF_reg_reg(iRegINoSp dst, fRegF src, fRegF ftmp, iRegINoSp xtmp) %{
+ match(Set dst (ConvF2HF src));
+ effect(TEMP_DEF dst, TEMP ftmp, TEMP xtmp);
+ format %{ "fcvt.h.s $ftmp, $src\t# convert single precision to half\n\t"
+ "fmv.x.h $dst, $ftmp\t# move result from $ftmp to $dst"
+ %}
+ ins_encode %{
+ __ float_to_float16($dst$$Register, $src$$FloatRegister, $ftmp$$FloatRegister, $xtmp$$Register);
+ %}
+ ins_pipe(pipe_slow);
+%}
+
// float <-> int
instruct convF2I_reg_reg(iRegINoSp dst, fRegF src) %{
@@ -10325,7 +10333,7 @@ instruct clearArray_reg_reg(iRegL_R29 cnt, iRegP_R28 base, iRegP_R30 tmp1,
ins_encode %{
address tpc = __ zero_words($base$$Register, $cnt$$Register);
- if (tpc == NULL) {
+ if (tpc == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -10362,23 +10370,7 @@ instruct string_equalsL(iRegP_R11 str1, iRegP_R13 str2, iRegI_R14 cnt,
ins_encode %{
// Count is in 8-bit bytes; non-Compact chars are 16 bits.
__ string_equals($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 1);
- %}
- ins_pipe(pipe_class_memory);
-%}
-
-instruct string_equalsU(iRegP_R11 str1, iRegP_R13 str2, iRegI_R14 cnt,
- iRegI_R10 result, rFlagsReg cr)
-%{
- predicate(!UseRVV && ((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU);
- match(Set result (StrEquals (Binary str1 str2) cnt));
- effect(USE_KILL str1, USE_KILL str2, USE_KILL cnt, KILL cr);
-
- format %{ "String Equals $str1, $str2, $cnt -> $result\t#@string_equalsU" %}
- ins_encode %{
- // Count is in 8-bit bytes; non-Compact chars are 16 bits.
- __ string_equals($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 2);
+ $result$$Register, $cnt$$Register);
%}
ins_pipe(pipe_class_memory);
%}
diff --git a/src/hotspot/cpu/riscv/riscv_v.ad b/src/hotspot/cpu/riscv/riscv_v.ad
index c163325fc81..d07f4dc7c9a 100644
--- a/src/hotspot/cpu/riscv/riscv_v.ad
+++ b/src/hotspot/cpu/riscv/riscv_v.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2020, 2023, Arm Limited. All rights reserved.
// Copyright (c) 2020, 2022, Huawei Technologies Co., Ltd. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
@@ -46,7 +46,7 @@ source %{
}
}
- bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+ bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
return match_rule_supported_vector(opcode, vlen, bt);
}
@@ -2614,24 +2614,7 @@ instruct vstring_equalsL(iRegP_R11 str1, iRegP_R13 str2, iRegI_R14 cnt,
ins_encode %{
// Count is in 8-bit bytes; non-Compact chars are 16 bits.
__ string_equals_v($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 1);
- %}
- ins_pipe(pipe_class_memory);
-%}
-
-instruct vstring_equalsU(iRegP_R11 str1, iRegP_R13 str2, iRegI_R14 cnt,
- iRegI_R10 result, vReg_V2 v2,
- vReg_V3 v3, vReg_V4 v4, vReg_V5 v5, rFlagsReg cr)
-%{
- predicate(UseRVV && ((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU);
- match(Set result (StrEquals (Binary str1 str2) cnt));
- effect(USE_KILL str1, USE_KILL str2, USE_KILL cnt, TEMP v2, TEMP v3, TEMP v4, TEMP v5, KILL cr);
-
- format %{ "String Equals $str1, $str2, $cnt -> $result\t#@string_equalsU" %}
- ins_encode %{
- // Count is in 8-bit bytes; non-Compact chars are 16 bits.
- __ string_equals_v($str1$$Register, $str2$$Register,
- $result$$Register, $cnt$$Register, 2);
+ $result$$Register, $cnt$$Register);
%}
ins_pipe(pipe_class_memory);
%}
@@ -3217,7 +3200,7 @@ instruct vcvtBtoX(vReg dst, vReg src) %{
__ csrwi(CSR_FRM, C2_MacroAssembler::rne);
__ vfcvt_f_x_v(as_VectorRegister($dst$$reg), as_VectorRegister($dst$$reg));
} else {
- __ integer_extend_v(as_VectorRegister($dst$$reg), bt,
+ __ integer_extend_v(as_VectorRegister($dst$$reg), bt,
Matcher::vector_length(this), as_VectorRegister($src$$reg), T_BYTE);
}
%}
diff --git a/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp b/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp
index 01fe307bc27..7435b552d15 100644
--- a/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp
+++ b/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp
@@ -29,7 +29,6 @@
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
@@ -38,7 +37,6 @@
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
#include "nativeInst_riscv.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
#include "prims/methodHandles.hpp"
@@ -622,10 +620,8 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
address c2i_unverified_entry = __ pc();
Label skip_fixup;
- Label ok;
-
- const Register holder = t1;
const Register receiver = j_rarg0;
+ const Register data = t1;
const Register tmp = t2; // A call-clobbered register not used for arg passing
// -------------------------------------------------------------------------
@@ -639,16 +635,10 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
{
__ block_comment("c2i_unverified_entry {");
- __ load_klass(t0, receiver, tmp);
- __ ld(tmp, Address(holder, CompiledICHolder::holder_klass_offset()));
- __ ld(xmethod, Address(holder, CompiledICHolder::holder_metadata_offset()));
- __ beq(t0, tmp, ok);
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
- __ bind(ok);
- // Method might have been compiled since the call site was patched to
- // interpreted; if that is the case treat it as a miss so we can get
- // the call site corrected.
+ __ ic_check();
+ __ ld(xmethod, Address(data, CompiledICData::speculated_method_offset()));
+
__ ld(t0, Address(xmethod, in_bytes(Method::code_offset())));
__ beqz(t0, skip_fixup);
__ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
@@ -985,7 +975,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
__ j(exit);
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, tr_call);
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, tr_call);
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1051,7 +1041,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
}
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, tr_call);
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, tr_call);
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1271,6 +1261,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
in_ByteSize(-1),
oop_maps,
exception_offset);
+ if (nm == nullptr) return nm;
if (method->is_continuation_enter_intrinsic()) {
ContinuationEntry::set_enter_code(nm, interpreted_entry_offset);
} else if (method->is_continuation_yield_intrinsic()) {
@@ -1424,19 +1415,10 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
const Register ic_reg = t1;
const Register receiver = j_rarg0;
- Label hit;
- Label exception_pending;
-
__ verify_oop(receiver);
- assert_different_registers(ic_reg, receiver, t0, t2);
- __ cmp_klass(receiver, ic_reg, t0, t2 /* call-clobbered t2 as a tmp */, hit);
-
- __ far_jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
+ assert_different_registers(receiver, t0, t1);
- // Verified entry point must be aligned
- __ align(8);
-
- __ bind(hit);
+ __ ic_check();
int vep_offset = ((intptr_t)__ pc()) - start;
@@ -1871,6 +1853,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
__ leave();
// Any exception pending?
+ Label exception_pending;
__ ld(t0, Address(xthread, in_bytes(Thread::pending_exception_offset())));
__ bnez(t0, exception_pending);
diff --git a/src/hotspot/cpu/riscv/templateTable_riscv.cpp b/src/hotspot/cpu/riscv/templateTable_riscv.cpp
index 73b4d1e28cc..58f57f32b2f 100644
--- a/src/hotspot/cpu/riscv/templateTable_riscv.cpp
+++ b/src/hotspot/cpu/riscv/templateTable_riscv.cpp
@@ -3547,11 +3547,10 @@ void TemplateTable::_new() {
// get InstanceKlass
__ load_resolved_klass_at_offset(x14, x13, x14, t0);
- // make sure klass is initialized & doesn't have finalizer
- // make sure klass is fully initialized
- __ lbu(t0, Address(x14, InstanceKlass::init_state_offset()));
- __ sub(t1, t0, (u1)InstanceKlass::fully_initialized);
- __ bnez(t1, slow_case);
+ // make sure klass is initialized
+ assert(VM_Version::supports_fast_class_init_checks(),
+ "Optimization requires support for fast class initialization checks");
+ __ clinit_barrier(x14, t0, nullptr /*L_fast_path*/, &slow_case);
// get instance_size in InstanceKlass (scaled to a count of bytes)
__ lwu(x13, Address(x14, Klass::layout_helper_offset()));
diff --git a/src/hotspot/cpu/riscv/vtableStubs_riscv.cpp b/src/hotspot/cpu/riscv/vtableStubs_riscv.cpp
index 9d08796681f..5d945dbc323 100644
--- a/src/hotspot/cpu/riscv/vtableStubs_riscv.cpp
+++ b/src/hotspot/cpu/riscv/vtableStubs_riscv.cpp
@@ -27,10 +27,10 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_riscv.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klassVtable.hpp"
#include "runtime/sharedRuntime.hpp"
@@ -171,22 +171,22 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
assert(VtableStub::receiver_location() == j_rarg0->as_VMReg(), "receiver expected in j_rarg0");
// Entry arguments:
- // t1: CompiledICHolder
+ // t1: CompiledICData
// j_rarg0: Receiver
// This stub is called from compiled code which has no callee-saved registers,
// so all registers except arguments are free at this point.
const Register recv_klass_reg = x18;
- const Register holder_klass_reg = x19; // declaring interface klass (DECC)
+ const Register holder_klass_reg = x19; // declaring interface klass (DEFC)
const Register resolved_klass_reg = x30; // resolved interface klass (REFC)
const Register temp_reg = x28;
const Register temp_reg2 = x29;
- const Register icholder_reg = t1;
+ const Register icdata_reg = t1;
Label L_no_such_interface;
- __ ld(resolved_klass_reg, Address(icholder_reg, CompiledICHolder::holder_klass_offset()));
- __ ld(holder_klass_reg, Address(icholder_reg, CompiledICHolder::holder_metadata_offset()));
+ __ ld(resolved_klass_reg, Address(icdata_reg, CompiledICData::itable_refc_klass_offset()));
+ __ ld(holder_klass_reg, Address(icdata_reg, CompiledICData::itable_defc_klass_offset()));
start_pc = __ pc();
diff --git a/src/hotspot/cpu/s390/assembler_s390.hpp b/src/hotspot/cpu/s390/assembler_s390.hpp
index 9bb143001b9..91cc7e611bf 100644
--- a/src/hotspot/cpu/s390/assembler_s390.hpp
+++ b/src/hotspot/cpu/s390/assembler_s390.hpp
@@ -107,7 +107,7 @@ class RelAddr {
static bool is_in_range_of_RelAddr(address target, address pc, bool shortForm) {
// Guard against illegal branch targets, e.g. -1. Occurrences in
- // CompiledStaticCall and ad-file. Do not assert (it's a test
+ // CompiledDirectCall and ad-file. Do not assert (it's a test
// function!). Just return false in case of illegal operands.
if ((((uint64_t)target) & 0x0001L) != 0) return false;
if ((((uint64_t)pc) & 0x0001L) != 0) return false;
diff --git a/src/hotspot/cpu/s390/c1_LIRAssembler_s390.cpp b/src/hotspot/cpu/s390/c1_LIRAssembler_s390.cpp
index 542ade8ed0e..13c45bb9fe7 100644
--- a/src/hotspot/cpu/s390/c1_LIRAssembler_s390.cpp
+++ b/src/hotspot/cpu/s390/c1_LIRAssembler_s390.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2023 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -76,10 +76,7 @@ int LIR_Assembler::initial_frame_size_in_bytes() const {
// We fetch the class of the receiver and compare it with the cached class.
// If they do not match we jump to the slow case.
int LIR_Assembler::check_icache() {
- Register receiver = receiverOpr()->as_register();
- int offset = __ offset();
- __ inline_cache_check(receiver, Z_inline_cache);
- return offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::clinit_barrier(ciMethod* method) {
@@ -2480,23 +2477,30 @@ void LIR_Assembler::emit_typecheck_helper(LIR_OpTypeCheck *op, Label* success, L
assert_different_registers(obj, k_RInfo, klass_RInfo);
if (op->should_profile()) {
+ Register mdo = klass_RInfo;
+ metadata2reg(md->constant_encoding(), mdo);
NearLabel not_null;
__ compareU64_and_branch(obj, (intptr_t) 0, Assembler::bcondNotEqual, not_null);
// Object is null; update MDO and exit.
- Register mdo = klass_RInfo;
- metadata2reg(md->constant_encoding(), mdo);
Address data_addr(mdo, md->byte_offset_of_slot(data, DataLayout::header_offset()));
int header_bits = DataLayout::flag_mask_to_header_mask(BitData::null_seen_byte_constant());
__ or2mem_8(data_addr, header_bits);
__ branch_optimized(Assembler::bcondAlways, *obj_is_null);
__ bind(not_null);
+
+ NearLabel update_done;
+ Register recv = k_RInfo;
+ __ load_klass(recv, obj);
+ type_profile_helper(mdo, md, data, recv, Rtmp1, &update_done);
+ Address counter_addr(mdo, md->byte_offset_of_slot(data, CounterData::count_offset()));
+ __ add2mem_64(counter_addr, DataLayout::counter_increment, Rtmp1);
+ __ bind(update_done);
} else {
__ compareU64_and_branch(obj, (intptr_t) 0, Assembler::bcondEqual, *obj_is_null);
}
- NearLabel profile_cast_failure, profile_cast_success;
- Label *failure_target = op->should_profile() ? &profile_cast_failure : failure;
- Label *success_target = op->should_profile() ? &profile_cast_success : success;
+ Label *failure_target = failure;
+ Label *success_target = success;
// Patching may screw with our temporaries,
// so let's do it before loading the class.
@@ -2536,28 +2540,12 @@ void LIR_Assembler::emit_typecheck_helper(LIR_OpTypeCheck *op, Label* success, L
store_parameter(klass_RInfo, 0); // sub
store_parameter(k_RInfo, 1); // super
emit_call_c(a); // Sets condition code 0 for match (2 otherwise).
- CHECK_BAILOUT2(profile_cast_failure, profile_cast_success);
__ branch_optimized(Assembler::bcondNotEqual, *failure_target);
// Fall through to success case.
}
}
- if (op->should_profile()) {
- Register mdo = klass_RInfo, recv = k_RInfo;
- assert_different_registers(obj, mdo, recv);
- __ bind(profile_cast_success);
- metadata2reg(md->constant_encoding(), mdo);
- __ load_klass(recv, obj);
- type_profile_helper(mdo, md, data, recv, Rtmp1, success);
- __ branch_optimized(Assembler::bcondAlways, *success);
-
- __ bind(profile_cast_failure);
- metadata2reg(md->constant_encoding(), mdo);
- __ add2mem_64(Address(mdo, md->byte_offset_of_slot(data, CounterData::count_offset())), -(int)DataLayout::counter_increment, Rtmp1);
- __ branch_optimized(Assembler::bcondAlways, *failure);
- } else {
- __ branch_optimized(Assembler::bcondAlways, *success);
- }
+ __ branch_optimized(Assembler::bcondAlways, *success);
}
void LIR_Assembler::emit_opTypeCheck(LIR_OpTypeCheck* op) {
@@ -2587,21 +2575,29 @@ void LIR_Assembler::emit_opTypeCheck(LIR_OpTypeCheck* op) {
assert(data != nullptr, "need data for type check");
assert(data->is_ReceiverTypeData(), "need ReceiverTypeData for type check");
}
- NearLabel profile_cast_success, profile_cast_failure, done;
- Label *success_target = op->should_profile() ? &profile_cast_success : &done;
- Label *failure_target = op->should_profile() ? &profile_cast_failure : stub->entry();
+ NearLabel done;
+ Label *success_target = &done;
+ Label *failure_target = stub->entry();
if (op->should_profile()) {
+ Register mdo = klass_RInfo;
+ metadata2reg(md->constant_encoding(), mdo);
NearLabel not_null;
__ compareU64_and_branch(value, (intptr_t) 0, Assembler::bcondNotEqual, not_null);
// Object is null; update MDO and exit.
- Register mdo = klass_RInfo;
- metadata2reg(md->constant_encoding(), mdo);
Address data_addr(mdo, md->byte_offset_of_slot(data, DataLayout::header_offset()));
int header_bits = DataLayout::flag_mask_to_header_mask(BitData::null_seen_byte_constant());
__ or2mem_8(data_addr, header_bits);
__ branch_optimized(Assembler::bcondAlways, done);
__ bind(not_null);
+
+ NearLabel update_done;
+ Register recv = k_RInfo;
+ __ load_klass(recv, value);
+ type_profile_helper(mdo, md, data, recv, Rtmp1, &update_done);
+ Address counter_addr(mdo, md->byte_offset_of_slot(data, CounterData::count_offset()));
+ __ add2mem_64(counter_addr, DataLayout::counter_increment, Rtmp1);
+ __ bind(update_done);
} else {
__ compareU64_and_branch(value, (intptr_t) 0, Assembler::bcondEqual, done);
}
@@ -2619,25 +2615,9 @@ void LIR_Assembler::emit_opTypeCheck(LIR_OpTypeCheck* op) {
store_parameter(klass_RInfo, 0); // sub
store_parameter(k_RInfo, 1); // super
emit_call_c(a); // Sets condition code 0 for match (2 otherwise).
- CHECK_BAILOUT3(profile_cast_success, profile_cast_failure, done);
__ branch_optimized(Assembler::bcondNotEqual, *failure_target);
// Fall through to success case.
- if (op->should_profile()) {
- Register mdo = klass_RInfo, recv = k_RInfo;
- assert_different_registers(value, mdo, recv);
- __ bind(profile_cast_success);
- metadata2reg(md->constant_encoding(), mdo);
- __ load_klass(recv, value);
- type_profile_helper(mdo, md, data, recv, Rtmp1, &done);
- __ branch_optimized(Assembler::bcondAlways, done);
-
- __ bind(profile_cast_failure);
- metadata2reg(md->constant_encoding(), mdo);
- __ add2mem_64(Address(mdo, md->byte_offset_of_slot(data, CounterData::count_offset())), -(int)DataLayout::counter_increment, Rtmp1);
- __ branch_optimized(Assembler::bcondAlways, *stub->entry());
- }
-
__ bind(done);
} else {
if (code == lir_checkcast) {
diff --git a/src/hotspot/cpu/s390/c1_LIRAssembler_s390.hpp b/src/hotspot/cpu/s390/c1_LIRAssembler_s390.hpp
index 229216ef20d..c8815f3a729 100644
--- a/src/hotspot/cpu/s390/c1_LIRAssembler_s390.hpp
+++ b/src/hotspot/cpu/s390/c1_LIRAssembler_s390.hpp
@@ -45,7 +45,7 @@
}
enum {
- _call_stub_size = 512, // See Compile::MAX_stubs_size and CompiledStaticCall::emit_to_interp_stub.
+ _call_stub_size = 512, // See Compile::MAX_stubs_size and CompiledDirectCall::emit_to_interp_stub.
_exception_handler_size = DEBUG_ONLY(1*K) NOT_DEBUG(128),
_deopt_handler_size = DEBUG_ONLY(1*K) NOT_DEBUG(64)
};
diff --git a/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp b/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp
index 40edca6559a..5dddc7a756f 100644
--- a/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp
+++ b/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp
@@ -40,31 +40,6 @@
#include "runtime/stubRoutines.hpp"
#include "utilities/macros.hpp"
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache) {
- Label ic_miss, ic_hit;
- verify_oop(receiver, FILE_AND_LINE);
- int klass_offset = oopDesc::klass_offset_in_bytes();
-
- if (!ImplicitNullChecks || MacroAssembler::needs_explicit_null_check(klass_offset)) {
- if (VM_Version::has_CompareBranch()) {
- z_cgij(receiver, 0, Assembler::bcondEqual, ic_miss);
- } else {
- z_ltgr(receiver, receiver);
- z_bre(ic_miss);
- }
- }
-
- compare_klass_ptr(iCache, klass_offset, receiver, false);
- z_bre(ic_hit);
-
- // If icache check fails, then jump to runtime routine.
- // Note: RECEIVER must still contain the receiver!
- load_const_optimized(Z_R1_scratch, AddressLiteral(SharedRuntime::get_ic_miss_stub()));
- z_br(Z_R1_scratch);
- align(CodeEntryAlignment);
- bind(ic_hit);
-}
-
void C1_MacroAssembler::explicit_null_check(Register base) {
ShouldNotCallThis(); // unused
}
diff --git a/src/hotspot/cpu/s390/c1_Runtime1_s390.cpp b/src/hotspot/cpu/s390/c1_Runtime1_s390.cpp
index 257148827be..decb3a1cafc 100644
--- a/src/hotspot/cpu/s390/c1_Runtime1_s390.cpp
+++ b/src/hotspot/cpu/s390/c1_Runtime1_s390.cpp
@@ -35,7 +35,6 @@
#include "interpreter/interpreter.hpp"
#include "memory/universe.hpp"
#include "nativeInst_s390.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_s390.hpp"
diff --git a/src/hotspot/cpu/s390/compiledIC_s390.cpp b/src/hotspot/cpu/s390/compiledIC_s390.cpp
index 7ea90c1de7c..3adcfbc85f1 100644
--- a/src/hotspot/cpu/s390/compiledIC_s390.cpp
+++ b/src/hotspot/cpu/s390/compiledIC_s390.cpp
@@ -26,7 +26,6 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "memory/resourceArea.hpp"
#include "runtime/mutexLocker.hpp"
@@ -40,7 +39,7 @@
#undef __
#define __ _masm.
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/* = nullptr*/) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/* = nullptr*/) {
#ifdef COMPILER2
// Stub is fixed up when the corresponding call is converted from calling
// compiled code to calling interpreted code.
@@ -54,7 +53,7 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/*
// That's why we must use the macroassembler to generate a stub.
MacroAssembler _masm(&cbuf);
- address stub = __ start_a_stub(CompiledStaticCall::to_interp_stub_size());
+ address stub = __ start_a_stub(CompiledDirectCall::to_interp_stub_size());
if (stub == nullptr) {
return nullptr; // CodeBuffer::expand failed.
}
@@ -81,27 +80,20 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark/*
#undef __
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return 2 * MacroAssembler::load_const_from_toc_size() +
2; // branch
}
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 5; // 4 in emit_java_to_interp + 1 in Java_Static_Call
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder = nativeMovConstReg_at(stub + NativeCall::get_IC_pos_in_java_to_interp_stub());
NativeJump* jump = nativeJump_at(method_holder->next_instruction_address());
@@ -115,7 +107,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
// Reset stub.
address stub = static_stub->addr();
assert(stub != nullptr, "stub not found");
@@ -131,7 +123,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/s390/icBuffer_s390.cpp b/src/hotspot/cpu/s390/icBuffer_s390.cpp
deleted file mode 100644
index 0dc936d6fad..00000000000
--- a/src/hotspot/cpu/s390/icBuffer_s390.cpp
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016 SAP SE. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/macroAssembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_s390.hpp"
-#include "oops/oop.inline.hpp"
-
-#define __ masm.
-
-int InlineCacheBuffer::ic_stub_code_size() {
- return MacroAssembler::load_const_size() + Assembler::z_brul_size();
-}
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_oop, address entry_point) {
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler masm(&code);
- // Note: even though the code contains an embedded oop, we do not need reloc info
- // because
- // (1) the oop is old (i.e., doesn't matter for scavenges)
- // (2) these ICStubs are removed *before* a GC happens, so the roots disappear.
-
- // Load the oop,
- __ load_const(Z_method, (address) cached_oop); // inline cache reg = Z_method
- // and do a tail-call (pc-relative).
- __ z_brul((address) entry_point);
- __ flush();
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // Creation also verifies the object.
- return MacroAssembler::get_target_addr_pcrel(move->next_instruction_address());
-}
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // Creation also verifies the object.
- return (void*)move->data();
-}
diff --git a/src/hotspot/cpu/s390/interp_masm_s390.cpp b/src/hotspot/cpu/s390/interp_masm_s390.cpp
index ea691342b23..9ee38c619f0 100644
--- a/src/hotspot/cpu/s390/interp_masm_s390.cpp
+++ b/src/hotspot/cpu/s390/interp_masm_s390.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2023 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -447,9 +447,6 @@ void InterpreterMacroAssembler::gen_subtype_check(Register Rsub_klass,
// Do the check.
check_klass_subtype(Rsub_klass, Rsuper_klass, Rtmp1, Rtmp2, ok_is_subtype);
-
- // Profile the failure of the check.
- profile_typecheck_failed(Rtmp1, Rtmp2);
}
// Pop topmost element from stack. It just disappears.
@@ -1425,7 +1422,7 @@ void InterpreterMacroAssembler::profile_virtual_call(Register receiver,
}
// Record the receiver type.
- record_klass_in_profile(receiver, mdp, reg2, true);
+ record_klass_in_profile(receiver, mdp, reg2);
bind(skip_receiver_profile);
// The method data pointer needs to be updated to reflect the new target.
@@ -1448,11 +1445,9 @@ void InterpreterMacroAssembler::profile_virtual_call(Register receiver,
void InterpreterMacroAssembler::record_klass_in_profile_helper(
Register receiver, Register mdp,
Register reg2, int start_row,
- Label& done, bool is_virtual_call) {
+ Label& done) {
if (TypeProfileWidth == 0) {
- if (is_virtual_call) {
- increment_mdp_data_at(mdp, in_bytes(CounterData::count_offset()));
- }
+ increment_mdp_data_at(mdp, in_bytes(CounterData::count_offset()));
return;
}
@@ -1487,23 +1482,19 @@ void InterpreterMacroAssembler::record_klass_in_profile_helper(
z_ltgr(reg2, reg2);
if (start_row == last_row) {
// The only thing left to do is handle the null case.
- if (is_virtual_call) {
- z_brz(found_null);
- // Receiver did not match any saved receiver and there is no empty row for it.
- // Increment total counter to indicate polymorphic case.
- increment_mdp_data_at(mdp, in_bytes(CounterData::count_offset()));
- z_bru(done);
- bind(found_null);
- } else {
- z_brnz(done);
- }
+ z_brz(found_null);
+ // Receiver did not match any saved receiver and there is no empty row for it.
+ // Increment total counter to indicate polymorphic case.
+ increment_mdp_data_at(mdp, in_bytes(CounterData::count_offset()));
+ z_bru(done);
+ bind(found_null);
break;
}
// Since null is rare, make it be the branch-taken case.
z_brz(found_null);
// Put all the "Case 3" tests here.
- record_klass_in_profile_helper(receiver, mdp, reg2, start_row + 1, done, is_virtual_call);
+ record_klass_in_profile_helper(receiver, mdp, reg2, start_row + 1, done);
// Found a null. Keep searching for a matching receiver,
// but remember that this is an empty (unused) slot.
@@ -1550,12 +1541,11 @@ void InterpreterMacroAssembler::record_klass_in_profile_helper(
// done:
void InterpreterMacroAssembler::record_klass_in_profile(Register receiver,
- Register mdp, Register reg2,
- bool is_virtual_call) {
+ Register mdp, Register reg2) {
assert(ProfileInterpreter, "must be profiling");
Label done;
- record_klass_in_profile_helper(receiver, mdp, reg2, 0, done, is_virtual_call);
+ record_klass_in_profile_helper(receiver, mdp, reg2, 0, done);
bind (done);
}
@@ -1615,24 +1605,6 @@ void InterpreterMacroAssembler::profile_null_seen(Register mdp) {
}
}
-void InterpreterMacroAssembler::profile_typecheck_failed(Register mdp, Register tmp) {
- if (ProfileInterpreter && TypeProfileCasts) {
- Label profile_continue;
-
- // If no method data exists, go to profile_continue.
- test_method_data_pointer(mdp, profile_continue);
-
- int count_offset = in_bytes(CounterData::count_offset());
- // Back up the address, since we have already bumped the mdp.
- count_offset -= in_bytes(VirtualCallData::virtual_call_data_size());
-
- // *Decrement* the counter. We expect to see zero or small negatives.
- increment_mdp_data_at(mdp, count_offset, tmp, true);
-
- bind (profile_continue);
- }
-}
-
void InterpreterMacroAssembler::profile_typecheck(Register mdp, Register klass, Register reg2) {
if (ProfileInterpreter) {
Label profile_continue;
@@ -1646,7 +1618,7 @@ void InterpreterMacroAssembler::profile_typecheck(Register mdp, Register klass,
mdp_delta = in_bytes(VirtualCallData::virtual_call_data_size());
// Record the object type.
- record_klass_in_profile(klass, mdp, reg2, false);
+ record_klass_in_profile(klass, mdp, reg2);
}
update_mdp_by_constant(mdp, mdp_delta);
diff --git a/src/hotspot/cpu/s390/interp_masm_s390.hpp b/src/hotspot/cpu/s390/interp_masm_s390.hpp
index 56abe295876..f94473b1700 100644
--- a/src/hotspot/cpu/s390/interp_masm_s390.hpp
+++ b/src/hotspot/cpu/s390/interp_masm_s390.hpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2023 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -281,10 +281,10 @@ class InterpreterMacroAssembler: public MacroAssembler {
Label& not_equal_continue);
void record_klass_in_profile(Register receiver, Register mdp,
- Register reg2, bool is_virtual_call);
+ Register reg2);
void record_klass_in_profile_helper(Register receiver, Register mdp,
Register reg2, int start_row,
- Label& done, bool is_virtual_call);
+ Label& done);
void update_mdp_by_offset(Register mdp_in, int offset_of_offset);
void update_mdp_by_offset(Register mdp_in, Register dataidx, int offset_of_disp);
@@ -301,7 +301,6 @@ class InterpreterMacroAssembler: public MacroAssembler {
void profile_ret(Register return_bci, Register mdp);
void profile_null_seen(Register mdp);
void profile_typecheck(Register mdp, Register klass, Register scratch);
- void profile_typecheck_failed(Register mdp, Register tmp);
void profile_switch_default(Register mdp);
void profile_switch_case(Register index_in_scratch, Register mdp,
Register scratch1, Register scratch2);
diff --git a/src/hotspot/cpu/s390/macroAssembler_s390.cpp b/src/hotspot/cpu/s390/macroAssembler_s390.cpp
index d95a0b3a3c5..0226d494c89 100644
--- a/src/hotspot/cpu/s390/macroAssembler_s390.cpp
+++ b/src/hotspot/cpu/s390/macroAssembler_s390.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2016, 2023 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -26,6 +26,7 @@
#include "precompiled.hpp"
#include "asm/codeBuffer.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/disassembler.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
@@ -1097,7 +1098,13 @@ void MacroAssembler::clear_mem(const Address& addr, unsigned int size) {
}
void MacroAssembler::align(int modulus) {
- while (offset() % modulus != 0) z_nop();
+ align(modulus, offset());
+}
+
+void MacroAssembler::align(int modulus, int target) {
+ assert(((modulus % 2 == 0) && (target % 2 == 0)), "needs to be even");
+ int delta = target - offset();
+ while ((offset() + delta) % modulus != 0) z_nop();
}
// Special version for non-relocateable code if required alignment
@@ -2150,6 +2157,45 @@ void MacroAssembler::call_VM_leaf_base(address entry_point) {
call_VM_leaf_base(entry_point, allow_relocation);
}
+int MacroAssembler::ic_check_size() {
+ return 30 + (ImplicitNullChecks ? 0 : 6);
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ Register R2_receiver = Z_ARG1;
+ Register R0_scratch = Z_R0_scratch;
+ Register R1_scratch = Z_R1_scratch;
+ Register R9_data = Z_inline_cache;
+ Label success, failure;
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, offset() + ic_check_size());
+
+ int uep_offset = offset();
+ if (!ImplicitNullChecks) {
+ z_cgij(R2_receiver, 0, Assembler::bcondEqual, failure);
+ }
+
+ if (UseCompressedClassPointers) {
+ z_llgf(R1_scratch, Address(R2_receiver, oopDesc::klass_offset_in_bytes()));
+ } else {
+ z_lg(R1_scratch, Address(R2_receiver, oopDesc::klass_offset_in_bytes()));
+ }
+ z_cg(R1_scratch, Address(R9_data, in_bytes(CompiledICData::speculated_klass_offset())));
+ z_bre(success);
+
+ bind(failure);
+ load_const(R1_scratch, AddressLiteral(SharedRuntime::get_ic_miss_stub()));
+ z_br(R1_scratch);
+ bind(success);
+
+ assert((offset() % end_alignment) == 0, "Misaligned verified entry point, offset() = %d, end_alignment = %d", offset(), end_alignment);
+ return uep_offset;
+}
+
void MacroAssembler::call_VM_base(Register oop_result,
Register last_java_sp,
address entry_point,
@@ -3744,7 +3790,7 @@ void MacroAssembler::compare_klass_ptr(Register Rop1, int64_t disp, Register Rba
Register current = Rop1;
Label done;
- if (maybenull) { // null ptr must be preserved!
+ if (maybenull) { // null pointer must be preserved!
z_ltgr(Z_R0, current);
z_bre(done);
current = Z_R0;
@@ -3883,7 +3929,7 @@ void MacroAssembler::compare_heap_oop(Register Rop1, Address mem, bool maybenull
Label done;
int pow2_offset = get_oop_base_complement(Z_R1, ((uint64_t)(intptr_t)base));
- if (maybenull) { // null ptr must be preserved!
+ if (maybenull) { // null pointer must be preserved!
z_ltgr(Z_R0, Rop1);
z_bre(done);
}
@@ -4123,7 +4169,7 @@ void MacroAssembler::oop_decoder(Register Rdst, Register Rsrc, bool maybenull, R
Label done;
// Rsrc contains a narrow oop. Thus we are sure the leftmost bits will never be set.
- if (maybenull) { // null ptr must be preserved!
+ if (maybenull) { // null pointer must be preserved!
z_slag(Rdst, Rsrc, oop_shift); // Arithmetic shift sets the condition code.
z_bre(done);
} else {
@@ -4185,7 +4231,7 @@ void MacroAssembler::oop_decoder(Register Rdst, Register Rsrc, bool maybenull, R
// Scale oop and check for null.
// Rsrc contains a narrow oop. Thus we are sure the leftmost bits will never be set.
- if (maybenull) { // null ptr must be preserved!
+ if (maybenull) { // null pointer must be preserved!
z_slag(Rdst_tmp, Rsrc, oop_shift); // Arithmetic shift sets the condition code.
z_bre(done);
} else {
diff --git a/src/hotspot/cpu/s390/macroAssembler_s390.hpp b/src/hotspot/cpu/s390/macroAssembler_s390.hpp
index bf14b42e2d1..924583abdf5 100644
--- a/src/hotspot/cpu/s390/macroAssembler_s390.hpp
+++ b/src/hotspot/cpu/s390/macroAssembler_s390.hpp
@@ -257,6 +257,7 @@ class MacroAssembler: public Assembler {
// nop padding
void align(int modulus);
+ void align(int modulus, int target);
void align_address(int modulus);
//
@@ -566,6 +567,9 @@ class MacroAssembler: public Assembler {
// Get the pc where the last call will return to. Returns _last_calls_return_pc.
inline address last_calls_return_pc();
+ static int ic_check_size();
+ int ic_check(int end_alignment);
+
private:
static bool is_call_far_patchable_variant0_at(address instruction_addr); // Dynamic TOC: load target addr from CP and call.
static bool is_call_far_patchable_variant2_at(address instruction_addr); // PC-relative call, prefixed with NOPs.
diff --git a/src/hotspot/cpu/s390/s390.ad b/src/hotspot/cpu/s390/s390.ad
index 9f4e182a9e4..5db2db9d32c 100644
--- a/src/hotspot/cpu/s390/s390.ad
+++ b/src/hotspot/cpu/s390/s390.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
// Copyright (c) 2017, 2022 SAP SE. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
@@ -740,7 +740,7 @@ int emit_call_reloc(C2_MacroAssembler &_masm, intptr_t entry_point, relocInfo::r
if (rtype == relocInfo::runtime_call_w_cp_type) {
assert((__ offset() & 2) == 0, "misaligned emit_call_reloc");
address call_addr = __ call_c_opt((address)entry_point);
- if (call_addr == NULL) {
+ if (call_addr == nullptr) {
Compile::current()->env()->record_out_of_memory_failure();
return -1;
}
@@ -835,7 +835,7 @@ void MachPrologNode::format(PhaseRegAlloc *ra_, outputStream *st) const {
st->print_cr("push_frame %d", (int)-framesize);
st->print("\t");
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
st->print("nmethod entry barrier\n\t");
}
}
@@ -890,7 +890,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
constant_table.set_table_base_offset(constant_table.calculate_table_base_offset());
}
- if (C->stub_function() == NULL) {
+ if (C->stub_function() == nullptr) {
BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler();
bs->nmethod_entry_barrier(&_masm);
}
@@ -1056,7 +1056,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
bool src12 = Immediate::is_uimm12(src_offset);
bool dst12 = Immediate::is_uimm12(dst_offset);
- const char *mnemo = NULL;
+ const char *mnemo = nullptr;
unsigned long opc = 0;
// Memory->Memory Spill. Use Z_R0 to hold the value.
@@ -1199,7 +1199,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer *cbuf, PhaseRegAlloc *ra_, boo
#if !defined(PRODUCT)
void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream *os) const {
if (ra_ && ra_->node_regs_max_index() > 0) {
- implementation(NULL, ra_, false, os);
+ implementation(nullptr, ra_, false, os);
} else {
if (req() == 2 && in(1)) {
os->print("N%d = N%d\n", _idx, in(1)->_idx);
@@ -1217,11 +1217,11 @@ void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream *os) const {
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation(&cbuf, ra_, false, NULL);
+ implementation(&cbuf, ra_, false, nullptr);
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
- return implementation(NULL, ra_, true, NULL);
+ return implementation(nullptr, ra_, true, nullptr);
}
//=============================================================================
@@ -1341,51 +1341,9 @@ void MachUEPNode::format(PhaseRegAlloc *ra_, outputStream *os) const {
#endif
void MachUEPNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
+ // This is Unverified Entry Point
C2_MacroAssembler _masm(&cbuf);
- const int ic_miss_offset = 2;
-
- // Inline_cache contains a klass.
- Register ic_klass = as_Register(Matcher::inline_cache_reg_encode());
- // ARG1 is the receiver oop.
- Register R2_receiver = Z_ARG1;
- int klass_offset = oopDesc::klass_offset_in_bytes();
- AddressLiteral icmiss(SharedRuntime::get_ic_miss_stub());
- Register R1_ic_miss_stub_addr = Z_R1_scratch;
-
- // Null check of receiver.
- // This is the null check of the receiver that actually should be
- // done in the caller. It's here because in case of implicit null
- // checks we get it for free.
- assert(!MacroAssembler::needs_explicit_null_check(oopDesc::klass_offset_in_bytes()),
- "second word in oop should not require explicit null check.");
- if (!ImplicitNullChecks) {
- Label valid;
- if (VM_Version::has_CompareBranch()) {
- __ z_cgij(R2_receiver, 0, Assembler::bcondNotEqual, valid);
- } else {
- __ z_ltgr(R2_receiver, R2_receiver);
- __ z_bre(valid);
- }
- // The ic_miss_stub will handle the null pointer exception.
- __ load_const_optimized(R1_ic_miss_stub_addr, icmiss);
- __ z_br(R1_ic_miss_stub_addr);
- __ bind(valid);
- }
-
- // Check whether this method is the proper implementation for the class of
- // the receiver (ic miss check).
- {
- Label valid;
- // Compare cached class against klass from receiver.
- // This also does an implicit null check!
- __ compare_klass_ptr(ic_klass, klass_offset, R2_receiver, false);
- __ z_bre(valid);
- // The inline cache points to the wrong method. Call the
- // ic_miss_stub to find the proper method.
- __ load_const_optimized(R1_ic_miss_stub_addr, icmiss);
- __ z_br(R1_ic_miss_stub_addr);
- __ bind(valid);
- }
+ __ ic_check(CodeEntryAlignment);
}
uint MachUEPNode::size(PhaseRegAlloc *ra_) const {
@@ -1446,7 +1404,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer &cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) {
+ if (base == nullptr) {
return 0; // CodeBuffer::expand failed
}
@@ -1467,7 +1425,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf) {
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) {
+ if (base == nullptr) {
return 0; // CodeBuffer::expand failed
}
@@ -1513,7 +1471,7 @@ bool Matcher::match_rule_supported(int opcode) {
return true; // Per default match rules are supported.
}
-bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
return match_rule_supported_vector(opcode, vlen, bt);
}
@@ -1533,11 +1491,11 @@ bool Matcher::vector_needs_partial_operations(Node* node, const TypeVect* vt) {
}
const RegMask* Matcher::predicate_reg_mask(void) {
- return NULL;
+ return nullptr;
}
const TypeVectMask* Matcher::predicate_reg_type(const Type* elemTy, int length) {
- return NULL;
+ return nullptr;
}
// Vector calling convention not yet implemented.
@@ -1574,7 +1532,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return max_vector_size(bt); // Same as max.
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
return Matcher::max_vector_size(bt);
}
@@ -1602,7 +1560,7 @@ bool Matcher::is_short_branch_offset(int rule, int br_size, int offset) {
MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* original_opnd, uint ideal_reg, bool is_temp) {
ShouldNotReachHere(); // generic vector operands not supported
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -1948,12 +1906,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
__ z_brul(l);
%}
@@ -1961,12 +1919,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
__ z_bru(l);
%}
@@ -1974,12 +1932,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
__ z_brcl((Assembler::branch_condition)$cmp$$cmpcode, l);
%}
@@ -1987,12 +1945,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
__ z_brc((Assembler::branch_condition)$cmp$$cmpcode, l);
%}
@@ -2000,12 +1958,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
Assembler::branch_condition cc = (Assembler::branch_condition)$cmp$$cmpcode;
unsigned long instr = $primary;
if (instr == CRJ_ZOPC) {
@@ -2024,12 +1982,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
unsigned long instr = $primary;
if (instr == CR_ZOPC) {
@@ -2050,12 +2008,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
Assembler::branch_condition cc = (Assembler::branch_condition)$cmp$$cmpcode;
unsigned long instr = $primary;
@@ -2075,12 +2033,12 @@ encode %{
C2_MacroAssembler _masm(&cbuf);
Label* p = $lbl$$label;
- // 'p' is `NULL' when this encoding class is used only to
+ // 'p' is `nullptr' when this encoding class is used only to
// determine the size of the encoded instruction.
// Use a bound dummy label in that case.
Label d;
__ bind(d);
- Label& l = (NULL == p) ? d : *(p);
+ Label& l = (nullptr == p) ? d : *(p);
unsigned long instr = $primary;
if (instr == CHI_ZOPC) {
@@ -2112,7 +2070,7 @@ encode %{
assert((__ offset() & 2) == 0, "misaligned z_enc_java_to_runtime_call");
address call_addr = __ call_c_opt((address)$meth$$method);
- if (call_addr == NULL) {
+ if (call_addr == nullptr) {
Compile::current()->env()->record_out_of_memory_failure();
return;
}
@@ -2143,11 +2101,11 @@ encode %{
static_call_Relocation::spec(method_index));
}
}
- assert(__ inst_mark() != NULL, "emit_call_reloc must set_inst_mark()");
+ assert(__ inst_mark() != nullptr, "emit_call_reloc must set_inst_mark()");
if (_method) { // Emit stub for static call.
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2162,7 +2120,7 @@ encode %{
int vtable_index = this->_vtable_index;
if (vtable_index == -4) {
Register ic_reg = reg_to_register_object(Matcher::inline_cache_reg_encode());
- address virtual_call_oop_addr = NULL;
+ address virtual_call_oop_addr = nullptr;
AddressLiteral empty_ic((address) Universe::non_oop_word());
virtual_call_oop_addr = __ pc();
@@ -2929,7 +2887,7 @@ operand immP8() %{
// POINTER specific values
//-----------------------------------
-// Pointer Immediate: NULL
+// Pointer Immediate: nullptr
operand immP0() %{
predicate(n->get_ptr() == 0);
match(ConP);
@@ -2966,7 +2924,7 @@ operand immN8() %{
interface(CONST_INTER);
%}
-// Narrow NULL Pointer Immediate
+// Narrow Null Pointer Immediate
operand immN0() %{
predicate(n->get_narrowcon() == 0);
match(ConN);
@@ -3383,7 +3341,7 @@ operand inline_cache_regP(iRegP reg) %{
// Operands to remove register moves in unscaled mode.
// Match read/write registers with an EncodeP node if neither shift nor add are required.
operand iRegP2N(iRegP reg) %{
- predicate(CompressedOops::shift() == 0 && _leaf->as_EncodeP()->in(0) == NULL);
+ predicate(CompressedOops::shift() == 0 && _leaf->as_EncodeP()->in(0) == nullptr);
constraint(ALLOC_IN_RC(z_memory_ptr_reg));
match(EncodeP reg);
format %{ "$reg" %}
@@ -3391,8 +3349,8 @@ operand iRegP2N(iRegP reg) %{
%}
operand iRegN2P(iRegN reg) %{
- predicate(CompressedOops::base() == NULL && CompressedOops::shift() == 0 &&
- _leaf->as_DecodeN()->in(0) == NULL);
+ predicate(CompressedOops::base() == nullptr && CompressedOops::shift() == 0 &&
+ _leaf->as_DecodeN()->in(0) == nullptr);
constraint(ALLOC_IN_RC(z_memory_ptr_reg));
match(DecodeN reg);
format %{ "$reg" %}
@@ -4236,7 +4194,7 @@ instruct loadConL_pcrelTOC(iRegL dst, immL src) %{
format %{ "LGRL $dst,[pcrelTOC]\t # load long $src from table" %}
ins_encode %{
address long_address = __ long_constant($src$$constant);
- if (long_address == NULL) {
+ if (long_address == nullptr) {
Compile::current()->env()->record_out_of_memory_failure();
return;
}
@@ -4292,14 +4250,14 @@ instruct loadConP_pcrelTOC(iRegP dst, immP src) %{
} else if (constant_reloc == relocInfo::metadata_type) {
AddressLiteral a = __ constant_metadata_address((Metadata *)$src$$constant);
address const_toc_addr = __ address_constant((address)a.value(), RelocationHolder::none);
- if (const_toc_addr == NULL) {
+ if (const_toc_addr == nullptr) {
Compile::current()->env()->record_out_of_memory_failure();
return;
}
__ load_long_pcrelative($dst$$Register, const_toc_addr);
} else { // Non-oop pointers, e.g. card mark base, heap top.
address long_address = __ long_constant((jlong)$src$$constant);
- if (long_address == NULL) {
+ if (long_address == nullptr) {
Compile::current()->env()->record_out_of_memory_failure();
return;
}
@@ -4314,7 +4272,7 @@ instruct loadConP0(iRegP dst, immP0 src, flagsReg cr) %{
match(Set dst src);
effect(KILL cr);
size(4);
- format %{ "XGR $dst,$dst\t # NULL ptr" %}
+ format %{ "XGR $dst,$dst\t # null pointer" %}
opcode(XGR_ZOPC);
ins_encode(z_rreform(dst, dst));
ins_pipe(pipe_class_dummy);
@@ -4660,7 +4618,7 @@ instruct loadConNKlass(iRegN dst, immNKlass src) %{
instruct decodeLoadN(iRegP dst, memory mem) %{
match(Set dst (DecodeN (LoadN mem)));
- predicate(false && (CompressedOops::base()==NULL)&&(CompressedOops::shift()==0));
+ predicate(false && (CompressedOops::base()==nullptr)&&(CompressedOops::shift()==0));
ins_cost(MEMORY_REF_COST);
size(Z_DISP3_SIZE);
format %{ "DecodeLoadN $dst,$mem\t # (cOop Load+Decode)" %}
@@ -4671,7 +4629,7 @@ instruct decodeLoadN(iRegP dst, memory mem) %{
instruct decodeLoadNKlass(iRegP dst, memory mem) %{
match(Set dst (DecodeNKlass (LoadNKlass mem)));
- predicate(false && (CompressedKlassPointers::base()==NULL)&&(CompressedKlassPointers::shift()==0));
+ predicate(false && (CompressedKlassPointers::base()==nullptr)&&(CompressedKlassPointers::shift()==0));
ins_cost(MEMORY_REF_COST);
size(Z_DISP3_SIZE);
format %{ "DecodeLoadNKlass $dst,$mem\t # (load/decode NKlass)" %}
@@ -4699,7 +4657,7 @@ instruct decodeLoadConNKlass(iRegP dst, immNKlass src) %{
instruct decodeN(iRegP dst, iRegN src, flagsReg cr) %{
match(Set dst (DecodeN src));
effect(KILL cr);
- predicate(CompressedOops::base() == NULL || !ExpandLoadingBaseDecode);
+ predicate(CompressedOops::base() == nullptr || !ExpandLoadingBaseDecode);
ins_cost(MEMORY_REF_COST+3 * DEFAULT_COST + BRANCH_COST);
// TODO: s390 port size(VARIABLE_SIZE);
format %{ "decodeN $dst,$src\t # (decode cOop)" %}
@@ -4723,7 +4681,7 @@ instruct decodeN_NN(iRegP dst, iRegN src, flagsReg cr) %{
effect(KILL cr);
predicate((n->bottom_type()->make_ptr()->ptr() == TypePtr::NotNull ||
n->bottom_type()->is_oopptr()->ptr() == TypePtr::Constant) &&
- (CompressedOops::base()== NULL || !ExpandLoadingBaseDecode_NN));
+ (CompressedOops::base()== nullptr || !ExpandLoadingBaseDecode_NN));
ins_cost(MEMORY_REF_COST+2 * DEFAULT_COST);
// TODO: s390 port size(VARIABLE_SIZE);
format %{ "decodeN $dst,$src\t # (decode cOop NN)" %}
@@ -4749,7 +4707,7 @@ instruct decodeN_NN(iRegP dst, iRegN src, flagsReg cr) %{
effect(KILL cr);
predicate(false);
// TODO: s390 port size(VARIABLE_SIZE);
- format %{ "decodeN $dst = ($src == 0) ? NULL : ($src << 3) + $base + pow2_offset\t # (decode cOop)" %}
+ format %{ "decodeN $dst = ($src == 0) ? nullptr : ($src << 3) + $base + pow2_offset\t # (decode cOop)" %}
ins_encode %{
__ oop_decoder($dst$$Register, $src$$Register, true, $base$$Register,
(jlong)MacroAssembler::get_oop_base_pow2_offset((uint64_t)(intptr_t)CompressedOops::base()));
@@ -4774,7 +4732,7 @@ instruct decodeN_NN(iRegP dst, iRegN src, flagsReg cr) %{
// Decoder for heapbased mode peeling off loading the base.
instruct decodeN_Ex(iRegP dst, iRegN src, flagsReg cr) %{
match(Set dst (DecodeN src));
- predicate(CompressedOops::base() != NULL && ExpandLoadingBaseDecode);
+ predicate(CompressedOops::base() != nullptr && ExpandLoadingBaseDecode);
ins_cost(MEMORY_REF_COST+3 * DEFAULT_COST + BRANCH_COST);
// TODO: s390 port size(VARIABLE_SIZE);
expand %{
@@ -4790,7 +4748,7 @@ instruct decodeN_NN_Ex(iRegP dst, iRegN src, flagsReg cr) %{
match(Set dst (DecodeN src));
predicate((n->bottom_type()->make_ptr()->ptr() == TypePtr::NotNull ||
n->bottom_type()->is_oopptr()->ptr() == TypePtr::Constant) &&
- CompressedOops::base() != NULL && ExpandLoadingBaseDecode_NN);
+ CompressedOops::base() != nullptr && ExpandLoadingBaseDecode_NN);
ins_cost(MEMORY_REF_COST+2 * DEFAULT_COST);
// TODO: s390 port size(VARIABLE_SIZE);
expand %{
@@ -6069,7 +6027,7 @@ instruct addP_reg_reg_imm12(iRegP dst, memoryRegP src1, iRegL src2, uimmL12 con)
instruct addP_regN_reg_imm12(iRegP dst, iRegP_N2P src1, iRegL src2, uimmL12 con) %{
match(Set dst (AddP (AddP src1 src2) con));
- predicate( PreferLAoverADD && CompressedOops::base() == NULL && CompressedOops::shift() == 0);
+ predicate( PreferLAoverADD && CompressedOops::base() == nullptr && CompressedOops::shift() == 0);
ins_cost(DEFAULT_COST_LOW);
size(4);
format %{ "LA $dst,$con($src1,$src2)\t # ptr d12(x,b)" %}
@@ -6091,7 +6049,7 @@ instruct addP_reg_reg_imm20(iRegP dst, memoryRegP src1, iRegL src2, immL20 con)
instruct addP_regN_reg_imm20(iRegP dst, iRegP_N2P src1, iRegL src2, immL20 con) %{
match(Set dst (AddP (AddP src1 src2) con));
- predicate( PreferLAoverADD && CompressedOops::base() == NULL && CompressedOops::shift() == 0);
+ predicate( PreferLAoverADD && CompressedOops::base() == nullptr && CompressedOops::shift() == 0);
ins_cost(DEFAULT_COST);
// TODO: s390 port size(FIXED_SIZE);
format %{ "LAY $dst,$con($src1,$src2)\t # ptr d20(x,b)" %}
@@ -8427,7 +8385,7 @@ instruct compP_reg_imm0(flagsReg cr, iRegP_N2P op1, immP0 op2) %{
// Don't use LTGFR which performs sign extend.
instruct compP_decode_reg_imm0(flagsReg cr, iRegN op1, immP0 op2) %{
match(Set cr (CmpP (DecodeN op1) op2));
- predicate(CompressedOops::base() == NULL && CompressedOops::shift() == 0);
+ predicate(CompressedOops::base() == nullptr && CompressedOops::shift() == 0);
ins_cost(DEFAULT_COST_LOW);
size(2);
format %{ "LTR $op1, $op1\t # ptr" %}
@@ -9843,24 +9801,10 @@ instruct string_equalsL(iRegP str1, iRegP str2, iRegI cnt, iRegI result, roddReg
ins_pipe(pipe_class_dummy);
%}
-instruct string_equalsU(iRegP str1, iRegP str2, iRegI cnt, iRegI result, roddRegL oddReg, revenRegL evenReg, flagsReg cr) %{
- match(Set result (StrEquals (Binary str1 str2) cnt));
- effect(TEMP oddReg, TEMP evenReg, KILL cr); // R0, R1 are killed, too.
- predicate(((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU || ((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::none);
- ins_cost(300);
- format %{ "String Equals char[] $str1,$str2,$cnt -> $result" %}
- ins_encode %{
- __ array_equals(false, $str1$$Register, $str2$$Register,
- $cnt$$Register, $oddReg$$Register, $evenReg$$Register,
- $result$$Register, false /* byte */);
- %}
- ins_pipe(pipe_class_dummy);
-%}
-
instruct string_equals_imm(iRegP str1, iRegP str2, uimmI8 cnt, iRegI result, flagsReg cr) %{
match(Set result (StrEquals (Binary str1 str2) cnt));
effect(KILL cr); // R0 is killed, too.
- predicate(((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::LL || ((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::UU);
+ predicate(((StrEqualsNode*)n)->encoding() == StrIntrinsicNode::LL);
ins_cost(100);
format %{ "String Equals byte[] $str1,$str2,$cnt -> $result" %}
ins_encode %{
diff --git a/src/hotspot/cpu/s390/sharedRuntime_s390.cpp b/src/hotspot/cpu/s390/sharedRuntime_s390.cpp
index ed1795cfa33..11e1e617d8e 100644
--- a/src/hotspot/cpu/s390/sharedRuntime_s390.cpp
+++ b/src/hotspot/cpu/s390/sharedRuntime_s390.cpp
@@ -26,8 +26,8 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
#include "gc/shared/gcLocker.hpp"
@@ -35,7 +35,6 @@
#include "interpreter/interp_masm.hpp"
#include "memory/resourceArea.hpp"
#include "nativeInst_s390.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "prims/methodHandles.hpp"
#include "registerSaver_s390.hpp"
@@ -1500,17 +1499,15 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
unsigned int wrapper_FrameDone;
unsigned int wrapper_CRegsSet;
Label handle_pending_exception;
- Label ic_miss;
//---------------------------------------------------------------------
// Unverified entry point (UEP)
//---------------------------------------------------------------------
- wrapper_UEPStart = __ offset();
// check ic: object class <-> cached class
- if (!method_is_static) __ nmethod_UEP(ic_miss);
- // Fill with nops (alignment of verified entry point).
- __ align(CodeEntryAlignment);
+ if (!method_is_static) {
+ wrapper_UEPStart = __ ic_check(CodeEntryAlignment /* end_alignment */);
+ }
//---------------------------------------------------------------------
// Verified entry point (VEP)
@@ -2026,13 +2023,7 @@ nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm,
__ restore_return_pc();
__ z_br(Z_R1_scratch);
- //---------------------------------------------------------------------
- // Handler for a cache miss (out-of-line)
- //---------------------------------------------------------------------
- __ call_ic_miss_handler(ic_miss, 0x77, 0, Z_R1_scratch);
__ flush();
-
-
//////////////////////////////////////////////////////////////////////
// end of code generation
//////////////////////////////////////////////////////////////////////
@@ -2318,9 +2309,6 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
Label skip_fixup;
{
Label ic_miss;
- const int klass_offset = oopDesc::klass_offset_in_bytes();
- const int holder_klass_offset = in_bytes(CompiledICHolder::holder_klass_offset());
- const int holder_metadata_offset = in_bytes(CompiledICHolder::holder_metadata_offset());
// Out-of-line call to ic_miss handler.
__ call_ic_miss_handler(ic_miss, 0x11, 0, Z_R1_scratch);
@@ -2329,27 +2317,11 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
__ align(CodeEntryAlignment);
c2i_unverified_entry = __ pc();
- // Check the pointers.
- if (!ImplicitNullChecks || MacroAssembler::needs_explicit_null_check(klass_offset)) {
- __ z_ltgr(Z_ARG1, Z_ARG1);
- __ z_bre(ic_miss);
- }
- __ verify_oop(Z_ARG1, FILE_AND_LINE);
-
- // Check ic: object class <-> cached class
- // Compress cached class for comparison. That's more efficient.
- if (UseCompressedClassPointers) {
- __ z_lg(Z_R11, holder_klass_offset, Z_method); // Z_R11 is overwritten a few instructions down anyway.
- __ compare_klass_ptr(Z_R11, klass_offset, Z_ARG1, false); // Cached class can't be zero.
- } else {
- __ z_clc(klass_offset, sizeof(void *)-1, Z_ARG1, holder_klass_offset, Z_method);
- }
- __ z_brne(ic_miss); // Cache miss: call runtime to handle this.
-
+ __ ic_check(2);
+ __ z_lg(Z_method, Address(Z_inline_cache, CompiledICData::speculated_method_offset()));
// This def MUST MATCH code in gen_c2i_adapter!
const Register code = Z_R11;
- __ z_lg(Z_method, holder_metadata_offset, Z_method);
__ load_and_test_long(Z_R0, method_(code));
__ z_brne(ic_miss); // Cache miss: call runtime to handle this.
diff --git a/src/hotspot/cpu/s390/templateTable_s390.cpp b/src/hotspot/cpu/s390/templateTable_s390.cpp
index 62d62a9842f..02b9405ad31 100644
--- a/src/hotspot/cpu/s390/templateTable_s390.cpp
+++ b/src/hotspot/cpu/s390/templateTable_s390.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2023 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -2868,7 +2868,7 @@ void TemplateTable::jvmti_post_field_mod(Register cache,
__ z_lgr(fieldEntry, cache);
if (is_static) {
- // Life is simple. NULL the object pointer.
+ // Life is simple. Null the object pointer.
__ clear_reg(obj, true, false); // Don't set CC.
} else {
// Life is harder. The stack holds the value on top, followed by
@@ -3914,20 +3914,15 @@ void TemplateTable::_new() {
__ z_cli(0, tmp, JVM_CONSTANT_Class);
__ z_brne(slow_case);
- __ z_sllg(offset, offset, LogBytesPerWord); // Convert to to offset.
+ __ z_sllg(offset, offset, LogBytesPerWord); // Convert to offset.
// Get InstanceKlass.
Register iklass = cpool;
__ load_resolved_klass_at_offset(cpool, offset, iklass);
- // Make sure klass is initialized & doesn't have finalizer.
- // Make sure klass is fully initialized.
- const int state_offset = in_bytes(InstanceKlass::init_state_offset());
- if (Immediate::is_uimm12(state_offset)) {
- __ z_cli(state_offset, iklass, InstanceKlass::fully_initialized);
- } else {
- __ z_cliy(state_offset, iklass, InstanceKlass::fully_initialized);
- }
- __ z_brne(slow_case);
+ // make sure klass is initialized
+ assert(VM_Version::supports_fast_class_init_checks(),
+ "Optimization requires support for fast class initialization checks");
+ __ clinit_barrier(iklass, Z_thread, nullptr /*L_fast_path*/, &slow_case);
// Get instance_size in InstanceKlass (scaled to a count of bytes).
Register Rsize = offset;
diff --git a/src/hotspot/cpu/s390/vm_version_s390.hpp b/src/hotspot/cpu/s390/vm_version_s390.hpp
index 93d5b11c473..7ac60a10ae7 100644
--- a/src/hotspot/cpu/s390/vm_version_s390.hpp
+++ b/src/hotspot/cpu/s390/vm_version_s390.hpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2016, 2022, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2022 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -567,7 +567,6 @@ class VM_Version: public Abstract_VM_Version {
static unsigned long z_SIGSEGV();
static void initialize_cpu_information(void);
- static bool profile_all_receivers_at_type_check() { return false; }
};
#endif // CPU_S390_VM_VERSION_S390_HPP
diff --git a/src/hotspot/cpu/s390/vtableStubs_s390.cpp b/src/hotspot/cpu/s390/vtableStubs_s390.cpp
index 5a79369ceab..573c23d7967 100644
--- a/src/hotspot/cpu/s390/vtableStubs_s390.cpp
+++ b/src/hotspot/cpu/s390/vtableStubs_s390.cpp
@@ -1,6 +1,6 @@
/*
* Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2016, 2021 SAP SE. All rights reserved.
+ * Copyright (c) 2016, 2023 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,10 +25,10 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_s390.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klass.inline.hpp"
#include "oops/klassVtable.hpp"
@@ -197,12 +197,12 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
__ load_klass(rcvr_klass, Z_ARG1);
// Receiver subtype check against REFC.
- __ z_lg(interface, Address(Z_method, CompiledICHolder::holder_klass_offset()));
+ __ z_lg(interface, Address(Z_method, CompiledICData::itable_refc_klass_offset()));
__ lookup_interface_method(rcvr_klass, interface, noreg,
noreg, Z_R1, no_such_interface, /*return_method=*/ false);
// Get Method* and entrypoint for compiler
- __ z_lg(interface, Address(Z_method, CompiledICHolder::holder_metadata_offset()));
+ __ z_lg(interface, Address(Z_method, CompiledICData::itable_defc_klass_offset()));
__ lookup_interface_method(rcvr_klass, interface, itable_index,
Z_method, Z_R1, no_such_interface, /*return_method=*/ true);
diff --git a/src/hotspot/cpu/x86/assembler_x86.cpp b/src/hotspot/cpu/x86/assembler_x86.cpp
index 63d6ec9b00f..9482537d84f 100644
--- a/src/hotspot/cpu/x86/assembler_x86.cpp
+++ b/src/hotspot/cpu/x86/assembler_x86.cpp
@@ -1089,7 +1089,7 @@ address Assembler::locate_operand(address inst, WhichOperand which) {
break;
case 0x62: // EVEX_4bytes
- assert(VM_Version::supports_evex(), "shouldn't have EVEX prefix");
+ assert(VM_Version::cpu_supports_evex(), "shouldn't have EVEX prefix");
assert(ip == inst+1, "no prefixes allowed");
// no EVEX collisions, all instructions that have 0x62 opcodes
// have EVEX versions and are subopcodes of 0x66
diff --git a/src/hotspot/cpu/x86/assembler_x86.hpp b/src/hotspot/cpu/x86/assembler_x86.hpp
index 7b907218f35..8b512fac6bc 100644
--- a/src/hotspot/cpu/x86/assembler_x86.hpp
+++ b/src/hotspot/cpu/x86/assembler_x86.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -816,8 +816,8 @@ class Assembler : public AbstractAssembler {
void check_relocation(RelocationHolder const& rspec, int format);
#endif
- void emit_data(jint data, relocInfo::relocType rtype, int format);
- void emit_data(jint data, RelocationHolder const& rspec, int format);
+ void emit_data(jint data, relocInfo::relocType rtype, int format = 0);
+ void emit_data(jint data, RelocationHolder const& rspec, int format = 0);
void emit_data64(jlong data, relocInfo::relocType rtype, int format = 0);
void emit_data64(jlong data, RelocationHolder const& rspec, int format = 0);
diff --git a/src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp b/src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp
index ff0726840d3..3b7a3cec2d8 100644
--- a/src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp
+++ b/src/hotspot/cpu/x86/c1_LIRAssembler_x86.cpp
@@ -72,7 +72,6 @@ static jlong *double_signflip_pool = double_quadword(&fp_signmask_pool[4*2], (jl
NEEDS_CLEANUP // remove this definitions ?
-const Register IC_Klass = rax; // where the IC klass is cached
const Register SYNC_header = rax; // synchronization header
const Register SHIFT_count = rcx; // where count for shift operations must be
@@ -336,23 +335,7 @@ void LIR_Assembler::osr_entry() {
// inline cache check; done before the frame is built.
int LIR_Assembler::check_icache() {
- Register receiver = FrameMap::receiver_opr->as_register();
- Register ic_klass = IC_Klass;
- const int ic_cmp_size = LP64_ONLY(10) NOT_LP64(9);
- const bool do_post_padding = VerifyOops || UseCompressedClassPointers;
- if (!do_post_padding) {
- // insert some nops so that the verified entry point is aligned on CodeEntryAlignment
- __ align(CodeEntryAlignment, __ offset() + ic_cmp_size);
- }
- int offset = __ offset();
- __ inline_cache_check(receiver, IC_Klass);
- assert(__ offset() % CodeEntryAlignment == 0 || do_post_padding, "alignment must be correct");
- if (do_post_padding) {
- // force alignment after the cache check.
- // It's been verified to be aligned if !VerifyOops
- __ align(CodeEntryAlignment);
- }
- return offset;
+ return __ ic_check(CodeEntryAlignment);
}
void LIR_Assembler::clinit_barrier(ciMethod* method) {
diff --git a/src/hotspot/cpu/x86/c1_LIRGenerator_x86.cpp b/src/hotspot/cpu/x86/c1_LIRGenerator_x86.cpp
index b6a27abf0f3..7088cf33cf6 100644
--- a/src/hotspot/cpu/x86/c1_LIRGenerator_x86.cpp
+++ b/src/hotspot/cpu/x86/c1_LIRGenerator_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1207,9 +1207,10 @@ void LIRGenerator::do_vectorizedMismatch(Intrinsic* x) {
__ move(result_reg, result);
}
+#ifndef _LP64
// _i2l, _i2f, _i2d, _l2i, _l2f, _l2d, _f2i, _f2l, _f2d, _d2i, _d2l, _d2f
// _i2b, _i2c, _i2s
-LIR_Opr fixed_register_for(BasicType type) {
+static LIR_Opr fixed_register_for(BasicType type) {
switch (type) {
case T_FLOAT: return FrameMap::fpu0_float_opr;
case T_DOUBLE: return FrameMap::fpu0_double_opr;
@@ -1218,6 +1219,7 @@ LIR_Opr fixed_register_for(BasicType type) {
default: ShouldNotReachHere(); return LIR_OprFact::illegalOpr;
}
}
+#endif
void LIRGenerator::do_Convert(Convert* x) {
#ifdef _LP64
diff --git a/src/hotspot/cpu/x86/c1_MacroAssembler_x86.cpp b/src/hotspot/cpu/x86/c1_MacroAssembler_x86.cpp
index 78361a305ae..0c4544f5bc4 100644
--- a/src/hotspot/cpu/x86/c1_MacroAssembler_x86.cpp
+++ b/src/hotspot/cpu/x86/c1_MacroAssembler_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,6 +25,7 @@
#include "precompiled.hpp"
#include "c1/c1_MacroAssembler.hpp"
#include "c1/c1_Runtime1.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/compilerDefinitions.inline.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetAssembler.hpp"
@@ -34,10 +35,12 @@
#include "oops/arrayOop.hpp"
#include "oops/markWord.hpp"
#include "runtime/basicLock.hpp"
+#include "runtime/globals.hpp"
#include "runtime/os.hpp"
#include "runtime/sharedRuntime.hpp"
#include "runtime/stubRoutines.hpp"
#include "utilities/checkedCast.hpp"
+#include "utilities/globalDefinitions.hpp"
int C1_MacroAssembler::lock_object(Register hdr, Register obj, Register disp_hdr, Register tmp, Label& slow_case) {
const int aligned_mask = BytesPerWord -1;
@@ -60,9 +63,6 @@ int C1_MacroAssembler::lock_object(Register hdr, Register obj, Register disp_hdr
jcc(Assembler::notZero, slow_case);
}
- // Load object header
- movptr(hdr, Address(obj, hdr_offset));
-
if (LockingMode == LM_LIGHTWEIGHT) {
#ifdef _LP64
const Register thread = r15_thread;
@@ -73,6 +73,8 @@ int C1_MacroAssembler::lock_object(Register hdr, Register obj, Register disp_hdr
lightweight_lock(obj, hdr, thread, tmp, slow_case);
} else if (LockingMode == LM_LEGACY) {
Label done;
+ // Load object header
+ movptr(hdr, Address(obj, hdr_offset));
// and mark it as unlocked
orptr(hdr, markWord::unlocked_value);
// save unlocked object header into the displaced header location on the stack
@@ -134,9 +136,14 @@ void C1_MacroAssembler::unlock_object(Register hdr, Register obj, Register disp_
verify_oop(obj);
if (LockingMode == LM_LIGHTWEIGHT) {
- movptr(disp_hdr, Address(obj, hdr_offset));
- andptr(disp_hdr, ~(int32_t)markWord::lock_mask_in_place);
- lightweight_unlock(obj, disp_hdr, hdr, slow_case);
+#ifdef _LP64
+ lightweight_unlock(obj, disp_hdr, r15_thread, hdr, slow_case);
+#else
+ // This relies on the implementation of lightweight_unlock being able to handle
+ // that the reg_rax and thread Register parameters may alias each other.
+ get_thread(disp_hdr);
+ lightweight_unlock(obj, disp_hdr, disp_hdr, hdr, slow_case);
+#endif
} else if (LockingMode == LM_LEGACY) {
// test if object header is pointing to the displaced header, and if so, restore
// the displaced header in the object - if the object header is not pointing to
@@ -295,30 +302,6 @@ void C1_MacroAssembler::allocate_array(Register obj, Register len, Register t1,
verify_oop(obj);
}
-
-
-void C1_MacroAssembler::inline_cache_check(Register receiver, Register iCache) {
- verify_oop(receiver);
- // explicit null check not needed since load from [klass_offset] causes a trap
- // check against inline cache
- assert(!MacroAssembler::needs_explicit_null_check(oopDesc::klass_offset_in_bytes()), "must add explicit null check");
- int start_offset = offset();
-
- if (UseCompressedClassPointers) {
- load_klass(rscratch1, receiver, rscratch2);
- cmpptr(rscratch1, iCache);
- } else {
- cmpptr(iCache, Address(receiver, oopDesc::klass_offset_in_bytes()));
- }
- // if icache check fails, then jump to runtime routine
- // Note: RECEIVER must still contain the receiver!
- jump_cc(Assembler::notEqual,
- RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
- const int ic_cmp_size = LP64_ONLY(10) NOT_LP64(9);
- assert(UseCompressedClassPointers || offset() - start_offset == ic_cmp_size, "check alignment in emit_method_entry");
-}
-
-
void C1_MacroAssembler::build_frame(int frame_size_in_bytes, int bang_size_in_bytes) {
assert(bang_size_in_bytes >= frame_size_in_bytes, "stack bang size incorrect");
// Make sure there is enough stack space for this method's activation.
diff --git a/src/hotspot/cpu/x86/c1_Runtime1_x86.cpp b/src/hotspot/cpu/x86/c1_Runtime1_x86.cpp
index 8b56f464f27..2c24c0c2cfb 100644
--- a/src/hotspot/cpu/x86/c1_Runtime1_x86.cpp
+++ b/src/hotspot/cpu/x86/c1_Runtime1_x86.cpp
@@ -38,7 +38,6 @@
#include "interpreter/interpreter.hpp"
#include "memory/universe.hpp"
#include "nativeInst_x86.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/oop.inline.hpp"
#include "prims/jvmtiExport.hpp"
#include "register_x86.hpp"
diff --git a/src/hotspot/cpu/x86/c2_CodeStubs_x86.cpp b/src/hotspot/cpu/x86/c2_CodeStubs_x86.cpp
index b9b4e8af02c..6dc8d14064a 100644
--- a/src/hotspot/cpu/x86/c2_CodeStubs_x86.cpp
+++ b/src/hotspot/cpu/x86/c2_CodeStubs_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -73,26 +73,74 @@ void C2EntryBarrierStub::emit(C2_MacroAssembler& masm) {
__ jmp(continuation(), false /* maybe_short */);
}
-#ifdef _LP64
-int C2HandleAnonOMOwnerStub::max_size() const {
- // Max size of stub has been determined by testing with 0, in which case
- // C2CodeStubList::emit() will throw an assertion and report the actual size that
- // is needed.
- return DEBUG_ONLY(36) NOT_DEBUG(21);
+int C2FastUnlockLightweightStub::max_size() const {
+ return 128;
}
-void C2HandleAnonOMOwnerStub::emit(C2_MacroAssembler& masm) {
- __ bind(entry());
- Register mon = monitor();
- Register t = tmp();
- __ movptr(Address(mon, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)), r15_thread);
- __ subl(Address(r15_thread, JavaThread::lock_stack_top_offset()), oopSize);
+void C2FastUnlockLightweightStub::emit(C2_MacroAssembler& masm) {
+ assert(_t == rax, "must be");
+
+ Label restore_held_monitor_count_and_slow_path;
+
+ { // Restore lock-stack and handle the unlock in runtime.
+
+ __ bind(_push_and_slow_path);
#ifdef ASSERT
- __ movl(t, Address(r15_thread, JavaThread::lock_stack_top_offset()));
- __ movptr(Address(r15_thread, t), 0);
+ // The obj was only cleared in debug.
+ __ movl(_t, Address(_thread, JavaThread::lock_stack_top_offset()));
+ __ movptr(Address(_thread, _t), _obj);
#endif
- __ jmp(continuation());
-}
+ __ addl(Address(_thread, JavaThread::lock_stack_top_offset()), oopSize);
+ }
+
+ { // Restore held monitor count and slow path.
+
+ __ bind(restore_held_monitor_count_and_slow_path);
+ // Restore held monitor count.
+ __ increment(Address(_thread, JavaThread::held_monitor_count_offset()));
+ // increment will always result in ZF = 0 (no overflows).
+ __ jmp(slow_path_continuation());
+ }
+
+ { // Handle monitor medium path.
+
+ __ bind(_check_successor);
+
+ Label fix_zf_and_unlocked;
+ const Register monitor = _mark;
+
+#ifndef _LP64
+ __ jmpb(restore_held_monitor_count_and_slow_path);
+#else // _LP64
+ // successor null check.
+ __ cmpptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(succ)), NULL_WORD);
+ __ jccb(Assembler::equal, restore_held_monitor_count_and_slow_path);
+
+ // Release lock.
+ __ movptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)), NULL_WORD);
+
+ // Fence.
+ // Instead of MFENCE we use a dummy locked add of 0 to the top-of-stack.
+ __ lock(); __ addl(Address(rsp, 0), 0);
+
+ // Recheck successor.
+ __ cmpptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(succ)), NULL_WORD);
+ // Observed a successor after the release -> fence we have handed off the monitor
+ __ jccb(Assembler::notEqual, fix_zf_and_unlocked);
+
+ // Try to relock, if it fails the monitor has been handed over
+ // TODO: Caveat, this may fail due to deflation, which does
+ // not handle the monitor handoff. Currently only works
+ // due to the responsible thread.
+ __ xorptr(rax, rax);
+ __ lock(); __ cmpxchgptr(_thread, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)));
+ __ jccb (Assembler::equal, restore_held_monitor_count_and_slow_path);
#endif
+ __ bind(fix_zf_and_unlocked);
+ __ xorl(rax, rax);
+ __ jmp(unlocked_continuation());
+ }
+}
+
#undef __
diff --git a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp
index 3817c38f4ba..b6ecde62af6 100644
--- a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp
+++ b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp
@@ -33,9 +33,13 @@
#include "opto/output.hpp"
#include "opto/opcodes.hpp"
#include "opto/subnode.hpp"
+#include "runtime/globals.hpp"
#include "runtime/objectMonitor.hpp"
#include "runtime/stubRoutines.hpp"
#include "utilities/checkedCast.hpp"
+#include "utilities/globalDefinitions.hpp"
+#include "utilities/powerOfTwo.hpp"
+#include "utilities/sizes.hpp"
#ifdef PRODUCT
#define BLOCK_COMMENT(str) /* nothing */
@@ -554,6 +558,7 @@ void C2_MacroAssembler::fast_lock(Register objReg, Register boxReg, Register tmp
RTMLockingCounters* stack_rtm_counters,
Metadata* method_data,
bool use_rtm, bool profile_rtm) {
+ assert(LockingMode != LM_LIGHTWEIGHT, "lightweight locking should use fast_lock_lightweight");
// Ensure the register assignments are disjoint
assert(tmpReg == rax, "");
@@ -605,7 +610,8 @@ void C2_MacroAssembler::fast_lock(Register objReg, Register boxReg, Register tmp
if (LockingMode == LM_MONITOR) {
// Clear ZF so that we take the slow path at the DONE label. objReg is known to be not 0.
testptr(objReg, objReg);
- } else if (LockingMode == LM_LEGACY) {
+ } else {
+ assert(LockingMode == LM_LEGACY, "must be");
// Attempt stack-locking ...
orptr (tmpReg, markWord::unlocked_value);
movptr(Address(boxReg, 0), tmpReg); // Anticipate successful CAS
@@ -620,10 +626,6 @@ void C2_MacroAssembler::fast_lock(Register objReg, Register boxReg, Register tmp
// Next instruction set ZFlag == 1 (Success) if difference is less then one page.
andptr(tmpReg, (int32_t) (NOT_LP64(0xFFFFF003) LP64_ONLY(7 - (int)os::vm_page_size())) );
movptr(Address(boxReg, 0), tmpReg);
- } else {
- assert(LockingMode == LM_LIGHTWEIGHT, "");
- lightweight_lock(objReg, tmpReg, thread, scrReg, NO_COUNT);
- jmp(COUNT);
}
jmp(DONE_LABEL);
@@ -754,6 +756,7 @@ void C2_MacroAssembler::fast_lock(Register objReg, Register boxReg, Register tmp
// Xcheck:jni is enabled.
void C2_MacroAssembler::fast_unlock(Register objReg, Register boxReg, Register tmpReg, bool use_rtm) {
+ assert(LockingMode != LM_LIGHTWEIGHT, "lightweight locking should use fast_unlock_lightweight");
assert(boxReg == rax, "");
assert_different_registers(objReg, boxReg, tmpReg);
@@ -784,23 +787,6 @@ void C2_MacroAssembler::fast_unlock(Register objReg, Register boxReg, Register t
}
// It's inflated.
- if (LockingMode == LM_LIGHTWEIGHT) {
- // If the owner is ANONYMOUS, we need to fix it - in an outline stub.
- testb(Address(tmpReg, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)), (int32_t) ObjectMonitor::ANONYMOUS_OWNER);
-#ifdef _LP64
- if (!Compile::current()->output()->in_scratch_emit_size()) {
- C2HandleAnonOMOwnerStub* stub = new (Compile::current()->comp_arena()) C2HandleAnonOMOwnerStub(tmpReg, boxReg);
- Compile::current()->output()->add_stub(stub);
- jcc(Assembler::notEqual, stub->entry());
- bind(stub->continuation());
- } else
-#endif
- {
- // We can't easily implement this optimization on 32 bit because we don't have a thread register.
- // Call the slow-path instead.
- jcc(Assembler::notEqual, NO_COUNT);
- }
- }
#if INCLUDE_RTM_OPT
if (use_rtm) {
@@ -922,19 +908,14 @@ void C2_MacroAssembler::fast_unlock(Register objReg, Register boxReg, Register t
jmpb (DONE_LABEL);
#endif
- if (LockingMode != LM_MONITOR) {
+ if (LockingMode == LM_LEGACY) {
bind (Stacked);
- if (LockingMode == LM_LIGHTWEIGHT) {
- mov(boxReg, tmpReg);
- lightweight_unlock(objReg, boxReg, tmpReg, NO_COUNT);
- jmp(COUNT);
- } else if (LockingMode == LM_LEGACY) {
- movptr(tmpReg, Address (boxReg, 0)); // re-fetch
- lock();
- cmpxchgptr(tmpReg, Address(objReg, oopDesc::mark_offset_in_bytes())); // Uses RAX which is box
- }
+ movptr(tmpReg, Address (boxReg, 0)); // re-fetch
+ lock();
+ cmpxchgptr(tmpReg, Address(objReg, oopDesc::mark_offset_in_bytes())); // Uses RAX which is box
// Intentional fall-thru into DONE_LABEL
}
+
bind(DONE_LABEL);
// ZFlag == 1 count in fast path
@@ -955,6 +936,247 @@ void C2_MacroAssembler::fast_unlock(Register objReg, Register boxReg, Register t
bind(NO_COUNT);
}
+void C2_MacroAssembler::fast_lock_lightweight(Register obj, Register box, Register rax_reg,
+ Register t, Register thread) {
+ assert(LockingMode == LM_LIGHTWEIGHT, "must be");
+ assert(rax_reg == rax, "Used for CAS");
+ assert_different_registers(obj, box, rax_reg, t, thread);
+
+ // Handle inflated monitor.
+ Label inflated;
+ // Finish fast lock successfully. ZF value is irrelevant.
+ Label locked;
+ // Finish fast lock unsuccessfully. MUST jump with ZF == 0
+ Label slow_path;
+
+ if (DiagnoseSyncOnValueBasedClasses != 0) {
+ load_klass(rax_reg, obj, t);
+ movl(rax_reg, Address(rax_reg, Klass::access_flags_offset()));
+ testl(rax_reg, JVM_ACC_IS_VALUE_BASED_CLASS);
+ jcc(Assembler::notZero, slow_path);
+ }
+
+ const Register mark = t;
+
+ { // Lightweight Lock
+
+ Label push;
+
+ const Register top = box;
+
+ // Load the mark.
+ movptr(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
+
+ // Prefetch top.
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+
+ // Check for monitor (0b10).
+ testptr(mark, markWord::monitor_value);
+ jcc(Assembler::notZero, inflated);
+
+ // Check if lock-stack is full.
+ cmpl(top, LockStack::end_offset() - 1);
+ jcc(Assembler::greater, slow_path);
+
+ // Check if recursive.
+ cmpptr(obj, Address(thread, top, Address::times_1, -oopSize));
+ jccb(Assembler::equal, push);
+
+ // Try to lock. Transition lock bits 0b01 => 0b00
+ movptr(rax_reg, mark);
+ orptr(rax_reg, markWord::unlocked_value);
+ andptr(mark, ~(int32_t)markWord::unlocked_value);
+ lock(); cmpxchgptr(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
+ jcc(Assembler::notEqual, slow_path);
+
+ bind(push);
+ // After successful lock, push object on lock-stack.
+ movptr(Address(thread, top), obj);
+ addl(Address(thread, JavaThread::lock_stack_top_offset()), oopSize);
+ jmpb(locked);
+ }
+
+ { // Handle inflated monitor.
+ bind(inflated);
+
+ const Register tagged_monitor = mark;
+
+ // CAS owner (null => current thread).
+ xorptr(rax_reg, rax_reg);
+ lock(); cmpxchgptr(thread, Address(tagged_monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)));
+ jccb(Assembler::equal, locked);
+
+ // Check if recursive.
+ cmpptr(thread, rax_reg);
+ jccb(Assembler::notEqual, slow_path);
+
+ // Recursive.
+ increment(Address(tagged_monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(recursions)));
+ }
+
+ bind(locked);
+ increment(Address(thread, JavaThread::held_monitor_count_offset()));
+ // Set ZF = 1
+ xorl(rax_reg, rax_reg);
+
+#ifdef ASSERT
+ // Check that locked label is reached with ZF set.
+ Label zf_correct;
+ jccb(Assembler::zero, zf_correct);
+ stop("Fast Lock ZF != 1");
+#endif
+
+ bind(slow_path);
+#ifdef ASSERT
+ // Check that slow_path label is reached with ZF not set.
+ jccb(Assembler::notZero, zf_correct);
+ stop("Fast Lock ZF != 0");
+ bind(zf_correct);
+#endif
+ // C2 uses the value of ZF to determine the continuation.
+}
+
+void C2_MacroAssembler::fast_unlock_lightweight(Register obj, Register reg_rax, Register t, Register thread) {
+ assert(LockingMode == LM_LIGHTWEIGHT, "must be");
+ assert(reg_rax == rax, "Used for CAS");
+ assert_different_registers(obj, reg_rax, t);
+
+ // Handle inflated monitor.
+ Label inflated, inflated_check_lock_stack;
+ // Finish fast unlock successfully. MUST jump with ZF == 1
+ Label unlocked;
+
+ // Assume success.
+ decrement(Address(thread, JavaThread::held_monitor_count_offset()));
+
+ const Register mark = t;
+ const Register top = reg_rax;
+
+ Label dummy;
+ C2FastUnlockLightweightStub* stub = nullptr;
+
+ if (!Compile::current()->output()->in_scratch_emit_size()) {
+ stub = new (Compile::current()->comp_arena()) C2FastUnlockLightweightStub(obj, mark, reg_rax, thread);
+ Compile::current()->output()->add_stub(stub);
+ }
+
+ Label& push_and_slow_path = stub == nullptr ? dummy : stub->push_and_slow_path();
+ Label& check_successor = stub == nullptr ? dummy : stub->check_successor();
+
+ { // Lightweight Unlock
+
+ // Load top.
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+
+ // Prefetch mark.
+ movptr(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
+
+ // Check if obj is top of lock-stack.
+ cmpptr(obj, Address(thread, top, Address::times_1, -oopSize));
+ // Top of lock stack was not obj. Must be monitor.
+ jcc(Assembler::notEqual, inflated_check_lock_stack);
+
+ // Pop lock-stack.
+ DEBUG_ONLY(movptr(Address(thread, top, Address::times_1, -oopSize), 0);)
+ subl(Address(thread, JavaThread::lock_stack_top_offset()), oopSize);
+
+ // Check if recursive.
+ cmpptr(obj, Address(thread, top, Address::times_1, -2 * oopSize));
+ jcc(Assembler::equal, unlocked);
+
+ // We elide the monitor check, let the CAS fail instead.
+
+ // Try to unlock. Transition lock bits 0b00 => 0b01
+ movptr(reg_rax, mark);
+ andptr(reg_rax, ~(int32_t)markWord::lock_mask);
+ orptr(mark, markWord::unlocked_value);
+ lock(); cmpxchgptr(mark, Address(obj, oopDesc::mark_offset_in_bytes()));
+ jcc(Assembler::notEqual, push_and_slow_path);
+ jmp(unlocked);
+ }
+
+
+ { // Handle inflated monitor.
+ bind(inflated_check_lock_stack);
+#ifdef ASSERT
+ Label check_done;
+ subl(top, oopSize);
+ cmpl(top, in_bytes(JavaThread::lock_stack_base_offset()));
+ jcc(Assembler::below, check_done);
+ cmpptr(obj, Address(thread, top));
+ jccb(Assembler::notEqual, inflated_check_lock_stack);
+ stop("Fast Unlock lock on stack");
+ bind(check_done);
+ testptr(mark, markWord::monitor_value);
+ jccb(Assembler::notZero, inflated);
+ stop("Fast Unlock not monitor");
+#endif
+
+ bind(inflated);
+
+ // mark contains the tagged ObjectMonitor*.
+ const Register monitor = mark;
+
+#ifndef _LP64
+ // Check if recursive.
+ xorptr(reg_rax, reg_rax);
+ orptr(reg_rax, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(recursions)));
+ jcc(Assembler::notZero, check_successor);
+
+ // Check if the entry lists are empty.
+ movptr(reg_rax, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(EntryList)));
+ orptr(reg_rax, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(cxq)));
+ jcc(Assembler::notZero, check_successor);
+
+ // Release lock.
+ movptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)), NULL_WORD);
+#else // _LP64
+ Label recursive;
+
+ // Check if recursive.
+ cmpptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(recursions)), 0);
+ jccb(Assembler::notEqual, recursive);
+
+ // Check if the entry lists are empty.
+ movptr(reg_rax, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(cxq)));
+ orptr(reg_rax, Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(EntryList)));
+ jcc(Assembler::notZero, check_successor);
+
+ // Release lock.
+ movptr(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(owner)), NULL_WORD);
+ jmpb(unlocked);
+
+ // Recursive unlock.
+ bind(recursive);
+ decrement(Address(monitor, OM_OFFSET_NO_MONITOR_VALUE_TAG(recursions)));
+ xorl(t, t);
+#endif
+ }
+
+ bind(unlocked);
+ if (stub != nullptr) {
+ bind(stub->unlocked_continuation());
+ }
+
+#ifdef ASSERT
+ // Check that unlocked label is reached with ZF set.
+ Label zf_correct;
+ jccb(Assembler::zero, zf_correct);
+ stop("Fast Unlock ZF != 1");
+#endif
+
+ if (stub != nullptr) {
+ bind(stub->slow_path_continuation());
+ }
+#ifdef ASSERT
+ // Check that stub->continuation() label is reached with ZF not set.
+ jccb(Assembler::notZero, zf_correct);
+ stop("Fast Unlock ZF != 0");
+ bind(zf_correct);
+#endif
+ // C2 uses the value of ZF to determine the continuation.
+}
+
//-------------------------------------------------------------------------------------------
// Generic instructions support for use in .ad files C2 code generation
@@ -5282,6 +5504,42 @@ void C2_MacroAssembler::vector_mask_compress(KRegister dst, KRegister src, Regis
kmov(dst, rtmp2);
}
+#ifdef _LP64
+void C2_MacroAssembler::vector_compress_expand_avx2(int opcode, XMMRegister dst, XMMRegister src,
+ XMMRegister mask, Register rtmp, Register rscratch,
+ XMMRegister permv, XMMRegister xtmp, BasicType bt,
+ int vec_enc) {
+ assert(type2aelembytes(bt) >= 4, "");
+ assert(opcode == Op_CompressV || opcode == Op_ExpandV, "");
+ address compress_perm_table = nullptr;
+ address expand_perm_table = nullptr;
+ if (type2aelembytes(bt) == 8) {
+ compress_perm_table = StubRoutines::x86::compress_perm_table64();
+ expand_perm_table = StubRoutines::x86::expand_perm_table64();
+ vmovmskpd(rtmp, mask, vec_enc);
+ } else {
+ compress_perm_table = StubRoutines::x86::compress_perm_table32();
+ expand_perm_table = StubRoutines::x86::expand_perm_table32();
+ vmovmskps(rtmp, mask, vec_enc);
+ }
+ shlq(rtmp, 5); // for 32 byte permute row.
+ if (opcode == Op_CompressV) {
+ lea(rscratch, ExternalAddress(compress_perm_table));
+ } else {
+ lea(rscratch, ExternalAddress(expand_perm_table));
+ }
+ addptr(rtmp, rscratch);
+ vmovdqu(permv, Address(rtmp));
+ vpermps(dst, permv, src, Assembler::AVX_256bit);
+ vpxor(xtmp, xtmp, xtmp, vec_enc);
+ // Blend the result with zero vector using permute mask, each column entry
+ // in a permute table row contains either a valid permute index or a -1 (default)
+ // value, this can potentially be used as a blending mask after
+ // compressing/expanding the source vector lanes.
+ vblendvps(dst, dst, xtmp, permv, vec_enc, false, permv);
+}
+#endif
+
void C2_MacroAssembler::vector_compress_expand(int opcode, XMMRegister dst, XMMRegister src, KRegister mask,
bool merge, BasicType bt, int vec_enc) {
if (opcode == Op_CompressV) {
diff --git a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp
index e9e1412957b..26f7fb44aa9 100644
--- a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp
+++ b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -43,6 +43,10 @@
bool use_rtm, bool profile_rtm);
void fast_unlock(Register obj, Register box, Register tmp, bool use_rtm);
+ void fast_lock_lightweight(Register obj, Register box, Register rax_reg,
+ Register t, Register thread);
+ void fast_unlock_lightweight(Register obj, Register reg_rax, Register t, Register thread);
+
#if INCLUDE_RTM_OPT
void rtm_counters_update(Register abort_status, Register rtm_counters);
void branch_on_random_using_rdtsc(Register tmp, Register scr, int count, Label& brLabel);
@@ -390,6 +394,10 @@
void vector_round_float_avx(XMMRegister dst, XMMRegister src, AddressLiteral float_sign_flip, AddressLiteral new_mxcsr, int vec_enc,
Register tmp, XMMRegister xtmp1, XMMRegister xtmp2, XMMRegister xtmp3, XMMRegister xtmp4);
+
+ void vector_compress_expand_avx2(int opcode, XMMRegister dst, XMMRegister src, XMMRegister mask,
+ Register rtmp, Register rscratch, XMMRegister permv, XMMRegister xtmp,
+ BasicType bt, int vec_enc);
#endif // _LP64
void udivI(Register rax, Register divisor, Register rdx);
diff --git a/src/hotspot/cpu/x86/compiledIC_x86.cpp b/src/hotspot/cpu/x86/compiledIC_x86.cpp
index 8fc001039fb..95b41f62b6a 100644
--- a/src/hotspot/cpu/x86/compiledIC_x86.cpp
+++ b/src/hotspot/cpu/x86/compiledIC_x86.cpp
@@ -26,7 +26,6 @@
#include "asm/macroAssembler.inline.hpp"
#include "code/codeCache.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
@@ -36,7 +35,7 @@
// ----------------------------------------------------------------------------
#define __ _masm.
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
// Stub is fixed up when the corresponding call is converted from
// calling compiled code to calling interpreted code.
// movq rbx, 0
@@ -66,32 +65,25 @@ address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark)
}
#undef __
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
return NOT_LP64(10) // movl; jmp
LP64_ONLY(15); // movq (1+1+8); jmp (1+4)
}
-int CompiledStaticCall::to_trampoline_stub_size() {
+int CompiledDirectCall::to_trampoline_stub_size() {
// x86 doesn't use trampolines.
return 0;
}
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
return 4; // 3 in emit_to_interp_stub + 1 in emit_call
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
address stub = find_stub();
guarantee(stub != nullptr, "stub not found");
- {
- ResourceMark rm;
- log_trace(inlinecache)("CompiledDirectStaticCall@" INTPTR_FORMAT ": set_to_interpreted %s",
- p2i(instruction_address()),
- callee->name_and_sig_as_C_string());
- }
-
// Creation also verifies the object.
NativeMovConstReg* method_holder = nativeMovConstReg_at(stub);
NativeJump* jump = nativeJump_at(method_holder->next_instruction_address());
@@ -105,7 +97,7 @@ void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, ad
set_destination_mt_safe(stub);
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
assert(CompiledICLocker::is_safe(static_stub->addr()), "mt unsafe call");
// Reset stub.
address stub = static_stub->addr();
@@ -122,7 +114,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
// Verify call.
_call->verify();
_call->verify_alignment();
diff --git a/src/hotspot/cpu/x86/icBuffer_x86.cpp b/src/hotspot/cpu/x86/icBuffer_x86.cpp
deleted file mode 100644
index af374b57416..00000000000
--- a/src/hotspot/cpu/x86/icBuffer_x86.cpp
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/macroAssembler.hpp"
-#include "asm/macroAssembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_x86.hpp"
-#include "oops/oop.inline.hpp"
-
-int InlineCacheBuffer::ic_stub_code_size() {
- // Worst case, if destination is not a near call:
- // lea rax, lit1
- // lea scratch, lit2
- // jmp scratch
-
- // Best case
- // lea rax, lit1
- // jmp lit2
-
- int best = NativeMovConstReg::instruction_size + NativeJump::instruction_size;
- int worst = 2 * NativeMovConstReg::instruction_size + 3;
- return MAX2(best, worst);
-}
-
-
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point) {
- ResourceMark rm;
- CodeBuffer code(code_begin, ic_stub_code_size());
- MacroAssembler* masm = new MacroAssembler(&code);
- // note: even though the code contains an embedded value, we do not need reloc info
- // because
- // (1) the value is old (i.e., doesn't matter for scavenges)
- // (2) these ICStubs are removed *before* a GC happens, so the roots disappear
- // assert(cached_value == nullptr || cached_oop->is_perm(), "must be perm oop");
- masm->lea(rax, AddressLiteral((address) cached_value, relocInfo::metadata_type));
- masm->jump(ExternalAddress(entry_point));
-}
-
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin); // creation also verifies the object
- address jmp = move->next_instruction_address();
- NativeInstruction* ni = nativeInstruction_at(jmp);
- if (ni->is_jump()) {
- NativeJump* jump = nativeJump_at(jmp);
- return jump->jump_destination();
- } else {
- assert(ni->is_far_jump(), "unexpected instruction");
- NativeFarJump* jump = nativeFarJump_at(jmp);
- return jump->jump_destination();
- }
-}
-
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- // creation also verifies the object
- NativeMovConstReg* move = nativeMovConstReg_at(code_begin);
- // Verifies the jump
- address jmp = move->next_instruction_address();
- NativeInstruction* ni = nativeInstruction_at(jmp);
- if (ni->is_jump()) {
- NativeJump* jump = nativeJump_at(jmp);
- } else {
- assert(ni->is_far_jump(), "unexpected instruction");
- NativeFarJump* jump = nativeFarJump_at(jmp);
- }
- void* o = (void*)move->data();
- return o;
-}
diff --git a/src/hotspot/cpu/x86/interp_masm_x86.cpp b/src/hotspot/cpu/x86/interp_masm_x86.cpp
index f5f83ae21f4..33570f3155b 100644
--- a/src/hotspot/cpu/x86/interp_masm_x86.cpp
+++ b/src/hotspot/cpu/x86/interp_masm_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1192,8 +1192,6 @@ void InterpreterMacroAssembler::lock_object(Register lock_reg) {
const Register thread = lock_reg;
get_thread(thread);
#endif
- // Load object header, prepare for CAS from unlocked to locked.
- movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
lightweight_lock(obj_reg, swap_reg, thread, tmp_reg, slow_case);
} else if (LockingMode == LM_LEGACY) {
// Load immediate 1 into swap_reg %rax
@@ -1311,20 +1309,13 @@ void InterpreterMacroAssembler::unlock_object(Register lock_reg) {
if (LockingMode == LM_LIGHTWEIGHT) {
#ifdef _LP64
- const Register thread = r15_thread;
+ lightweight_unlock(obj_reg, swap_reg, r15_thread, header_reg, slow_case);
#else
- const Register thread = header_reg;
- get_thread(thread);
+ // This relies on the implementation of lightweight_unlock being able to handle
+ // that the reg_rax and thread Register parameters may alias each other.
+ get_thread(swap_reg);
+ lightweight_unlock(obj_reg, swap_reg, swap_reg, header_reg, slow_case);
#endif
- // Handle unstructured locking.
- Register tmp = swap_reg;
- movl(tmp, Address(thread, JavaThread::lock_stack_top_offset()));
- cmpptr(obj_reg, Address(thread, tmp, Address::times_1, -oopSize));
- jcc(Assembler::notEqual, slow_case);
- // Try to swing header from locked to unlocked.
- movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
- andptr(swap_reg, ~(int32_t)markWord::lock_mask_in_place);
- lightweight_unlock(obj_reg, swap_reg, header_reg, slow_case);
} else if (LockingMode == LM_LEGACY) {
// Load the old header from BasicLock structure
movptr(header_reg, Address(swap_reg,
diff --git a/src/hotspot/cpu/x86/macroAssembler_x86.cpp b/src/hotspot/cpu/x86/macroAssembler_x86.cpp
index 88296656485..f0e7a08dd5f 100644
--- a/src/hotspot/cpu/x86/macroAssembler_x86.cpp
+++ b/src/hotspot/cpu/x86/macroAssembler_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,6 +25,7 @@
#include "precompiled.hpp"
#include "asm/assembler.hpp"
#include "asm/assembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/compiler_globals.hpp"
#include "compiler/disassembler.hpp"
#include "crc32c.h"
@@ -1341,13 +1342,45 @@ void MacroAssembler::ic_call(address entry, jint method_index) {
RelocationHolder rh = virtual_call_Relocation::spec(pc(), method_index);
#ifdef _LP64
// Needs full 64-bit immediate for later patching.
- mov64(rax, (intptr_t)Universe::non_oop_word());
+ mov64(rax, (int64_t)Universe::non_oop_word());
#else
movptr(rax, (intptr_t)Universe::non_oop_word());
#endif
call(AddressLiteral(entry, rh));
}
+int MacroAssembler::ic_check_size() {
+ return LP64_ONLY(14) NOT_LP64(12);
+}
+
+int MacroAssembler::ic_check(int end_alignment) {
+ Register receiver = LP64_ONLY(j_rarg0) NOT_LP64(rcx);
+ Register data = rax;
+ Register temp = LP64_ONLY(rscratch1) NOT_LP64(rbx);
+
+ // The UEP of a code blob ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately. That's why we align
+ // before the inline cache check here, and not after
+ align(end_alignment, offset() + ic_check_size());
+
+ int uep_offset = offset();
+
+ if (UseCompressedClassPointers) {
+ movl(temp, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ cmpl(temp, Address(data, CompiledICData::speculated_klass_offset()));
+ } else {
+ movptr(temp, Address(receiver, oopDesc::klass_offset_in_bytes()));
+ cmpptr(temp, Address(data, CompiledICData::speculated_klass_offset()));
+ }
+
+ // if inline cache check fails, then jump to runtime routine
+ jump_cc(Assembler::notEqual, RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
+ assert((offset() % end_alignment) == 0, "Misaligned verified entry point");
+
+ return uep_offset;
+}
+
void MacroAssembler::emit_static_call_stub() {
// Static stub relocation also tags the Method* in the code-stream.
mov_metadata(rbx, (Metadata*) nullptr); // Method is zapped till fixup time.
@@ -2568,7 +2601,9 @@ void MacroAssembler::movptr(Register dst, Address src) {
// src should NEVER be a real pointer. Use AddressLiteral for true pointers
void MacroAssembler::movptr(Register dst, intptr_t src) {
#ifdef _LP64
- if (is_simm32(src)) {
+ if (is_uimm32(src)) {
+ movl(dst, checked_cast(src));
+ } else if (is_simm32(src)) {
movq(dst, checked_cast(src));
} else {
mov64(dst, src);
@@ -4085,8 +4120,9 @@ static void restore_xmm_register(MacroAssembler* masm, int offset, XMMRegister r
}
}
-int register_section_sizes(RegSet gp_registers, XMMRegSet xmm_registers, bool save_fpu,
- int& gp_area_size, int& fp_area_size, int& xmm_area_size) {
+static int register_section_sizes(RegSet gp_registers, XMMRegSet xmm_registers,
+ bool save_fpu, int& gp_area_size,
+ int& fp_area_size, int& xmm_area_size) {
gp_area_size = align_up(gp_registers.size() * Register::max_slots_per_register * VMRegImpl::stack_slot_size,
StackAlignmentInBytes);
@@ -4352,7 +4388,7 @@ void MacroAssembler::lookup_interface_method(Register recv_klass,
}
// Look up the method for a megamorphic invokeinterface call in a single pass over itable:
-// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICHolder
+// - check recv_klass (actual object class) is a subtype of resolved_klass from CompiledICData
// - find a holder_klass (class that implements the method) vtable offset and get the method from vtable by index
// The target method is determined by .
// The receiver klass is in recv_klass.
@@ -9875,68 +9911,116 @@ void MacroAssembler::check_stack_alignment(Register sp, const char* msg, unsigne
}
// Implements lightweight-locking.
-// Branches to slow upon failure to lock the object, with ZF cleared.
-// Falls through upon success with unspecified ZF.
//
// obj: the object to be locked
-// hdr: the (pre-loaded) header of the object, must be rax
+// reg_rax: rax
// thread: the thread which attempts to lock obj
// tmp: a temporary register
-void MacroAssembler::lightweight_lock(Register obj, Register hdr, Register thread, Register tmp, Label& slow) {
- assert(hdr == rax, "header must be in rax for cmpxchg");
- assert_different_registers(obj, hdr, thread, tmp);
-
- // First we need to check if the lock-stack has room for pushing the object reference.
- // Note: we subtract 1 from the end-offset so that we can do a 'greater' comparison, instead
- // of 'greaterEqual' below, which readily clears the ZF. This makes C2 code a little simpler and
- // avoids one branch.
- cmpl(Address(thread, JavaThread::lock_stack_top_offset()), LockStack::end_offset() - 1);
- jcc(Assembler::greater, slow);
-
- // Now we attempt to take the fast-lock.
- // Clear lock_mask bits (locked state).
- andptr(hdr, ~(int32_t)markWord::lock_mask_in_place);
- movptr(tmp, hdr);
- // Set unlocked_value bit.
- orptr(hdr, markWord::unlocked_value);
- lock();
- cmpxchgptr(tmp, Address(obj, oopDesc::mark_offset_in_bytes()));
+void MacroAssembler::lightweight_lock(Register obj, Register reg_rax, Register thread, Register tmp, Label& slow) {
+ assert(reg_rax == rax, "");
+ assert_different_registers(obj, reg_rax, thread, tmp);
+
+ Label push;
+ const Register top = tmp;
+
+ // Preload the markWord. It is important that this is the first
+ // instruction emitted as it is part of C1's null check semantics.
+ movptr(reg_rax, Address(obj, oopDesc::mark_offset_in_bytes()));
+
+ // Load top.
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+
+ // Check if the lock-stack is full.
+ cmpl(top, LockStack::end_offset());
+ jcc(Assembler::greaterEqual, slow);
+
+ // Check for recursion.
+ cmpptr(obj, Address(thread, top, Address::times_1, -oopSize));
+ jcc(Assembler::equal, push);
+
+ // Check header for monitor (0b10).
+ testptr(reg_rax, markWord::monitor_value);
+ jcc(Assembler::notZero, slow);
+
+ // Try to lock. Transition lock bits 0b01 => 0b00
+ movptr(tmp, reg_rax);
+ andptr(tmp, ~(int32_t)markWord::unlocked_value);
+ orptr(reg_rax, markWord::unlocked_value);
+ lock(); cmpxchgptr(tmp, Address(obj, oopDesc::mark_offset_in_bytes()));
jcc(Assembler::notEqual, slow);
- // If successful, push object to lock-stack.
- movl(tmp, Address(thread, JavaThread::lock_stack_top_offset()));
- movptr(Address(thread, tmp), obj);
- incrementl(tmp, oopSize);
- movl(Address(thread, JavaThread::lock_stack_top_offset()), tmp);
+ // Restore top, CAS clobbers register.
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+
+ bind(push);
+ // After successful lock, push object on lock-stack.
+ movptr(Address(thread, top), obj);
+ incrementl(top, oopSize);
+ movl(Address(thread, JavaThread::lock_stack_top_offset()), top);
}
// Implements lightweight-unlocking.
-// Branches to slow upon failure, with ZF cleared.
-// Falls through upon success, with unspecified ZF.
//
// obj: the object to be unlocked
-// hdr: the (pre-loaded) header of the object, must be rax
+// reg_rax: rax
+// thread: the thread
// tmp: a temporary register
-void MacroAssembler::lightweight_unlock(Register obj, Register hdr, Register tmp, Label& slow) {
- assert(hdr == rax, "header must be in rax for cmpxchg");
- assert_different_registers(obj, hdr, tmp);
-
- // Mark-word must be lock_mask now, try to swing it back to unlocked_value.
- movptr(tmp, hdr); // The expected old value
- orptr(tmp, markWord::unlocked_value);
- lock();
- cmpxchgptr(tmp, Address(obj, oopDesc::mark_offset_in_bytes()));
+//
+// x86_32 Note: reg_rax and thread may alias each other due to limited register
+// availiability.
+void MacroAssembler::lightweight_unlock(Register obj, Register reg_rax, Register thread, Register tmp, Label& slow) {
+ assert(reg_rax == rax, "");
+ assert_different_registers(obj, reg_rax, tmp);
+ LP64_ONLY(assert_different_registers(obj, reg_rax, thread, tmp);)
+
+ Label unlocked, push_and_slow;
+ const Register top = tmp;
+
+ // Check if obj is top of lock-stack.
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+ cmpptr(obj, Address(thread, top, Address::times_1, -oopSize));
jcc(Assembler::notEqual, slow);
- // Pop the lock object from the lock-stack.
-#ifdef _LP64
- const Register thread = r15_thread;
-#else
- const Register thread = rax;
- get_thread(thread);
-#endif
+
+ // Pop lock-stack.
+ DEBUG_ONLY(movptr(Address(thread, top, Address::times_1, -oopSize), 0);)
subl(Address(thread, JavaThread::lock_stack_top_offset()), oopSize);
+
+ // Check if recursive.
+ cmpptr(obj, Address(thread, top, Address::times_1, -2 * oopSize));
+ jcc(Assembler::equal, unlocked);
+
+ // Not recursive. Check header for monitor (0b10).
+ movptr(reg_rax, Address(obj, oopDesc::mark_offset_in_bytes()));
+ testptr(reg_rax, markWord::monitor_value);
+ jcc(Assembler::notZero, push_and_slow);
+
+#ifdef ASSERT
+ // Check header not unlocked (0b01).
+ Label not_unlocked;
+ testptr(reg_rax, markWord::unlocked_value);
+ jcc(Assembler::zero, not_unlocked);
+ stop("lightweight_unlock already unlocked");
+ bind(not_unlocked);
+#endif
+
+ // Try to unlock. Transition lock bits 0b00 => 0b01
+ movptr(tmp, reg_rax);
+ orptr(tmp, markWord::unlocked_value);
+ lock(); cmpxchgptr(tmp, Address(obj, oopDesc::mark_offset_in_bytes()));
+ jcc(Assembler::equal, unlocked);
+
+ bind(push_and_slow);
+ // Restore lock-stack and handle the unlock in runtime.
+ if (thread == reg_rax) {
+ // On x86_32 we may lose the thread.
+ get_thread(thread);
+ }
#ifdef ASSERT
- movl(tmp, Address(thread, JavaThread::lock_stack_top_offset()));
- movptr(Address(thread, tmp), 0);
+ movl(top, Address(thread, JavaThread::lock_stack_top_offset()));
+ movptr(Address(thread, top), obj);
#endif
+ addl(Address(thread, JavaThread::lock_stack_top_offset()), oopSize);
+ jmp(slow);
+
+ bind(unlocked);
}
diff --git a/src/hotspot/cpu/x86/macroAssembler_x86.hpp b/src/hotspot/cpu/x86/macroAssembler_x86.hpp
index 4b301684527..4789b63decc 100644
--- a/src/hotspot/cpu/x86/macroAssembler_x86.hpp
+++ b/src/hotspot/cpu/x86/macroAssembler_x86.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -896,6 +896,8 @@ class MacroAssembler: public Assembler {
// Emit the CompiledIC call idiom
void ic_call(address entry, jint method_index = 0);
+ static int ic_check_size();
+ int ic_check(int end_alignment);
void emit_static_call_stub();
@@ -2031,8 +2033,8 @@ class MacroAssembler: public Assembler {
void check_stack_alignment(Register sp, const char* msg, unsigned bias = 0, Register tmp = noreg);
- void lightweight_lock(Register obj, Register hdr, Register thread, Register tmp, Label& slow);
- void lightweight_unlock(Register obj, Register hdr, Register tmp, Label& slow);
+ void lightweight_lock(Register obj, Register reg_rax, Register thread, Register tmp, Label& slow);
+ void lightweight_unlock(Register obj, Register reg_rax, Register thread, Register tmp, Label& slow);
};
/**
diff --git a/src/hotspot/cpu/x86/peephole_x86_64.cpp b/src/hotspot/cpu/x86/peephole_x86_64.cpp
index 8c956aeb053..92a29490eda 100644
--- a/src/hotspot/cpu/x86/peephole_x86_64.cpp
+++ b/src/hotspot/cpu/x86/peephole_x86_64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2022, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -33,8 +33,8 @@
// lea d, [s1 + s2] and
// mov d, s1; shl d, s2 into
// lea d, [s1 << s2] with s2 = 1, 2, 3
-bool lea_coalesce_helper(Block* block, int block_index, PhaseCFG* cfg_, PhaseRegAlloc* ra_,
- MachNode* (*new_root)(), uint inst0_rule, bool imm) {
+static bool lea_coalesce_helper(Block* block, int block_index, PhaseCFG* cfg_, PhaseRegAlloc* ra_,
+ MachNode* (*new_root)(), uint inst0_rule, bool imm) {
MachNode* inst0 = block->get_node(block_index)->as_Mach();
assert(inst0->rule() == inst0_rule, "sanity");
@@ -136,7 +136,7 @@ bool lea_coalesce_helper(Block* block, int block_index, PhaseCFG* cfg_, PhaseReg
// This helper func takes a condition and returns the flags that need to be set for the condition
// It uses the same flags as the test instruction, so if the e.g. the overflow bit is required,
// this func returns clears_overflow, as that is what the test instruction does and what the downstream path expects
-juint map_condition_to_required_test_flags(Assembler::Condition condition) {
+static juint map_condition_to_required_test_flags(Assembler::Condition condition) {
switch (condition) {
case Assembler::Condition::zero: // Same value as equal
case Assembler::Condition::notZero: // Same value as notEqual
diff --git a/src/hotspot/cpu/x86/sharedRuntime_x86_32.cpp b/src/hotspot/cpu/x86/sharedRuntime_x86_32.cpp
index 571160523cb..febc1b2c3b1 100644
--- a/src/hotspot/cpu/x86/sharedRuntime_x86_32.cpp
+++ b/src/hotspot/cpu/x86/sharedRuntime_x86_32.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,8 +25,8 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.hpp"
#include "asm/macroAssembler.inline.hpp"
+#include "code/compiledIC.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/oopMap.hpp"
@@ -36,7 +36,6 @@
#include "interpreter/interpreter.hpp"
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "prims/methodHandles.hpp"
#include "runtime/jniHandles.hpp"
@@ -944,25 +943,18 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
address c2i_unverified_entry = __ pc();
Label skip_fixup;
- Register holder = rax;
+ Register data = rax;
Register receiver = rcx;
Register temp = rbx;
{
-
- Label missed;
- __ movptr(temp, Address(receiver, oopDesc::klass_offset_in_bytes()));
- __ cmpptr(temp, Address(holder, CompiledICHolder::holder_klass_offset()));
- __ movptr(rbx, Address(holder, CompiledICHolder::holder_metadata_offset()));
- __ jcc(Assembler::notEqual, missed);
+ __ ic_check(1 /* end_alignment */);
+ __ movptr(rbx, Address(data, CompiledICData::speculated_method_offset()));
// Method might have been compiled since the call site was patched to
// interpreted if that is the case treat it as a miss so we can get
// the call site corrected.
__ cmpptr(Address(rbx, in_bytes(Method::code_offset())), NULL_WORD);
__ jcc(Assembler::equal, skip_fixup);
-
- __ bind(missed);
- __ jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
}
address c2i_entry = __ pc();
@@ -1449,23 +1441,12 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
// as far as the interpreter and the compiler(s) are concerned.
- const Register ic_reg = rax;
const Register receiver = rcx;
- Label hit;
Label exception_pending;
__ verify_oop(receiver);
- __ cmpptr(ic_reg, Address(receiver, oopDesc::klass_offset_in_bytes()));
- __ jcc(Assembler::equal, hit);
-
- __ jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
// verified entry must be aligned for code patching.
- // and the first 5 bytes must be in the same cache line
- // if we align at 8 then we will be sure 5 bytes are in the same line
- __ align(8);
-
- __ bind(hit);
+ __ ic_check(8 /* end_alignment */);
int vep_offset = ((intptr_t)__ pc()) - start;
@@ -1713,8 +1694,6 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
__ jcc(Assembler::notEqual, slow_path_lock);
} else {
assert(LockingMode == LM_LIGHTWEIGHT, "must be");
- // Load object header
- __ movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
__ lightweight_lock(obj_reg, swap_reg, thread, lock_reg, slow_path_lock);
}
__ bind(count_mon);
@@ -1872,9 +1851,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
__ dec_held_monitor_count();
} else {
assert(LockingMode == LM_LIGHTWEIGHT, "must be");
- __ movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
- __ andptr(swap_reg, ~(int32_t)markWord::lock_mask_in_place);
- __ lightweight_unlock(obj_reg, swap_reg, lock_reg, slow_path_unlock);
+ __ lightweight_unlock(obj_reg, swap_reg, thread, lock_reg, slow_path_unlock);
__ dec_held_monitor_count();
}
diff --git a/src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp b/src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
index faa423bcf8e..c666f982d0f 100644
--- a/src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
+++ b/src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -30,7 +30,6 @@
#include "asm/macroAssembler.inline.hpp"
#include "code/compiledIC.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/oopMap.hpp"
@@ -42,7 +41,6 @@
#include "logging/log.hpp"
#include "memory/resourceArea.hpp"
#include "memory/universe.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
#include "prims/methodHandles.hpp"
@@ -1000,20 +998,14 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm
address c2i_unverified_entry = __ pc();
Label skip_fixup;
- Label ok;
- Register holder = rax;
+ Register data = rax;
Register receiver = j_rarg0;
Register temp = rbx;
{
- __ load_klass(temp, receiver, rscratch1);
- __ cmpptr(temp, Address(holder, CompiledICHolder::holder_klass_offset()));
- __ movptr(rbx, Address(holder, CompiledICHolder::holder_metadata_offset()));
- __ jcc(Assembler::equal, ok);
- __ jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- __ bind(ok);
+ __ ic_check(1 /* end_alignment */);
+ __ movptr(rbx, Address(data, CompiledICData::speculated_method_offset()));
// Method might have been compiled since the call site was patched to
// interpreted if that is the case treat it as a miss so we can get
// the call site corrected.
@@ -1450,7 +1442,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
__ align(BytesPerWord, __ offset() + NativeCall::displacement_offset);
// Emit stub for static call
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, __ pc());
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, __ pc());
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1487,7 +1479,7 @@ static void gen_continuation_enter(MacroAssembler* masm,
// Emit stub for static call
CodeBuffer* cbuf = masm->code_section()->outer();
- address stub = CompiledStaticCall::emit_to_interp_stub(*cbuf, __ pc());
+ address stub = CompiledDirectCall::emit_to_interp_stub(*cbuf, __ pc());
if (stub == nullptr) {
fatal("CodeCache is full at gen_continuation_enter");
}
@@ -1744,6 +1736,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
in_ByteSize(-1),
oop_maps,
exception_offset);
+ if (nm == nullptr) return nm;
if (method->is_continuation_enter_intrinsic()) {
ContinuationEntry::set_enter_code(nm, interpreted_entry_offset);
} else if (method->is_continuation_yield_intrinsic()) {
@@ -1882,25 +1875,13 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
// restoring them except rbp. rbp is the only callee save register
// as far as the interpreter and the compiler(s) are concerned.
-
- const Register ic_reg = rax;
const Register receiver = j_rarg0;
- Label hit;
Label exception_pending;
- assert_different_registers(ic_reg, receiver, rscratch1, rscratch2);
+ assert_different_registers(receiver, rscratch1, rscratch2);
__ verify_oop(receiver);
- __ load_klass(rscratch1, receiver, rscratch2);
- __ cmpq(ic_reg, rscratch1);
- __ jcc(Assembler::equal, hit);
-
- __ jump(RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- // Verified entry point must be aligned
- __ align(8);
-
- __ bind(hit);
+ __ ic_check(8 /* end_alignment */);
int vep_offset = ((intptr_t)__ pc()) - start;
@@ -2189,8 +2170,6 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
__ jcc(Assembler::notEqual, slow_path_lock);
} else {
assert(LockingMode == LM_LIGHTWEIGHT, "must be");
- // Load object header
- __ movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
__ lightweight_lock(obj_reg, swap_reg, r15_thread, rscratch1, slow_path_lock);
}
__ bind(count_mon);
@@ -2333,9 +2312,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm,
__ dec_held_monitor_count();
} else {
assert(LockingMode == LM_LIGHTWEIGHT, "must be");
- __ movptr(swap_reg, Address(obj_reg, oopDesc::mark_offset_in_bytes()));
- __ andptr(swap_reg, ~(int32_t)markWord::lock_mask_in_place);
- __ lightweight_unlock(obj_reg, swap_reg, lock_reg, slow_path_unlock);
+ __ lightweight_unlock(obj_reg, swap_reg, r15_thread, lock_reg, slow_path_unlock);
__ dec_held_monitor_count();
}
diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64.cpp b/src/hotspot/cpu/x86/stubGenerator_x86_64.cpp
index cad9e6475c6..71aafdc1cd3 100644
--- a/src/hotspot/cpu/x86/stubGenerator_x86_64.cpp
+++ b/src/hotspot/cpu/x86/stubGenerator_x86_64.cpp
@@ -951,6 +951,92 @@ address StubGenerator::generate_fp_mask(const char *stub_name, int64_t mask) {
return start;
}
+address StubGenerator::generate_compress_perm_table(const char *stub_name, int32_t esize) {
+ __ align(CodeEntryAlignment);
+ StubCodeMark mark(this, "StubRoutines", stub_name);
+ address start = __ pc();
+ if (esize == 32) {
+ // Loop to generate 256 x 8 int compression permute index table. A row is
+ // accessed using 8 bit index computed using vector mask. An entry in
+ // a row holds either a valid permute index corresponding to set bit position
+ // or a -1 (default) value.
+ for (int mask = 0; mask < 256; mask++) {
+ int ctr = 0;
+ for (int j = 0; j < 8; j++) {
+ if (mask & (1 << j)) {
+ __ emit_data(j, relocInfo::none);
+ ctr++;
+ }
+ }
+ for (; ctr < 8; ctr++) {
+ __ emit_data(-1, relocInfo::none);
+ }
+ }
+ } else {
+ assert(esize == 64, "");
+ // Loop to generate 16 x 4 long compression permute index table. A row is
+ // accessed using 4 bit index computed using vector mask. An entry in
+ // a row holds either a valid permute index pair for a quadword corresponding
+ // to set bit position or a -1 (default) value.
+ for (int mask = 0; mask < 16; mask++) {
+ int ctr = 0;
+ for (int j = 0; j < 4; j++) {
+ if (mask & (1 << j)) {
+ __ emit_data(2 * j, relocInfo::none);
+ __ emit_data(2 * j + 1, relocInfo::none);
+ ctr++;
+ }
+ }
+ for (; ctr < 4; ctr++) {
+ __ emit_data64(-1L, relocInfo::none);
+ }
+ }
+ }
+ return start;
+}
+
+address StubGenerator::generate_expand_perm_table(const char *stub_name, int32_t esize) {
+ __ align(CodeEntryAlignment);
+ StubCodeMark mark(this, "StubRoutines", stub_name);
+ address start = __ pc();
+ if (esize == 32) {
+ // Loop to generate 256 x 8 int expand permute index table. A row is accessed
+ // using 8 bit index computed using vector mask. An entry in a row holds either
+ // a valid permute index (starting from least significant lane) placed at poisition
+ // corresponding to set bit position or a -1 (default) value.
+ for (int mask = 0; mask < 256; mask++) {
+ int ctr = 0;
+ for (int j = 0; j < 8; j++) {
+ if (mask & (1 << j)) {
+ __ emit_data(ctr++, relocInfo::none);
+ } else {
+ __ emit_data(-1, relocInfo::none);
+ }
+ }
+ }
+ } else {
+ assert(esize == 64, "");
+ // Loop to generate 16 x 4 long expand permute index table. A row is accessed
+ // using 4 bit index computed using vector mask. An entry in a row holds either
+ // a valid doubleword permute index pair representing a quadword index (starting
+ // from least significant lane) placed at poisition corresponding to set bit
+ // position or a -1 (default) value.
+ for (int mask = 0; mask < 16; mask++) {
+ int ctr = 0;
+ for (int j = 0; j < 4; j++) {
+ if (mask & (1 << j)) {
+ __ emit_data(2 * ctr, relocInfo::none);
+ __ emit_data(2 * ctr + 1, relocInfo::none);
+ ctr++;
+ } else {
+ __ emit_data64(-1L, relocInfo::none);
+ }
+ }
+ }
+ }
+ return start;
+}
+
address StubGenerator::generate_vector_mask(const char *stub_name, int64_t mask) {
__ align(CodeEntryAlignment);
StubCodeMark mark(this, "StubRoutines", stub_name);
@@ -4095,6 +4181,13 @@ void StubGenerator::generate_compiler_stubs() {
StubRoutines::x86::_vector_reverse_byte_perm_mask_int = generate_vector_reverse_byte_perm_mask_int("perm_mask_int");
StubRoutines::x86::_vector_reverse_byte_perm_mask_short = generate_vector_reverse_byte_perm_mask_short("perm_mask_short");
+ if (VM_Version::supports_avx2() && !VM_Version::supports_avx512vl()) {
+ StubRoutines::x86::_compress_perm_table32 = generate_compress_perm_table("compress_perm_table32", 32);
+ StubRoutines::x86::_compress_perm_table64 = generate_compress_perm_table("compress_perm_table64", 64);
+ StubRoutines::x86::_expand_perm_table32 = generate_expand_perm_table("expand_perm_table32", 32);
+ StubRoutines::x86::_expand_perm_table64 = generate_expand_perm_table("expand_perm_table64", 64);
+ }
+
if (VM_Version::supports_avx2() && !VM_Version::supports_avx512_vpopcntdq()) {
// lut implementation influenced by counting 1s algorithm from section 5-1 of Hackers' Delight.
StubRoutines::x86::_vector_popcount_lut = generate_popcount_avx_lut("popcount_lut");
diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp b/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp
index 6b7da718498..db43085d37f 100644
--- a/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp
+++ b/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -99,6 +99,10 @@ class StubGenerator: public StubCodeGenerator {
address generate_fp_mask(const char *stub_name, int64_t mask);
+ address generate_compress_perm_table(const char *stub_name, int32_t esize);
+
+ address generate_expand_perm_table(const char *stub_name, int32_t esize);
+
address generate_vector_mask(const char *stub_name, int64_t mask);
address generate_vector_byte_perm_mask(const char *stub_name);
diff --git a/src/hotspot/cpu/x86/stubRoutines_x86.cpp b/src/hotspot/cpu/x86/stubRoutines_x86.cpp
index cebf661ae75..bc1cbdbba26 100644
--- a/src/hotspot/cpu/x86/stubRoutines_x86.cpp
+++ b/src/hotspot/cpu/x86/stubRoutines_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2013, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2013, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -82,6 +82,10 @@ address StubRoutines::x86::_join_0_1_base64 = nullptr;
address StubRoutines::x86::_join_1_2_base64 = nullptr;
address StubRoutines::x86::_join_2_3_base64 = nullptr;
address StubRoutines::x86::_decoding_table_base64 = nullptr;
+address StubRoutines::x86::_compress_perm_table32 = nullptr;
+address StubRoutines::x86::_compress_perm_table64 = nullptr;
+address StubRoutines::x86::_expand_perm_table32 = nullptr;
+address StubRoutines::x86::_expand_perm_table64 = nullptr;
#endif
address StubRoutines::x86::_pshuffle_byte_flip_mask_addr = nullptr;
@@ -275,7 +279,7 @@ uint32_t _crc32c_pow_2k_table[TILL_CYCLE]; // because _crc32c_pow_2k_table[TILL_
// A. Kadatch and B. Jenkins / Everything we know about CRC but afraid to forget September 3, 2010 8
// Listing 1: Multiplication of normalized polynomials
// "a" and "b" occupy D least significant bits.
-uint32_t crc32c_multiply(uint32_t a, uint32_t b) {
+static uint32_t crc32c_multiply(uint32_t a, uint32_t b) {
uint32_t product = 0;
uint32_t b_pow_x_table[D + 1]; // b_pow_x_table[k] = (b * x**k) mod P
b_pow_x_table[0] = b;
@@ -299,7 +303,7 @@ uint32_t crc32c_multiply(uint32_t a, uint32_t b) {
#undef P
// A. Kadatch and B. Jenkins / Everything we know about CRC but afraid to forget September 3, 2010 9
-void crc32c_init_pow_2k(void) {
+static void crc32c_init_pow_2k(void) {
// _crc32c_pow_2k_table(0) =
// x^(2^k) mod P(x) = x mod P(x) = x
// Since we are operating on a reflected values
@@ -314,7 +318,7 @@ void crc32c_init_pow_2k(void) {
}
// x^N mod P(x)
-uint32_t crc32c_f_pow_n(uint32_t n) {
+static uint32_t crc32c_f_pow_n(uint32_t n) {
// result = 1 (polynomial)
uint32_t one, result = 0x80000000, i = 0;
diff --git a/src/hotspot/cpu/x86/stubRoutines_x86.hpp b/src/hotspot/cpu/x86/stubRoutines_x86.hpp
index 6c602324f3e..cfb91c5c083 100644
--- a/src/hotspot/cpu/x86/stubRoutines_x86.hpp
+++ b/src/hotspot/cpu/x86/stubRoutines_x86.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2013, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2013, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -37,7 +37,7 @@ enum platform_dependent_constants {
_continuation_stubs_code_size = 1000 LP64_ONLY(+1000),
// AVX512 intrinsics add more code in 64-bit VM,
// Windows have more code to save/restore registers
- _compiler_stubs_code_size = 20000 LP64_ONLY(+32000) WINDOWS_ONLY(+2000),
+ _compiler_stubs_code_size = 20000 LP64_ONLY(+39000) WINDOWS_ONLY(+2000),
_final_stubs_code_size = 10000 LP64_ONLY(+20000) WINDOWS_ONLY(+2000) ZGC_ONLY(+20000)
};
@@ -58,6 +58,10 @@ class x86 {
static address _float_sign_flip;
static address _double_sign_mask;
static address _double_sign_flip;
+ static address _compress_perm_table32;
+ static address _compress_perm_table64;
+ static address _expand_perm_table32;
+ static address _expand_perm_table64;
public:
@@ -338,6 +342,10 @@ class x86 {
static address base64_decoding_table_addr() { return _decoding_table_base64; }
static address base64_AVX2_decode_tables_addr() { return _avx2_decode_tables_base64; }
static address base64_AVX2_decode_LUT_tables_addr() { return _avx2_decode_lut_tables_base64; }
+ static address compress_perm_table32() { return _compress_perm_table32; }
+ static address compress_perm_table64() { return _compress_perm_table64; }
+ static address expand_perm_table32() { return _expand_perm_table32; }
+ static address expand_perm_table64() { return _expand_perm_table64; }
#endif
static address pshuffle_byte_flip_mask_addr() { return _pshuffle_byte_flip_mask_addr; }
static address arrays_hashcode_powers_of_31() { return (address)_arrays_hashcode_powers_of_31; }
diff --git a/src/hotspot/cpu/x86/stubRoutines_x86_64.cpp b/src/hotspot/cpu/x86/stubRoutines_x86_64.cpp
index 417b32eb4a6..eb6c11d7167 100644
--- a/src/hotspot/cpu/x86/stubRoutines_x86_64.cpp
+++ b/src/hotspot/cpu/x86/stubRoutines_x86_64.cpp
@@ -44,4 +44,3 @@ address StubRoutines::x86::_float_sign_mask = nullptr;
address StubRoutines::x86::_float_sign_flip = nullptr;
address StubRoutines::x86::_double_sign_mask = nullptr;
address StubRoutines::x86::_double_sign_flip = nullptr;
-
diff --git a/src/hotspot/cpu/x86/vm_version_x86.cpp b/src/hotspot/cpu/x86/vm_version_x86.cpp
index df1ea6edd30..2412c053106 100644
--- a/src/hotspot/cpu/x86/vm_version_x86.cpp
+++ b/src/hotspot/cpu/x86/vm_version_x86.cpp
@@ -809,7 +809,8 @@ void VM_Version::get_processor_features() {
_stepping = cpu_stepping();
if (cpu_family() > 4) { // it supports CPUID
- _features = feature_flags();
+ _features = feature_flags(); // These can be changed by VM settings
+ _cpu_features = _features; // Preserve features
// Logical processors are only available on P4s and above,
// and only if hyperthreading is available.
_logical_processors_per_package = logical_processor_count();
diff --git a/src/hotspot/cpu/x86/vm_version_x86.hpp b/src/hotspot/cpu/x86/vm_version_x86.hpp
index e521a6ee3bc..bf5e052e167 100644
--- a/src/hotspot/cpu/x86/vm_version_x86.hpp
+++ b/src/hotspot/cpu/x86/vm_version_x86.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -640,7 +640,7 @@ class VM_Version : public Abstract_VM_Version {
}
//
- // Feature identification
+ // Feature identification which can be affected by VM settings
//
static bool supports_cpuid() { return _features != 0; }
static bool supports_cmov() { return (_features & CPU_CMOV) != 0; }
@@ -703,6 +703,11 @@ class VM_Version : public Abstract_VM_Version {
static bool supports_cet_ss() { return (_features & CPU_CET_SS) != 0; }
static bool supports_cet_ibt() { return (_features & CPU_CET_IBT) != 0; }
+ //
+ // Feature identification not affected by VM flags
+ //
+ static bool cpu_supports_evex() { return (_cpu_features & CPU_AVX512F) != 0; }
+
// Intel features
static bool is_intel_family_core() { return is_intel() &&
extended_cpu_family() == CPU_FAMILY_INTEL_CORE; }
@@ -765,6 +770,10 @@ class VM_Version : public Abstract_VM_Version {
return true;
}
+ constexpr static bool supports_recursive_lightweight_locking() {
+ return true;
+ }
+
// For AVX CPUs only. f16c support is disabled if UseAVX == 0.
static bool supports_float16() {
return supports_f16c() || supports_avx512vl();
diff --git a/src/hotspot/cpu/x86/vtableStubs_x86_32.cpp b/src/hotspot/cpu/x86/vtableStubs_x86_32.cpp
index 0e78e0274d7..398f2e37eb5 100644
--- a/src/hotspot/cpu/x86/vtableStubs_x86_32.cpp
+++ b/src/hotspot/cpu/x86/vtableStubs_x86_32.cpp
@@ -24,10 +24,10 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_x86.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klassVtable.hpp"
#include "runtime/sharedRuntime.hpp"
@@ -176,21 +176,21 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
#endif /* PRODUCT */
// Entry arguments:
- // rax: CompiledICHolder
+ // rax: CompiledICData
// rcx: Receiver
// Most registers are in use; we'll use rax, rbx, rcx, rdx, rsi, rdi
// (If we need to make rsi, rdi callee-save, do a push/pop here.)
const Register recv_klass_reg = rsi;
- const Register holder_klass_reg = rax; // declaring interface klass (DECC)
+ const Register holder_klass_reg = rax; // declaring interface klass (DEFC)
const Register resolved_klass_reg = rdi; // resolved interface klass (REFC)
const Register temp_reg = rdx;
const Register method = rbx;
- const Register icholder_reg = rax;
+ const Register icdata_reg = rax;
const Register receiver = rcx;
- __ movptr(resolved_klass_reg, Address(icholder_reg, CompiledICHolder::holder_klass_offset()));
- __ movptr(holder_klass_reg, Address(icholder_reg, CompiledICHolder::holder_metadata_offset()));
+ __ movptr(resolved_klass_reg, Address(icdata_reg, CompiledICData::itable_refc_klass_offset()));
+ __ movptr(holder_klass_reg, Address(icdata_reg, CompiledICData::itable_defc_klass_offset()));
Label L_no_such_interface;
diff --git a/src/hotspot/cpu/x86/vtableStubs_x86_64.cpp b/src/hotspot/cpu/x86/vtableStubs_x86_64.cpp
index f162a651183..158d6f9c692 100644
--- a/src/hotspot/cpu/x86/vtableStubs_x86_64.cpp
+++ b/src/hotspot/cpu/x86/vtableStubs_x86_64.cpp
@@ -24,10 +24,10 @@
#include "precompiled.hpp"
#include "asm/macroAssembler.hpp"
+#include "code/compiledIC.hpp"
#include "code/vtableStubs.hpp"
#include "interp_masm_x86.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/klassVtable.hpp"
#include "runtime/sharedRuntime.hpp"
@@ -168,21 +168,21 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) {
#endif // PRODUCT
// Entry arguments:
- // rax: CompiledICHolder
+ // rax: CompiledICData
// j_rarg0: Receiver
// Most registers are in use; we'll use rax, rbx, r10, r11
// (various calling sequences use r[cd]x, r[sd]i, r[89]; stay away from them)
const Register recv_klass_reg = r10;
- const Register holder_klass_reg = rax; // declaring interface klass (DECC)
+ const Register holder_klass_reg = rax; // declaring interface klass (DEFC)
const Register resolved_klass_reg = r14; // resolved interface klass (REFC)
const Register temp_reg = r11;
const Register temp_reg2 = r13;
const Register method = rbx;
- const Register icholder_reg = rax;
+ const Register icdata_reg = rax;
- __ movptr(resolved_klass_reg, Address(icholder_reg, CompiledICHolder::holder_klass_offset()));
- __ movptr(holder_klass_reg, Address(icholder_reg, CompiledICHolder::holder_metadata_offset()));
+ __ movptr(resolved_klass_reg, Address(icdata_reg, CompiledICData::itable_refc_klass_offset()));
+ __ movptr(holder_klass_reg, Address(icdata_reg, CompiledICData::itable_defc_klass_offset()));
Label L_no_such_interface;
diff --git a/src/hotspot/cpu/x86/x86.ad b/src/hotspot/cpu/x86/x86.ad
index caa82aab99c..6df02d280bc 100644
--- a/src/hotspot/cpu/x86/x86.ad
+++ b/src/hotspot/cpu/x86/x86.ad
@@ -1312,7 +1312,7 @@ int HandlerImpl::emit_exception_handler(CodeBuffer& cbuf) {
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_exception_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -1330,7 +1330,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf) {
// That's why we must use the macroassembler to generate a handler.
C2_MacroAssembler _masm(&cbuf);
address base = __ start_a_stub(size_deopt_handler());
- if (base == NULL) {
+ if (base == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return 0; // CodeBuffer::expand failed
}
@@ -1358,7 +1358,7 @@ int HandlerImpl::emit_deopt_handler(CodeBuffer& cbuf) {
return offset;
}
-Assembler::Width widthForType(BasicType bt) {
+static Assembler::Width widthForType(BasicType bt) {
if (bt == T_BYTE) {
return Assembler::B;
} else if (bt == T_SHORT) {
@@ -1425,6 +1425,8 @@ bool Matcher::match_rule_supported(int opcode) {
return false;
}
break;
+ case Op_CompressV:
+ case Op_ExpandV:
case Op_PopCountVL:
if (UseAVX < 2) {
return false;
@@ -1659,12 +1661,6 @@ bool Matcher::match_rule_supported(int opcode) {
return false;
}
break;
- case Op_CompressV:
- case Op_ExpandV:
- if (!VM_Version::supports_avx512vl()) {
- return false;
- }
- break;
case Op_SqrtF:
if (UseSSE < 1) {
return false;
@@ -1703,7 +1699,7 @@ static inline bool is_pop_count_instr_target(BasicType bt) {
(is_non_subword_integral_type(bt) && VM_Version::supports_avx512_vpopcntdq());
}
-bool Matcher::match_rule_supported_superword(int opcode, int vlen, BasicType bt) {
+bool Matcher::match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt) {
return match_rule_supported_vector(opcode, vlen, bt);
}
@@ -1952,13 +1948,12 @@ bool Matcher::match_rule_supported_vector(int opcode, int vlen, BasicType bt) {
if (is_subword_type(bt) && !VM_Version::supports_avx512_vbmi2()) {
return false;
}
- if (size_in_bits < 128 ) {
+ if (!is_LP64 && !VM_Version::supports_avx512vl() && size_in_bits < 512) {
return false;
}
- if (size_in_bits < 512 && !VM_Version::supports_avx512vl()) {
+ if (size_in_bits < 128 ) {
return false;
}
- break;
case Op_VectorLongToMask:
if (UseAVX < 1 || !is_LP64) {
return false;
@@ -2186,7 +2181,7 @@ MachOper* Matcher::pd_specialize_generic_vector_operand(MachOper* generic_opnd,
}
}
ShouldNotReachHere();
- return NULL;
+ return nullptr;
}
bool Matcher::is_reg2reg_move(MachNode* m) {
@@ -2285,7 +2280,7 @@ int Matcher::min_vector_size(const BasicType bt) {
return MIN2(size,max_size);
}
-int Matcher::superword_max_vector_size(const BasicType bt) {
+int Matcher::max_vector_size_auto_vectorization(const BasicType bt) {
// Limit the max vector size for auto vectorization to 256 bits (32 bytes)
// by default on Cascade Lake
if (VM_Version::is_default_intel_cascade_lake()) {
@@ -2355,7 +2350,7 @@ class FusedPatternMatcher {
int _con_op;
static int match_next(Node* n, int next_op, int next_op_idx) {
- if (n->in(1) == NULL || n->in(2) == NULL) {
+ if (n->in(1) == nullptr || n->in(2) == nullptr) {
return -1;
}
@@ -2422,7 +2417,7 @@ class FusedPatternMatcher {
static bool is_bmi_pattern(Node* n, Node* m) {
assert(UseBMI1Instructions, "sanity");
- if (n != NULL && m != NULL) {
+ if (n != nullptr && m != nullptr) {
if (m->Opcode() == Op_LoadI) {
FusedPatternMatcher bmii(n, m, Op_ConI);
return bmii.match(Op_AndI, -1, Op_SubI, 1, 0) ||
@@ -7446,7 +7441,7 @@ instruct vround_reg_evex(vec dst, vec src, rRegP tmp, vec xtmp1, vec xtmp2, kReg
// --------------------------------- VectorMaskCmp --------------------------------------
instruct vcmpFD(legVec dst, legVec src1, legVec src2, immI8 cond) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL &&
+ predicate(n->bottom_type()->isa_vectmask() == nullptr &&
Matcher::vector_length_in_bytes(n->in(1)->in(1)) >= 8 && // src1
Matcher::vector_length_in_bytes(n->in(1)->in(1)) <= 32 && // src1
is_floating_point_type(Matcher::vector_element_basic_type(n->in(1)->in(1)))); // src1 T_FLOAT, T_DOUBLE
@@ -7466,7 +7461,7 @@ instruct vcmpFD(legVec dst, legVec src1, legVec src2, immI8 cond) %{
instruct evcmpFD64(vec dst, vec src1, vec src2, immI8 cond, kReg ktmp) %{
predicate(Matcher::vector_length_in_bytes(n->in(1)->in(1)) == 64 && // src1
- n->bottom_type()->isa_vectmask() == NULL &&
+ n->bottom_type()->isa_vectmask() == nullptr &&
is_floating_point_type(Matcher::vector_element_basic_type(n->in(1)->in(1)))); // src1 T_FLOAT, T_DOUBLE
match(Set dst (VectorMaskCmp (Binary src1 src2) cond));
effect(TEMP ktmp);
@@ -7506,7 +7501,7 @@ instruct evcmpFD(kReg dst, vec src1, vec src2, immI8 cond) %{
%}
instruct vcmp_direct(legVec dst, legVec src1, legVec src2, immI8 cond) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL &&
+ predicate(n->bottom_type()->isa_vectmask() == nullptr &&
!Matcher::is_unsigned_booltest_pred(n->in(2)->get_int()) &&
Matcher::vector_length_in_bytes(n->in(1)->in(1)) >= 4 && // src1
Matcher::vector_length_in_bytes(n->in(1)->in(1)) <= 32 && // src1
@@ -7526,7 +7521,7 @@ instruct vcmp_direct(legVec dst, legVec src1, legVec src2, immI8 cond) %{
%}
instruct vcmp_negate(legVec dst, legVec src1, legVec src2, immI8 cond, legVec xtmp) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL &&
+ predicate(n->bottom_type()->isa_vectmask() == nullptr &&
!Matcher::is_unsigned_booltest_pred(n->in(2)->get_int()) &&
Matcher::vector_length_in_bytes(n->in(1)->in(1)) >= 4 && // src1
Matcher::vector_length_in_bytes(n->in(1)->in(1)) <= 32 && // src1
@@ -7547,7 +7542,7 @@ instruct vcmp_negate(legVec dst, legVec src1, legVec src2, immI8 cond, legVec xt
%}
instruct vcmpu(legVec dst, legVec src1, legVec src2, immI8 cond, legVec xtmp) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL &&
+ predicate(n->bottom_type()->isa_vectmask() == nullptr &&
Matcher::is_unsigned_booltest_pred(n->in(2)->get_int()) &&
Matcher::vector_length_in_bytes(n->in(1)->in(1)) >= 4 && // src1
Matcher::vector_length_in_bytes(n->in(1)->in(1)) <= 32 && // src1
@@ -7574,7 +7569,7 @@ instruct vcmpu(legVec dst, legVec src1, legVec src2, immI8 cond, legVec xtmp) %{
%}
instruct vcmp64(vec dst, vec src1, vec src2, immI8 cond, kReg ktmp) %{
- predicate((n->bottom_type()->isa_vectmask() == NULL &&
+ predicate((n->bottom_type()->isa_vectmask() == nullptr &&
Matcher::vector_length_in_bytes(n->in(1)->in(1)) == 64) && // src1
is_integral_type(Matcher::vector_element_basic_type(n->in(1)->in(1)))); // src1
match(Set dst (VectorMaskCmp (Binary src1 src2) cond));
@@ -7790,7 +7785,7 @@ instruct blendvp(vec dst, vec src, vec mask, rxmm0 tmp) %{
instruct vblendvpI(legVec dst, legVec src1, legVec src2, legVec mask) %{
predicate(UseAVX > 0 && !EnableX86ECoreOpts &&
- n->in(2)->bottom_type()->isa_vectmask() == NULL &&
+ n->in(2)->bottom_type()->isa_vectmask() == nullptr &&
Matcher::vector_length_in_bytes(n) <= 32 &&
is_integral_type(Matcher::vector_element_basic_type(n)));
match(Set dst (VectorBlend (Binary src1 src2) mask));
@@ -7804,7 +7799,7 @@ instruct vblendvpI(legVec dst, legVec src1, legVec src2, legVec mask) %{
instruct vblendvpFD(legVec dst, legVec src1, legVec src2, legVec mask) %{
predicate(UseAVX > 0 && !EnableX86ECoreOpts &&
- n->in(2)->bottom_type()->isa_vectmask() == NULL &&
+ n->in(2)->bottom_type()->isa_vectmask() == nullptr &&
Matcher::vector_length_in_bytes(n) <= 32 &&
!is_integral_type(Matcher::vector_element_basic_type(n)));
match(Set dst (VectorBlend (Binary src1 src2) mask));
@@ -7818,7 +7813,7 @@ instruct vblendvpFD(legVec dst, legVec src1, legVec src2, legVec mask) %{
instruct vblendvp(legVec dst, legVec src1, legVec src2, legVec mask, legVec vtmp) %{
predicate(UseAVX > 0 && EnableX86ECoreOpts &&
- n->in(2)->bottom_type()->isa_vectmask() == NULL &&
+ n->in(2)->bottom_type()->isa_vectmask() == nullptr &&
Matcher::vector_length_in_bytes(n) <= 32);
match(Set dst (VectorBlend (Binary src1 src2) mask));
format %{ "vector_blend $dst,$src1,$src2,$mask\t! using $vtmp as TEMP" %}
@@ -7834,7 +7829,7 @@ instruct vblendvp(legVec dst, legVec src1, legVec src2, legVec mask, legVec vtmp
instruct evblendvp64(vec dst, vec src1, vec src2, vec mask, kReg ktmp) %{
predicate(Matcher::vector_length_in_bytes(n) == 64 &&
- n->in(2)->bottom_type()->isa_vectmask() == NULL);
+ n->in(2)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorBlend (Binary src1 src2) mask));
format %{ "vector_blend $dst,$src1,$src2,$mask\t! using k2 as TEMP" %}
effect(TEMP ktmp);
@@ -8051,7 +8046,7 @@ instruct ktest_ge8(rFlagsRegU cr, kReg src1, kReg src2) %{
//------------------------------------- LoadMask --------------------------------------------
instruct loadMask(legVec dst, legVec src) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL && !VM_Version::supports_avx512vlbw());
+ predicate(n->bottom_type()->isa_vectmask() == nullptr && !VM_Version::supports_avx512vlbw());
match(Set dst (VectorLoadMask src));
effect(TEMP dst);
format %{ "vector_loadmask_byte $dst, $src\n\t" %}
@@ -8091,7 +8086,7 @@ instruct loadMask_evex(kReg dst, vec src, vec xtmp) %{
//------------------------------------- StoreMask --------------------------------------------
instruct vstoreMask1B(vec dst, vec src, immI_1 size) %{
- predicate(Matcher::vector_length(n) < 64 && n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(Matcher::vector_length(n) < 64 && n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorStoreMask src size));
format %{ "vector_store_mask $dst, $src \t! elem size is $size byte[s]" %}
ins_encode %{
@@ -8109,7 +8104,7 @@ instruct vstoreMask1B(vec dst, vec src, immI_1 size) %{
%}
instruct vstoreMask2B(vec dst, vec src, vec xtmp, immI_2 size) %{
- predicate(Matcher::vector_length(n) <= 16 && n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(Matcher::vector_length(n) <= 16 && n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorStoreMask src size));
effect(TEMP_DEF dst, TEMP xtmp);
format %{ "vector_store_mask $dst, $src \t! elem size is $size byte[s]" %}
@@ -8132,7 +8127,7 @@ instruct vstoreMask2B(vec dst, vec src, vec xtmp, immI_2 size) %{
%}
instruct vstoreMask4B(vec dst, vec src, vec xtmp, immI_4 size) %{
- predicate(UseAVX <= 2 && Matcher::vector_length(n) <= 8 && n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(UseAVX <= 2 && Matcher::vector_length(n) <= 8 && n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorStoreMask src size));
format %{ "vector_store_mask $dst, $src \t! elem size is $size byte[s]" %}
effect(TEMP_DEF dst, TEMP xtmp);
@@ -8192,7 +8187,7 @@ instruct storeMask8B_avx(vec dst, vec src, immI_8 size, vec vtmp) %{
%}
instruct vstoreMask4B_evex_novectmask(vec dst, vec src, immI_4 size) %{
- predicate(UseAVX > 2 && n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(UseAVX > 2 && n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorStoreMask src size));
format %{ "vector_store_mask $dst, $src \t! elem size is $size byte[s]" %}
ins_encode %{
@@ -8208,7 +8203,7 @@ instruct vstoreMask4B_evex_novectmask(vec dst, vec src, immI_4 size) %{
%}
instruct vstoreMask8B_evex_novectmask(vec dst, vec src, immI_8 size) %{
- predicate(UseAVX > 2 && n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(UseAVX > 2 && n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorStoreMask src size));
format %{ "vector_store_mask $dst, $src \t! elem size is $size byte[s]" %}
ins_encode %{
@@ -9045,7 +9040,7 @@ instruct vmask_tolong_evex(rRegL dst, kReg mask, rFlagsReg cr) %{
%}
instruct vmask_tolong_bool(rRegL dst, vec mask, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskToLong mask));
format %{ "vector_tolong_bool $dst, $mask \t! using $xtmp as TEMP" %}
effect(TEMP_DEF dst, TEMP xtmp, KILL cr);
@@ -9061,7 +9056,7 @@ instruct vmask_tolong_bool(rRegL dst, vec mask, vec xtmp, rFlagsReg cr) %{
%}
instruct vmask_tolong_avx(rRegL dst, vec mask, immI size, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskToLong (VectorStoreMask mask size)));
format %{ "vector_tolong_avx $dst, $mask \t! using $xtmp as TEMP" %}
effect(TEMP_DEF dst, TEMP xtmp, KILL cr);
@@ -9094,7 +9089,7 @@ instruct vmask_truecount_evex(rRegI dst, kReg mask, rRegL tmp, rFlagsReg cr) %{
%}
instruct vmask_truecount_bool(rRegI dst, vec mask, rRegL tmp, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskTrueCount mask));
effect(TEMP_DEF dst, TEMP tmp, TEMP xtmp, KILL cr);
format %{ "vector_truecount_bool $dst, $mask \t! using $tmp, $xtmp as TEMP" %}
@@ -9110,7 +9105,7 @@ instruct vmask_truecount_bool(rRegI dst, vec mask, rRegL tmp, vec xtmp, rFlagsRe
%}
instruct vmask_truecount_avx(rRegI dst, vec mask, immI size, rRegL tmp, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskTrueCount (VectorStoreMask mask size)));
effect(TEMP_DEF dst, TEMP tmp, TEMP xtmp, KILL cr);
format %{ "vector_truecount_avx $dst, $mask \t! using $tmp, $xtmp as TEMP" %}
@@ -9144,7 +9139,7 @@ instruct vmask_first_or_last_true_evex(rRegI dst, kReg mask, rRegL tmp, rFlagsRe
%}
instruct vmask_first_or_last_true_bool(rRegI dst, vec mask, rRegL tmp, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskFirstTrue mask));
match(Set dst (VectorMaskLastTrue mask));
effect(TEMP_DEF dst, TEMP tmp, TEMP xtmp, KILL cr);
@@ -9161,7 +9156,7 @@ instruct vmask_first_or_last_true_bool(rRegI dst, vec mask, rRegL tmp, vec xtmp,
%}
instruct vmask_first_or_last_true_avx(rRegI dst, vec mask, immI size, rRegL tmp, vec xtmp, rFlagsReg cr) %{
- predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == NULL);
+ predicate(n->in(1)->in(1)->bottom_type()->isa_vectmask() == nullptr);
match(Set dst (VectorMaskFirstTrue (VectorStoreMask mask size)));
match(Set dst (VectorMaskLastTrue (VectorStoreMask mask size)));
effect(TEMP_DEF dst, TEMP tmp, TEMP xtmp, KILL cr);
@@ -9178,8 +9173,26 @@ instruct vmask_first_or_last_true_avx(rRegI dst, vec mask, immI size, rRegL tmp,
%}
// --------------------------------- Compress/Expand Operations ---------------------------
+#ifdef _LP64
+instruct vcompress_reg_avx(vec dst, vec src, vec mask, rRegI rtmp, rRegL rscratch, vec perm, vec xtmp, rFlagsReg cr) %{
+ predicate(!VM_Version::supports_avx512vl() && Matcher::vector_length_in_bytes(n) <= 32);
+ match(Set dst (CompressV src mask));
+ match(Set dst (ExpandV src mask));
+ effect(TEMP_DEF dst, TEMP perm, TEMP xtmp, TEMP rtmp, TEMP rscratch, KILL cr);
+ format %{ "vector_compress $dst, $src, $mask \t!using $xtmp, $rtmp, $rscratch and $perm as TEMP" %}
+ ins_encode %{
+ int opcode = this->ideal_Opcode();
+ int vlen_enc = vector_length_encoding(this);
+ BasicType bt = Matcher::vector_element_basic_type(this);
+ __ vector_compress_expand_avx2(opcode, $dst$$XMMRegister, $src$$XMMRegister, $mask$$XMMRegister, $rtmp$$Register,
+ $rscratch$$Register, $perm$$XMMRegister, $xtmp$$XMMRegister, bt, vlen_enc);
+ %}
+ ins_pipe( pipe_slow );
+%}
+#endif
instruct vcompress_expand_reg_evex(vec dst, vec src, kReg mask) %{
+ predicate(VM_Version::supports_avx512vl() || Matcher::vector_length_in_bytes(n) == 64);
match(Set dst (CompressV src mask));
match(Set dst (ExpandV src mask));
format %{ "vector_compress_expand $dst, $src, $mask" %}
@@ -9998,7 +10011,7 @@ instruct mask_not_imm(kReg dst, kReg src, immI_M1 cnt) %{
%}
instruct long_to_maskLE8_avx(vec dst, rRegL src, rRegL rtmp1, rRegL rtmp2, vec xtmp) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL && Matcher::vector_length(n) <= 8);
+ predicate(n->bottom_type()->isa_vectmask() == nullptr && Matcher::vector_length(n) <= 8);
match(Set dst (VectorLongToMask src));
effect(TEMP dst, TEMP rtmp1, TEMP rtmp2, TEMP xtmp);
format %{ "long_to_mask_avx $dst, $src\t! using $rtmp1, $rtmp2, $xtmp as TEMP" %}
@@ -10013,7 +10026,7 @@ instruct long_to_maskLE8_avx(vec dst, rRegL src, rRegL rtmp1, rRegL rtmp2, vec x
instruct long_to_maskGT8_avx(vec dst, rRegL src, rRegL rtmp1, rRegL rtmp2, vec xtmp1, rFlagsReg cr) %{
- predicate(n->bottom_type()->isa_vectmask() == NULL && Matcher::vector_length(n) > 8);
+ predicate(n->bottom_type()->isa_vectmask() == nullptr && Matcher::vector_length(n) > 8);
match(Set dst (VectorLongToMask src));
effect(TEMP dst, TEMP rtmp1, TEMP rtmp2, TEMP xtmp1, KILL cr);
format %{ "long_to_mask_avx $dst, $src\t! using $rtmp1, $rtmp2, $xtmp1, as TEMP" %}
diff --git a/src/hotspot/cpu/x86/x86_32.ad b/src/hotspot/cpu/x86/x86_32.ad
index f2e9c042b24..2fe655a5767 100644
--- a/src/hotspot/cpu/x86/x86_32.ad
+++ b/src/hotspot/cpu/x86/x86_32.ad
@@ -504,7 +504,7 @@ void emit_cmpfp_fixup(MacroAssembler& _masm) {
__ bind(exit);
}
-void emit_cmpfp3(MacroAssembler& _masm, Register dst) {
+static void emit_cmpfp3(MacroAssembler& _masm, Register dst) {
Label done;
__ movl(dst, -1);
__ jcc(Assembler::parity, done);
@@ -614,7 +614,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
int framesize = C->output()->frame_size_in_bytes();
int bangsize = C->output()->bang_size_in_bytes();
- __ verified_entry(framesize, C->output()->need_stack_bang(bangsize)?bangsize:0, C->in_24_bit_fp_mode(), C->stub_function() != NULL);
+ __ verified_entry(framesize, C->output()->need_stack_bang(bangsize)?bangsize:0, C->in_24_bit_fp_mode(), C->stub_function() != nullptr);
C->output()->set_frame_complete(cbuf.insts_size());
@@ -1052,7 +1052,7 @@ uint MachSpillCopyNode::implementation( CodeBuffer *cbuf, PhaseRegAlloc *ra_, bo
if( src_first == dst_first && src_second == dst_second )
return size; // Self copy, no move
- if (bottom_type()->isa_vect() != NULL && bottom_type()->isa_vectmask() == NULL) {
+ if (bottom_type()->isa_vect() != nullptr && bottom_type()->isa_vectmask() == nullptr) {
uint ireg = ideal_reg();
assert((src_first_rc != rc_int && dst_first_rc != rc_int), "sanity");
assert((src_first_rc != rc_float && dst_first_rc != rc_float), "sanity");
@@ -1320,12 +1320,12 @@ uint MachSpillCopyNode::implementation( CodeBuffer *cbuf, PhaseRegAlloc *ra_, bo
#ifndef PRODUCT
void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream* st) const {
- implementation( NULL, ra_, false, st );
+ implementation( nullptr, ra_, false, st );
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation( &cbuf, ra_, false, NULL );
+ implementation( &cbuf, ra_, false, nullptr );
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
@@ -1383,24 +1383,12 @@ void MachUEPNode::format( PhaseRegAlloc *ra_, outputStream* st ) const {
void MachUEPNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
MacroAssembler masm(&cbuf);
-#ifdef ASSERT
- uint insts_size = cbuf.insts_size();
-#endif
- masm.cmpptr(rax, Address(rcx, oopDesc::klass_offset_in_bytes()));
- masm.jump_cc(Assembler::notEqual,
- RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
- /* WARNING these NOPs are critical so that verified entry point is properly
- aligned for patching by NativeJump::patch_verified_entry() */
- int nops_cnt = 2;
- if( !OptoBreakpoint ) // Leave space for int3
- nops_cnt += 1;
- masm.nop(nops_cnt);
-
- assert(cbuf.insts_size() - insts_size == size(ra_), "checking code size of inline cache node");
+ masm.ic_check(CodeEntryAlignment);
}
uint MachUEPNode::size(PhaseRegAlloc *ra_) const {
- return OptoBreakpoint ? 11 : 12;
+ return MachNode::size(ra_); // too many variables; just compute it
+ // the hard way
}
@@ -1725,7 +1713,7 @@ encode %{
MacroAssembler _masm(&cbuf);
__ check_klass_subtype_slow_path(Resi, Reax, Recx, Redi,
- NULL, &miss,
+ nullptr, &miss,
/*set_cond_codes:*/ true);
if ($primary) {
__ xorptr(Redi, Redi);
@@ -1842,8 +1830,8 @@ encode %{
cbuf.shared_stub_to_interp_for(_method, cbuf.insts()->mark_off());
} else {
// Emit stubs for static call.
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf, mark);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf, mark);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -3396,7 +3384,7 @@ operand immP() %{
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immP0() %{
predicate( n->get_ptr() == 0 );
match(ConP);
@@ -13776,7 +13764,7 @@ instruct cmpFastLockRTM(eFlagsReg cr, eRegP object, eBXRegP box, eAXRegI tmp, eD
%}
instruct cmpFastLock(eFlagsReg cr, eRegP object, eBXRegP box, eAXRegI tmp, eRegP scr, eRegP thread) %{
- predicate(!Compile::current()->use_rtm());
+ predicate(LockingMode != LM_LIGHTWEIGHT && !Compile::current()->use_rtm());
match(Set cr (FastLock object box));
effect(TEMP tmp, TEMP scr, USE_KILL box, TEMP thread);
ins_cost(300);
@@ -13790,6 +13778,7 @@ instruct cmpFastLock(eFlagsReg cr, eRegP object, eBXRegP box, eAXRegI tmp, eRegP
%}
instruct cmpFastUnlock(eFlagsReg cr, eRegP object, eAXRegP box, eRegP tmp ) %{
+ predicate(LockingMode != LM_LIGHTWEIGHT);
match(Set cr (FastUnlock object box));
effect(TEMP tmp, USE_KILL box);
ins_cost(300);
@@ -13800,6 +13789,32 @@ instruct cmpFastUnlock(eFlagsReg cr, eRegP object, eAXRegP box, eRegP tmp ) %{
ins_pipe(pipe_slow);
%}
+instruct cmpFastLockLightweight(eFlagsReg cr, eRegP object, eBXRegP box, eAXRegI eax_reg, eRegP tmp, eRegP thread) %{
+ predicate(LockingMode == LM_LIGHTWEIGHT);
+ match(Set cr (FastLock object box));
+ effect(TEMP eax_reg, TEMP tmp, USE_KILL box, TEMP thread);
+ ins_cost(300);
+ format %{ "FASTLOCK $object,$box\t! kills $box,$eax_reg,$tmp" %}
+ ins_encode %{
+ __ get_thread($thread$$Register);
+ __ fast_lock_lightweight($object$$Register, $box$$Register, $eax_reg$$Register, $tmp$$Register, $thread$$Register);
+ %}
+ ins_pipe(pipe_slow);
+%}
+
+instruct cmpFastUnlockLightweight(eFlagsReg cr, eRegP object, eAXRegP eax_reg, eRegP tmp, eRegP thread) %{
+ predicate(LockingMode == LM_LIGHTWEIGHT);
+ match(Set cr (FastUnlock object eax_reg));
+ effect(TEMP tmp, USE_KILL eax_reg, TEMP thread);
+ ins_cost(300);
+ format %{ "FASTUNLOCK $object,$eax_reg\t! kills $eax_reg,$tmp" %}
+ ins_encode %{
+ __ get_thread($thread$$Register);
+ __ fast_unlock_lightweight($object$$Register, $eax_reg$$Register, $tmp$$Register, $thread$$Register);
+ %}
+ ins_pipe(pipe_slow);
+%}
+
instruct mask_all_evexL_LT32(kReg dst, eRegL src) %{
predicate(Matcher::vector_length(n) <= 32);
match(Set dst (MaskAll src));
diff --git a/src/hotspot/cpu/x86/x86_64.ad b/src/hotspot/cpu/x86/x86_64.ad
index 80f281a1bf9..a8136688cde 100644
--- a/src/hotspot/cpu/x86/x86_64.ad
+++ b/src/hotspot/cpu/x86/x86_64.ad
@@ -1,5 +1,5 @@
//
-// Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
// DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
//
// This code is free software; you can redistribute it and/or modify it
@@ -519,7 +519,7 @@ int CallDynamicJavaDirectNode::compute_padding(int current_offset) const
}
// This could be in MacroAssembler but it's fairly C2 specific
-void emit_cmpfp_fixup(MacroAssembler& _masm) {
+static void emit_cmpfp_fixup(MacroAssembler& _masm) {
Label exit;
__ jccb(Assembler::noParity, exit);
__ pushf();
@@ -539,7 +539,7 @@ void emit_cmpfp_fixup(MacroAssembler& _masm) {
__ bind(exit);
}
-void emit_cmpfp3(MacroAssembler& _masm, Register dst) {
+static void emit_cmpfp3(MacroAssembler& _masm, Register dst) {
Label done;
__ movl(dst, -1);
__ jcc(Assembler::parity, done);
@@ -558,10 +558,10 @@ void emit_cmpfp3(MacroAssembler& _masm, Register dst) {
// je #
// |-jz -> a | b # a & b
// | -> a #
-void emit_fp_min_max(MacroAssembler& _masm, XMMRegister dst,
- XMMRegister a, XMMRegister b,
- XMMRegister xmmt, Register rt,
- bool min, bool single) {
+static void emit_fp_min_max(MacroAssembler& _masm, XMMRegister dst,
+ XMMRegister a, XMMRegister b,
+ XMMRegister xmmt, Register rt,
+ bool min, bool single) {
Label nan, zero, below, above, done;
@@ -706,7 +706,7 @@ void MachPrologNode::format(PhaseRegAlloc* ra_, outputStream* st) const {
st->print("# stack alignment check");
#endif
}
- if (C->stub_function() != NULL && BarrierSet::barrier_set()->barrier_set_nmethod() != NULL) {
+ if (C->stub_function() != nullptr && BarrierSet::barrier_set()->barrier_set_nmethod() != nullptr) {
st->print("\n\t");
st->print("cmpl [r15_thread + #disarmed_guard_value_offset], #disarmed_guard_value\t");
st->print("\n\t");
@@ -741,7 +741,7 @@ void MachPrologNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
__ bind(L_skip_barrier);
}
- __ verified_entry(framesize, C->output()->need_stack_bang(bangsize)?bangsize:0, false, C->stub_function() != NULL);
+ __ verified_entry(framesize, C->output()->need_stack_bang(bangsize)?bangsize:0, false, C->stub_function() != nullptr);
C->output()->set_frame_complete(cbuf.insts_size());
@@ -970,7 +970,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer* cbuf,
PhaseRegAlloc* ra_,
bool do_size,
outputStream* st) const {
- assert(cbuf != NULL || st != NULL, "sanity");
+ assert(cbuf != nullptr || st != nullptr, "sanity");
// Get registers to move
OptoReg::Name src_second = ra_->get_reg_second(in(1));
OptoReg::Name src_first = ra_->get_reg_first(in(1));
@@ -989,7 +989,7 @@ uint MachSpillCopyNode::implementation(CodeBuffer* cbuf,
// Self copy, no move
return 0;
}
- if (bottom_type()->isa_vect() != NULL && bottom_type()->isa_vectmask() == NULL) {
+ if (bottom_type()->isa_vect() != nullptr && bottom_type()->isa_vectmask() == nullptr) {
uint ireg = ideal_reg();
assert((src_first_rc != rc_int && dst_first_rc != rc_int), "sanity");
assert((ireg == Op_VecS || ireg == Op_VecD || ireg == Op_VecX || ireg == Op_VecY || ireg == Op_VecZ ), "sanity");
@@ -1428,12 +1428,12 @@ uint MachSpillCopyNode::implementation(CodeBuffer* cbuf,
#ifndef PRODUCT
void MachSpillCopyNode::format(PhaseRegAlloc *ra_, outputStream* st) const {
- implementation(NULL, ra_, false, st);
+ implementation(nullptr, ra_, false, st);
}
#endif
void MachSpillCopyNode::emit(CodeBuffer &cbuf, PhaseRegAlloc *ra_) const {
- implementation(&cbuf, ra_, false, NULL);
+ implementation(&cbuf, ra_, false, nullptr);
}
uint MachSpillCopyNode::size(PhaseRegAlloc *ra_) const {
@@ -1472,40 +1472,19 @@ void MachUEPNode::format(PhaseRegAlloc* ra_, outputStream* st) const
{
if (UseCompressedClassPointers) {
st->print_cr("movl rscratch1, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
- st->print_cr("\tdecode_klass_not_null rscratch1, rscratch1");
- st->print_cr("\tcmpq rax, rscratch1\t # Inline cache check");
+ st->print_cr("\tcmpl rscratch1, [rax + CompiledICData::speculated_klass_offset()]\t # Inline cache check");
} else {
- st->print_cr("\tcmpq rax, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t"
- "# Inline cache check");
+ st->print_cr("movq rscratch1, [j_rarg0 + oopDesc::klass_offset_in_bytes()]\t# compressed klass");
+ st->print_cr("\tcmpq rscratch1, [rax + CompiledICData::speculated_klass_offset()]\t # Inline cache check");
}
st->print_cr("\tjne SharedRuntime::_ic_miss_stub");
- st->print_cr("\tnop\t# nops to align entry point");
}
#endif
void MachUEPNode::emit(CodeBuffer& cbuf, PhaseRegAlloc* ra_) const
{
MacroAssembler masm(&cbuf);
- uint insts_size = cbuf.insts_size();
- if (UseCompressedClassPointers) {
- masm.load_klass(rscratch1, j_rarg0, rscratch2);
- masm.cmpptr(rax, rscratch1);
- } else {
- masm.cmpptr(rax, Address(j_rarg0, oopDesc::klass_offset_in_bytes()));
- }
-
- masm.jump_cc(Assembler::notEqual, RuntimeAddress(SharedRuntime::get_ic_miss_stub()));
-
- /* WARNING these NOPs are critical so that verified entry point is properly
- 4 bytes aligned for patching by NativeJump::patch_verified_entry() */
- int nops_cnt = 4 - ((cbuf.insts_size() - insts_size) & 0x3);
- if (OptoBreakpoint) {
- // Leave space for int3
- nops_cnt -= 1;
- }
- nops_cnt &= 0x3; // Do not add nops if code is aligned.
- if (nops_cnt > 0)
- masm.nop(nops_cnt);
+ masm.ic_check(InteriorEntryAlignment);
}
uint MachUEPNode::size(PhaseRegAlloc* ra_) const
@@ -1784,7 +1763,7 @@ encode %{
MacroAssembler _masm(&cbuf);
__ check_klass_subtype_slow_path(Rrsi, Rrax, Rrcx, Rrdi,
- NULL, &miss,
+ nullptr, &miss,
/*set_cond_codes:*/ true);
if ($primary) {
__ xorptr(Rrdi, Rrdi);
@@ -1840,8 +1819,8 @@ encode %{
cbuf.shared_stub_to_interp_for(_method, call_offset);
} else {
// Emit stubs for static call.
- address stub = CompiledStaticCall::emit_to_interp_stub(cbuf, mark);
- if (stub == NULL) {
+ address stub = CompiledDirectCall::emit_to_interp_stub(cbuf, mark);
+ if (stub == nullptr) {
ciEnv::current()->record_failure("CodeCache is full");
return;
}
@@ -2179,7 +2158,7 @@ operand immP()
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immP0()
%{
predicate(n->get_ptr() == 0);
@@ -2207,7 +2186,7 @@ operand immNKlass() %{
interface(CONST_INTER);
%}
-// NULL Pointer Immediate
+// Null Pointer Immediate
operand immN0() %{
predicate(n->get_narrowcon() == 0);
match(ConN);
@@ -3121,7 +3100,7 @@ operand indPosIndexScaleOffset(any_RegP reg, immL32 off, rRegI idx, immI2 scale)
// Indirect Narrow Oop Plus Offset Operand
// Note: x86 architecture doesn't support "scale * index + offset" without a base
-// we can't free r12 even with CompressedOops::base() == NULL.
+// we can't free r12 even with CompressedOops::base() == nullptr.
operand indCompressedOopOffset(rRegN reg, immL32 off) %{
predicate(UseCompressedOops && (CompressedOops::shift() == Address::times_8));
constraint(ALLOC_IN_RC(ptr_reg));
@@ -4902,7 +4881,7 @@ instruct loadConF(regF dst, immF con) %{
instruct loadConN0(rRegN dst, immN0 src, rFlagsReg cr) %{
match(Set dst src);
effect(KILL cr);
- format %{ "xorq $dst, $src\t# compressed NULL ptr" %}
+ format %{ "xorq $dst, $src\t# compressed null pointer" %}
ins_encode %{
__ xorq($dst$$Register, $dst$$Register);
%}
@@ -4916,7 +4895,7 @@ instruct loadConN(rRegN dst, immN src) %{
format %{ "movl $dst, $src\t# compressed ptr" %}
ins_encode %{
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
__ set_narrow_oop($dst$$Register, (jobject)$src$$constant);
@@ -4932,7 +4911,7 @@ instruct loadConNKlass(rRegN dst, immNKlass src) %{
format %{ "movl $dst, $src\t# compressed klass ptr" %}
ins_encode %{
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
ShouldNotReachHere();
} else {
__ set_narrow_klass($dst$$Register, (Klass*)$src$$constant);
@@ -5158,7 +5137,7 @@ instruct storeP(memory mem, any_RegP src)
instruct storeImmP0(memory mem, immP0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL) && n->as_Store()->barrier_data() == 0);
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr) && n->as_Store()->barrier_data() == 0);
match(Set mem (StoreP mem zero));
ins_cost(125); // XXX
@@ -5169,7 +5148,7 @@ instruct storeImmP0(memory mem, immP0 zero)
ins_pipe(ialu_mem_reg);
%}
-// Store NULL Pointer, mark word, or other simple pointer constant.
+// Store Null Pointer, mark word, or other simple pointer constant.
instruct storeImmP(memory mem, immP31 src)
%{
predicate(n->as_Store()->barrier_data() == 0);
@@ -5210,7 +5189,7 @@ instruct storeNKlass(memory mem, rRegN src)
instruct storeImmN0(memory mem, immN0 zero)
%{
- predicate(CompressedOops::base() == NULL);
+ predicate(CompressedOops::base() == nullptr);
match(Set mem (StoreN mem zero));
ins_cost(125); // XXX
@@ -5229,7 +5208,7 @@ instruct storeImmN(memory mem, immN src)
format %{ "movl $mem, $src\t# compressed ptr" %}
ins_encode %{
address con = (address)$src$$constant;
- if (con == NULL) {
+ if (con == nullptr) {
__ movl($mem$$Address, 0);
} else {
__ set_narrow_oop($mem$$Address, (jobject)$src$$constant);
@@ -5253,7 +5232,7 @@ instruct storeImmNKlass(memory mem, immNKlass src)
// Store Integer Immediate
instruct storeImmI0(memory mem, immI_0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreI mem zero));
ins_cost(125); // XXX
@@ -5279,7 +5258,7 @@ instruct storeImmI(memory mem, immI src)
// Store Long Immediate
instruct storeImmL0(memory mem, immL0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreL mem zero));
ins_cost(125); // XXX
@@ -5305,7 +5284,7 @@ instruct storeImmL(memory mem, immL32 src)
// Store Short/Char Immediate
instruct storeImmC0(memory mem, immI_0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreC mem zero));
ins_cost(125); // XXX
@@ -5332,7 +5311,7 @@ instruct storeImmI16(memory mem, immI16 src)
// Store Byte Immediate
instruct storeImmB0(memory mem, immI_0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreB mem zero));
ins_cost(125); // XXX
@@ -5358,7 +5337,7 @@ instruct storeImmB(memory mem, immI8 src)
// Store CMS card-mark Immediate
instruct storeImmCM0_reg(memory mem, immI_0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreCM mem zero));
ins_cost(125); // XXX
@@ -5397,7 +5376,7 @@ instruct storeF(memory mem, regF src)
// Store immediate Float value (it is faster than store from XMM register)
instruct storeF0(memory mem, immF0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreF mem zero));
ins_cost(25); // XXX
@@ -5436,7 +5415,7 @@ instruct storeD(memory mem, regD src)
// Store immediate double 0.0 (it is faster than store from XMM register)
instruct storeD0_imm(memory mem, immD0 src)
%{
- predicate(!UseCompressedOops || (CompressedOops::base() != NULL));
+ predicate(!UseCompressedOops || (CompressedOops::base() != nullptr));
match(Set mem (StoreD mem src));
ins_cost(50);
@@ -5449,7 +5428,7 @@ instruct storeD0_imm(memory mem, immD0 src)
instruct storeD0(memory mem, immD0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL));
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr));
match(Set mem (StoreD mem zero));
ins_cost(25); // XXX
@@ -11680,7 +11659,7 @@ instruct testP_reg(rFlagsReg cr, rRegP src, immP0 zero)
// any compare to a zero should be eq/neq.
instruct testP_mem(rFlagsReg cr, memory op, immP0 zero)
%{
- predicate((!UseCompressedOops || (CompressedOops::base() != NULL)) &&
+ predicate((!UseCompressedOops || (CompressedOops::base() != nullptr)) &&
n->in(1)->as_Load()->barrier_data() == 0);
match(Set cr (CmpP (LoadP op) zero));
@@ -11694,7 +11673,7 @@ instruct testP_mem(rFlagsReg cr, memory op, immP0 zero)
instruct testP_mem_reg0(rFlagsReg cr, memory mem, immP0 zero)
%{
- predicate(UseCompressedOops && (CompressedOops::base() == NULL) &&
+ predicate(UseCompressedOops && (CompressedOops::base() == nullptr) &&
n->in(1)->as_Load()->barrier_data() == 0);
match(Set cr (CmpP (LoadP mem) zero));
@@ -11777,7 +11756,7 @@ instruct testN_reg(rFlagsReg cr, rRegN src, immN0 zero) %{
instruct testN_mem(rFlagsReg cr, memory mem, immN0 zero)
%{
- predicate(CompressedOops::base() != NULL);
+ predicate(CompressedOops::base() != nullptr);
match(Set cr (CmpN (LoadN mem) zero));
ins_cost(500); // XXX
@@ -11790,7 +11769,7 @@ instruct testN_mem(rFlagsReg cr, memory mem, immN0 zero)
instruct testN_mem_reg0(rFlagsReg cr, memory mem, immN0 zero)
%{
- predicate(CompressedOops::base() == NULL);
+ predicate(CompressedOops::base() == nullptr);
match(Set cr (CmpN (LoadN mem) zero));
format %{ "cmpl R12, $mem\t# compressed ptr (R12_heapbase==0)" %}
@@ -12404,7 +12383,7 @@ instruct cmpFastLockRTM(rFlagsReg cr, rRegP object, rbx_RegP box, rax_RegI tmp,
%}
instruct cmpFastLock(rFlagsReg cr, rRegP object, rbx_RegP box, rax_RegI tmp, rRegP scr) %{
- predicate(!Compile::current()->use_rtm());
+ predicate(LockingMode != LM_LIGHTWEIGHT && !Compile::current()->use_rtm());
match(Set cr (FastLock object box));
effect(TEMP tmp, TEMP scr, USE_KILL box);
ins_cost(300);
@@ -12417,6 +12396,7 @@ instruct cmpFastLock(rFlagsReg cr, rRegP object, rbx_RegP box, rax_RegI tmp, rRe
%}
instruct cmpFastUnlock(rFlagsReg cr, rRegP object, rax_RegP box, rRegP tmp) %{
+ predicate(LockingMode != LM_LIGHTWEIGHT);
match(Set cr (FastUnlock object box));
effect(TEMP tmp, USE_KILL box);
ins_cost(300);
@@ -12427,6 +12407,30 @@ instruct cmpFastUnlock(rFlagsReg cr, rRegP object, rax_RegP box, rRegP tmp) %{
ins_pipe(pipe_slow);
%}
+instruct cmpFastLockLightweight(rFlagsReg cr, rRegP object, rbx_RegP box, rax_RegI rax_reg, rRegP tmp) %{
+ predicate(LockingMode == LM_LIGHTWEIGHT);
+ match(Set cr (FastLock object box));
+ effect(TEMP rax_reg, TEMP tmp, USE_KILL box);
+ ins_cost(300);
+ format %{ "fastlock $object,$box\t! kills $box,$rax_reg,$tmp" %}
+ ins_encode %{
+ __ fast_lock_lightweight($object$$Register, $box$$Register, $rax_reg$$Register, $tmp$$Register, r15_thread);
+ %}
+ ins_pipe(pipe_slow);
+%}
+
+instruct cmpFastUnlockLightweight(rFlagsReg cr, rRegP object, rax_RegP rax_reg, rRegP tmp) %{
+ predicate(LockingMode == LM_LIGHTWEIGHT);
+ match(Set cr (FastUnlock object rax_reg));
+ effect(TEMP tmp, USE_KILL rax_reg);
+ ins_cost(300);
+ format %{ "fastunlock $object,$rax_reg\t! kills $rax_reg,$tmp" %}
+ ins_encode %{
+ __ fast_unlock_lightweight($object$$Register, $rax_reg$$Register, $tmp$$Register, r15_thread);
+ %}
+ ins_pipe(pipe_slow);
+%}
+
// ============================================================================
// Safepoint Instructions
diff --git a/src/hotspot/cpu/zero/compiledIC_zero.cpp b/src/hotspot/cpu/zero/compiledIC_zero.cpp
index b0564643af0..24153aeacc5 100644
--- a/src/hotspot/cpu/zero/compiledIC_zero.cpp
+++ b/src/hotspot/cpu/zero/compiledIC_zero.cpp
@@ -25,7 +25,6 @@
#include "precompiled.hpp"
#include "code/codeCache.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
@@ -43,27 +42,27 @@
// ----------------------------------------------------------------------------
-address CompiledStaticCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
+address CompiledDirectCall::emit_to_interp_stub(CodeBuffer &cbuf, address mark) {
ShouldNotReachHere(); // Only needed for COMPILER2.
return nullptr;
}
-int CompiledStaticCall::to_interp_stub_size() {
+int CompiledDirectCall::to_interp_stub_size() {
ShouldNotReachHere(); // Only needed for COMPILER2.
return 0;
}
// Relocation entries for call stub, compiled java to interpreter.
-int CompiledStaticCall::reloc_to_interp_stub() {
+int CompiledDirectCall::reloc_to_interp_stub() {
ShouldNotReachHere(); // Only needed for COMPILER2.
return 0;
}
-void CompiledDirectStaticCall::set_to_interpreted(const methodHandle& callee, address entry) {
+void CompiledDirectCall::set_to_interpreted(const methodHandle& callee, address entry) {
ShouldNotReachHere(); // Only needed for COMPILER2.
}
-void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
+void CompiledDirectCall::set_stub_to_clean(static_stub_Relocation* static_stub) {
ShouldNotReachHere(); // Only needed for COMPILER2.
}
@@ -71,7 +70,7 @@ void CompiledDirectStaticCall::set_stub_to_clean(static_stub_Relocation* static_
// Non-product mode code.
#ifndef PRODUCT
-void CompiledDirectStaticCall::verify() {
+void CompiledDirectCall::verify() {
ShouldNotReachHere(); // Only needed for COMPILER2.
}
diff --git a/src/hotspot/cpu/zero/icBuffer_zero.cpp b/src/hotspot/cpu/zero/icBuffer_zero.cpp
deleted file mode 100644
index adde916a4c4..00000000000
--- a/src/hotspot/cpu/zero/icBuffer_zero.cpp
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright 2007 Red Hat, Inc.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "asm/assembler.inline.hpp"
-#include "code/icBuffer.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/resourceArea.hpp"
-#include "nativeInst_zero.hpp"
-#include "oops/oop.inline.hpp"
-
-int InlineCacheBuffer::ic_stub_code_size() {
- // NB set this once the functions below are implemented
- return 4;
-}
-
-void InlineCacheBuffer::assemble_ic_buffer_code(address code_begin,
- void* cached_oop,
- address entry_point) {
- // NB ic_stub_code_size() must return the size of the code we generate
- ShouldNotCallThis();
-}
-
-address InlineCacheBuffer::ic_buffer_entry_point(address code_begin) {
- // NB ic_stub_code_size() must return the size of the code we generate
- ShouldNotCallThis();
- return nullptr;
-}
-
-void* InlineCacheBuffer::ic_buffer_cached_value(address code_begin) {
- ShouldNotCallThis();
- return nullptr;
-}
diff --git a/src/hotspot/cpu/zero/sharedRuntime_zero.cpp b/src/hotspot/cpu/zero/sharedRuntime_zero.cpp
index 4244b5817db..986cee68512 100644
--- a/src/hotspot/cpu/zero/sharedRuntime_zero.cpp
+++ b/src/hotspot/cpu/zero/sharedRuntime_zero.cpp
@@ -26,10 +26,8 @@
#include "precompiled.hpp"
#include "asm/assembler.inline.hpp"
#include "code/debugInfoRec.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
-#include "oops/compiledICHolder.hpp"
#include "runtime/interfaceSupport.inline.hpp"
#include "runtime/sharedRuntime.hpp"
#include "runtime/vframeArray.hpp"
diff --git a/src/hotspot/os/aix/attachListener_aix.cpp b/src/hotspot/os/aix/attachListener_aix.cpp
index 2e079c9b3a1..51e2b2b3c0a 100644
--- a/src/hotspot/os/aix/attachListener_aix.cpp
+++ b/src/hotspot/os/aix/attachListener_aix.cpp
@@ -478,14 +478,14 @@ AttachOperation* AttachListener::dequeue() {
void AttachListener::vm_start() {
char fn[UNIX_PATH_MAX];
- struct stat64 st;
+ struct stat st;
int ret;
int n = snprintf(fn, UNIX_PATH_MAX, "%s/.java_pid%d",
os::get_temp_directory(), os::current_process_id());
assert(n < (int)UNIX_PATH_MAX, "java_pid file name buffer overflow");
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == 0) {
ret = ::unlink(fn);
if (ret == -1) {
@@ -505,8 +505,8 @@ int AttachListener::pd_init() {
bool AttachListener::check_socket_file() {
int ret;
- struct stat64 st;
- ret = stat64(AixAttachListener::path(), &st);
+ struct stat st;
+ ret = stat(AixAttachListener::path(), &st);
if (ret == -1) { // need to restart attach listener.
log_debug(attach)("Socket file %s does not exist - Restart Attach Listener",
AixAttachListener::path());
@@ -545,14 +545,14 @@ bool AttachListener::is_init_trigger() {
}
char fn[PATH_MAX + 1];
int ret;
- struct stat64 st;
+ struct stat st;
os::snprintf_checked(fn, sizeof(fn), ".attach_pid%d", os::current_process_id());
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == -1) {
log_trace(attach)("Failed to find attach file: %s, trying alternate", fn);
snprintf(fn, sizeof(fn), "%s/.attach_pid%d",
os::get_temp_directory(), os::current_process_id());
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == -1) {
log_debug(attach)("Failed to find attach file: %s", fn);
}
diff --git a/src/hotspot/os/aix/globals_aix.hpp b/src/hotspot/os/aix/globals_aix.hpp
index a047e79b695..fb353348a53 100644
--- a/src/hotspot/os/aix/globals_aix.hpp
+++ b/src/hotspot/os/aix/globals_aix.hpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2005, 2020, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2018 SAP SE. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -49,11 +49,12 @@
"Allow VM to run with EXTSHM=ON.") \
\
/* Maximum expected size of the data segment. That correlates with the */ \
- /* to the maximum C Heap consumption we expect. */ \
- /* We need to know this because we need to leave "breathing space" for the */ \
- /* data segment when placing the java heap. If that space is too small, we */ \
- /* reduce our chance of getting a low heap address (needed for compressed */ \
- /* Oops). */ \
+ /* maximum C Heap consumption we expect. */ \
+ /* We need to leave "breathing space" for the data segment when */ \
+ /* placing the java heap. If the MaxExpectedDataSegmentSize setting */ \
+ /* is too small, we might run into resource issues creating many native */ \
+ /* threads, if it is too large, we reduce our chance of getting a low heap */ \
+ /* address (needed for compressed Oops). */ \
product(uintx, MaxExpectedDataSegmentSize, 8*G, \
"Maximum expected Data Segment Size.") \
\
diff --git a/src/hotspot/os/aix/libperfstat_aix.cpp b/src/hotspot/os/aix/libperfstat_aix.cpp
index f547b4c78e7..0185d7d041c 100644
--- a/src/hotspot/os/aix/libperfstat_aix.cpp
+++ b/src/hotspot/os/aix/libperfstat_aix.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2022, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2018 SAP SE. All rights reserved.
+ * Copyright (c) 2022, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024 SAP SE. All rights reserved.
* Copyright (c) 2022, IBM Corp.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -213,12 +213,8 @@ bool libperfstat::get_cpuinfo(cpuinfo_t* pci) {
if (-1 == libperfstat::perfstat_cpu_total(nullptr, &psct, sizeof(PERFSTAT_CPU_TOTAL_T_LATEST), 1)) {
if (-1 == libperfstat::perfstat_cpu_total(nullptr, &psct, sizeof(perfstat_cpu_total_t_71), 1)) {
- if (-1 == libperfstat::perfstat_cpu_total(nullptr, &psct, sizeof(perfstat_cpu_total_t_61), 1)) {
- if (-1 == libperfstat::perfstat_cpu_total(nullptr, &psct, sizeof(perfstat_cpu_total_t_53), 1)) {
- trcVerbose("perfstat_cpu_total() failed (errno=%d)", errno);
- return false;
- }
- }
+ trcVerbose("perfstat_cpu_total() failed (errno=%d)", errno);
+ return false;
}
}
@@ -252,14 +248,8 @@ bool libperfstat::get_partitioninfo(partitioninfo_t* ppi) {
if (-1 == libperfstat::perfstat_partition_total(nullptr, &pspt, sizeof(PERFSTAT_PARTITON_TOTAL_T_LATEST), 1)) {
if (-1 == libperfstat::perfstat_partition_total(nullptr, &pspt, sizeof(perfstat_partition_total_t_71), 1)) {
ame_details = false;
- if (-1 == libperfstat::perfstat_partition_total(nullptr, &pspt, sizeof(perfstat_partition_total_t_61), 1)) {
- if (-1 == libperfstat::perfstat_partition_total(nullptr, &pspt, sizeof(perfstat_partition_total_t_53), 1)) {
- if (-1 == libperfstat::perfstat_partition_total(nullptr, &pspt, sizeof(perfstat_partition_total_t_53_5), 1)) {
- trcVerbose("perfstat_partition_total() failed (errno=%d)", errno);
- return false;
- }
- }
- }
+ trcVerbose("perfstat_partition_total() failed (errno=%d)", errno);
+ return false;
}
}
@@ -324,10 +314,8 @@ bool libperfstat::get_wparinfo(wparinfo_t* pwi) {
memset (&pswt, '\0', sizeof(pswt));
if (-1 == libperfstat::perfstat_wpar_total(nullptr, &pswt, sizeof(PERFSTAT_WPAR_TOTAL_T_LATEST), 1)) {
- if (-1 == libperfstat::perfstat_wpar_total(nullptr, &pswt, sizeof(perfstat_wpar_total_t_61), 1)) {
- trcVerbose("perfstat_wpar_total() failed (errno=%d)", errno);
- return false;
- }
+ trcVerbose("perfstat_wpar_total() failed (errno=%d)", errno);
+ return false;
}
// WPAR type info.
diff --git a/src/hotspot/os/aix/libperfstat_aix.hpp b/src/hotspot/os/aix/libperfstat_aix.hpp
index 704c5b41a31..67cae0d08b6 100644
--- a/src/hotspot/os/aix/libperfstat_aix.hpp
+++ b/src/hotspot/os/aix/libperfstat_aix.hpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2022, 2022, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2012, 2018 SAP SE. All rights reserved.
+ * Copyright (c) 2022, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024 SAP SE. All rights reserved.
* Copyright (c) 2022, IBM Corp.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -38,16 +38,8 @@
#include
///////////////////////////////////////////////////////////////////////////////////////////////
-// These are excerpts from the AIX 5.3, 6.1, 7.1 libperfstat.h -
+// These are excerpts from the AIX 7.1 libperfstat.h -
// this is all we need from libperfstat.h and I want to avoid having to include
-//
-// Note: I define all structures as if I were to include libperfstat.h on an AIX 5.2
-// build machine.
-//
-// The ratio behind that is that if I would build on an AIX 5.2 build machine,
-// include libperfstat.h and hard-link against libperfstat.a, the program should
-// work without recompilation on all newer AIX versions.
-//
#define IDENTIFIER_LENGTH 64 /* length of strings included in the structures */
#define FIRST_CPU "" /* pseudo-name for fist CPU */
@@ -94,169 +86,6 @@ typedef struct { /* Virtual memory utilization */
} perfstat_memory_total_t;
-typedef struct { /* global cpu information AIX 5.3 < TL10 */
- int ncpus; /* number of active logical processors */
- int ncpus_cfg; /* number of configured processors */
- char description[IDENTIFIER_LENGTH]; /* processor description (type/official name) */
- u_longlong_t processorHZ; /* processor speed in Hz */
- u_longlong_t user; /* raw total number of clock ticks spent in user mode */
- u_longlong_t sys; /* raw total number of clock ticks spent in system mode */
- u_longlong_t idle; /* raw total number of clock ticks spent idle */
- u_longlong_t wait; /* raw total number of clock ticks spent waiting for I/O */
- u_longlong_t pswitch; /* number of process switches (change in currently running process) */
- u_longlong_t syscall; /* number of system calls executed */
- u_longlong_t sysread; /* number of read system calls executed */
- u_longlong_t syswrite; /* number of write system calls executed */
- u_longlong_t sysfork; /* number of forks system calls executed */
- u_longlong_t sysexec; /* number of execs system calls executed */
- u_longlong_t readch; /* number of characters transferred with read system call */
- u_longlong_t writech; /* number of characters transferred with write system call */
- u_longlong_t devintrs; /* number of device interrupts */
- u_longlong_t softintrs; /* number of software interrupts */
- time_t lbolt; /* number of ticks since last reboot */
- u_longlong_t loadavg[3]; /* (1<. */
- u_longlong_t runque; /* length of the run queue (processes ready) */
- u_longlong_t swpque; /* ength of the swap queue (processes waiting to be paged in) */
- u_longlong_t bread; /* number of blocks read */
- u_longlong_t bwrite; /* number of blocks written */
- u_longlong_t lread; /* number of logical read requests */
- u_longlong_t lwrite; /* number of logical write requests */
- u_longlong_t phread; /* number of physical reads (reads on raw devices) */
- u_longlong_t phwrite; /* number of physical writes (writes on raw devices) */
- u_longlong_t runocc; /* updated whenever runque is updated, i.e. the runqueue is occupied.
- * This can be used to compute the simple average of ready processes */
- u_longlong_t swpocc; /* updated whenever swpque is updated. i.e. the swpqueue is occupied.
- * This can be used to compute the simple average processes waiting to be paged in */
- u_longlong_t iget; /* number of inode lookups */
- u_longlong_t namei; /* number of vnode lookup from a path name */
- u_longlong_t dirblk; /* number of 512-byte block reads by the directory search routine to locate an entry for a file */
- u_longlong_t msg; /* number of IPC message operations */
- u_longlong_t sema; /* number of IPC semaphore operations */
- u_longlong_t rcvint; /* number of tty receive interrupts */
- u_longlong_t xmtint; /* number of tyy transmit interrupts */
- u_longlong_t mdmint; /* number of modem interrupts */
- u_longlong_t tty_rawinch; /* number of raw input characters */
- u_longlong_t tty_caninch; /* number of canonical input characters (always zero) */
- u_longlong_t tty_rawoutch; /* number of raw output characters */
- u_longlong_t ksched; /* number of kernel processes created */
- u_longlong_t koverf; /* kernel process creation attempts where:
- * -the user has forked to their maximum limit
- * -the configuration limit of processes has been reached */
- u_longlong_t kexit; /* number of kernel processes that became zombies */
- u_longlong_t rbread; /* number of remote read requests */
- u_longlong_t rcread; /* number of cached remote reads */
- u_longlong_t rbwrt; /* number of remote writes */
- u_longlong_t rcwrt; /* number of cached remote writes */
- u_longlong_t traps; /* number of traps */
- int ncpus_high; /* index of highest processor online */
- u_longlong_t puser; /* raw number of physical processor tics in user mode */
- u_longlong_t psys; /* raw number of physical processor tics in system mode */
- u_longlong_t pidle; /* raw number of physical processor tics idle */
- u_longlong_t pwait; /* raw number of physical processor tics waiting for I/O */
- u_longlong_t decrintrs; /* number of decrementer tics interrupts */
- u_longlong_t mpcrintrs; /* number of mpc's received interrupts */
- u_longlong_t mpcsintrs; /* number of mpc's sent interrupts */
- u_longlong_t phantintrs; /* number of phantom interrupts */
- u_longlong_t idle_donated_purr; /* number of idle cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_donated_spurr; /* number of idle spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_purr; /* number of busy cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_spurr; /* number of busy spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_stolen_purr; /* number of idle cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t idle_stolen_spurr; /* number of idle spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_purr; /* number of busy cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_spurr; /* number of busy spurr cycles stolen by the hypervisor from a dedicated partition */
- short iowait; /* number of processes that are asleep waiting for buffered I/O */
- short physio; /* number of processes waiting for raw I/O */
- longlong_t twait; /* number of threads that are waiting for filesystem direct(cio) */
- u_longlong_t hpi; /* number of hypervisor page-ins */
- u_longlong_t hpit; /* Time spent in hypervisor page-ins (in nanoseconds) */
-} perfstat_cpu_total_t_53;
-
-typedef struct { /* global cpu information AIX 6.1|5.3 > TL09 */
- int ncpus; /* number of active logical processors */
- int ncpus_cfg; /* number of configured processors */
- char description[IDENTIFIER_LENGTH]; /* processor description (type/official name) */
- u_longlong_t processorHZ; /* processor speed in Hz */
- u_longlong_t user; /* raw total number of clock ticks spent in user mode */
- u_longlong_t sys; /* raw total number of clock ticks spent in system mode */
- u_longlong_t idle; /* raw total number of clock ticks spent idle */
- u_longlong_t wait; /* raw total number of clock ticks spent waiting for I/O */
- u_longlong_t pswitch; /* number of process switches (change in currently running process) */
- u_longlong_t syscall; /* number of system calls executed */
- u_longlong_t sysread; /* number of read system calls executed */
- u_longlong_t syswrite; /* number of write system calls executed */
- u_longlong_t sysfork; /* number of forks system calls executed */
- u_longlong_t sysexec; /* number of execs system calls executed */
- u_longlong_t readch; /* number of characters transferred with read system call */
- u_longlong_t writech; /* number of characters transferred with write system call */
- u_longlong_t devintrs; /* number of device interrupts */
- u_longlong_t softintrs; /* number of software interrupts */
- time_t lbolt; /* number of ticks since last reboot */
- u_longlong_t loadavg[3]; /* (1<. */
- u_longlong_t runque; /* length of the run queue (processes ready) */
- u_longlong_t swpque; /* length of the swap queue (processes waiting to be paged in) */
- u_longlong_t bread; /* number of blocks read */
- u_longlong_t bwrite; /* number of blocks written */
- u_longlong_t lread; /* number of logical read requests */
- u_longlong_t lwrite; /* number of logical write requests */
- u_longlong_t phread; /* number of physical reads (reads on raw devices) */
- u_longlong_t phwrite; /* number of physical writes (writes on raw devices) */
- u_longlong_t runocc; /* updated whenever runque is updated, i.e. the runqueue is occupied.
- * This can be used to compute the simple average of ready processes */
- u_longlong_t swpocc; /* updated whenever swpque is updated. i.e. the swpqueue is occupied.
- * This can be used to compute the simple average processes waiting to be paged in */
- u_longlong_t iget; /* number of inode lookups */
- u_longlong_t namei; /* number of vnode lookup from a path name */
- u_longlong_t dirblk; /* number of 512-byte block reads by the directory search routine to locate an entry for a file */
- u_longlong_t msg; /* number of IPC message operations */
- u_longlong_t sema; /* number of IPC semaphore operations */
- u_longlong_t rcvint; /* number of tty receive interrupts */
- u_longlong_t xmtint; /* number of tyy transmit interrupts */
- u_longlong_t mdmint; /* number of modem interrupts */
- u_longlong_t tty_rawinch; /* number of raw input characters */
- u_longlong_t tty_caninch; /* number of canonical input characters (always zero) */
- u_longlong_t tty_rawoutch; /* number of raw output characters */
- u_longlong_t ksched; /* number of kernel processes created */
- u_longlong_t koverf; /* kernel process creation attempts where:
- * -the user has forked to their maximum limit
- * -the configuration limit of processes has been reached */
- u_longlong_t kexit; /* number of kernel processes that became zombies */
- u_longlong_t rbread; /* number of remote read requests */
- u_longlong_t rcread; /* number of cached remote reads */
- u_longlong_t rbwrt; /* number of remote writes */
- u_longlong_t rcwrt; /* number of cached remote writes */
- u_longlong_t traps; /* number of traps */
- int ncpus_high; /* index of highest processor online */
- u_longlong_t puser; /* raw number of physical processor tics in user mode */
- u_longlong_t psys; /* raw number of physical processor tics in system mode */
- u_longlong_t pidle; /* raw number of physical processor tics idle */
- u_longlong_t pwait; /* raw number of physical processor tics waiting for I/O */
- u_longlong_t decrintrs; /* number of decrementer tics interrupts */
- u_longlong_t mpcrintrs; /* number of mpc's received interrupts */
- u_longlong_t mpcsintrs; /* number of mpc's sent interrupts */
- u_longlong_t phantintrs; /* number of phantom interrupts */
- u_longlong_t idle_donated_purr; /* number of idle cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_donated_spurr; /* number of idle spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_purr; /* number of busy cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_spurr; /* number of busy spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_stolen_purr; /* number of idle cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t idle_stolen_spurr; /* number of idle spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_purr; /* number of busy cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_spurr; /* number of busy spurr cycles stolen by the hypervisor from a dedicated partition */
- short iowait; /* number of processes that are asleep waiting for buffered I/O */
- short physio; /* number of processes waiting for raw I/O */
- longlong_t twait; /* number of threads that are waiting for filesystem direct(cio) */
- u_longlong_t hpi; /* number of hypervisor page-ins */
- u_longlong_t hpit; /* Time spent in hypervisor page-ins (in nanoseconds) */
- u_longlong_t puser_spurr; /* number of spurr cycles spent in user mode */
- u_longlong_t psys_spurr; /* number of spurr cycles spent in kernel mode */
- u_longlong_t pidle_spurr; /* number of spurr cycles spent in idle mode */
- u_longlong_t pwait_spurr; /* number of spurr cycles spent in wait mode */
- int spurrflag; /* set if running in spurr mode */
-} perfstat_cpu_total_t_61;
-
typedef struct { /* global cpu information AIX 7.1 */
int ncpus; /* number of active logical processors */
int ncpus_cfg; /* number of configured processors */
@@ -564,166 +393,6 @@ typedef union {
} b;
} perfstat_partition_type_t;
-typedef struct { /* partition total information AIX 5.3 < TL6 */
- char name[IDENTIFIER_LENGTH]; /* name of the logical partition */
- perfstat_partition_type_t type; /* set of bits describing the partition */
- int lpar_id; /* logical partition identifier */
- int group_id; /* identifier of the LPAR group this partition is a member of */
- int pool_id; /* identifier of the shared pool of physical processors this partition is a member of */
- int online_cpus; /* number of virtual CPUs currently online on the partition */
- int max_cpus; /* maximum number of virtual CPUs this partition can ever have */
- int min_cpus; /* minimum number of virtual CPUs this partition must have */
- u_longlong_t online_memory; /* amount of memory currently online */
- u_longlong_t max_memory; /* maximum amount of memory this partition can ever have */
- u_longlong_t min_memory; /* minimum amount of memory this partition must have */
- int entitled_proc_capacity; /* number of processor units this partition is entitled to receive */
- int max_proc_capacity; /* maximum number of processor units this partition can ever have */
- int min_proc_capacity; /* minimum number of processor units this partition must have */
- int proc_capacity_increment; /* increment value to the entitled capacity */
- int unalloc_proc_capacity; /* number of processor units currently unallocated in the shared processor pool this partition belongs to */
- int var_proc_capacity_weight; /* partition priority weight to receive extra capacity */
- int unalloc_var_proc_capacity_weight; /* number of variable processor capacity weight units currently unallocated in the shared processor pool this partition belongs to */
- int online_phys_cpus_sys; /* number of physical CPUs currently active in the system containing this partition */
- int max_phys_cpus_sys; /* maximum possible number of physical CPUs in the system containing this partition */
- int phys_cpus_pool; /* number of the physical CPUs currently in the shared processor pool this partition belong to */
- u_longlong_t puser; /* raw number of physical processor tics in user mode */
- u_longlong_t psys; /* raw number of physical processor tics in system mode */
- u_longlong_t pidle; /* raw number of physical processor tics idle */
- u_longlong_t pwait; /* raw number of physical processor tics waiting for I/O */
- u_longlong_t pool_idle_time; /* number of clock tics a processor in the shared pool was idle */
- u_longlong_t phantintrs; /* number of phantom interrupts received by the partition */
- u_longlong_t invol_virt_cswitch; /* number involuntary virtual CPU context switches */
- u_longlong_t vol_virt_cswitch; /* number voluntary virtual CPU context switches */
- u_longlong_t timebase_last; /* most recently cpu time base */
- u_longlong_t reserved_pages; /* Currently number of 16GB pages. Cannot participate in DR operations */
- u_longlong_t reserved_pagesize; /* Currently 16GB pagesize Cannot participate in DR operations */
-} perfstat_partition_total_t_53_5;
-
-typedef struct { /* partition total information AIX 5.3 < TL10 */
- char name[IDENTIFIER_LENGTH]; /* name of the logical partition */
- perfstat_partition_type_t type; /* set of bits describing the partition */
- int lpar_id; /* logical partition identifier */
- int group_id; /* identifier of the LPAR group this partition is a member of */
- int pool_id; /* identifier of the shared pool of physical processors this partition is a member of */
- int online_cpus; /* number of virtual CPUs currently online on the partition */
- int max_cpus; /* maximum number of virtual CPUs this partition can ever have */
- int min_cpus; /* minimum number of virtual CPUs this partition must have */
- u_longlong_t online_memory; /* amount of memory currently online */
- u_longlong_t max_memory; /* maximum amount of memory this partition can ever have */
- u_longlong_t min_memory; /* minimum amount of memory this partition must have */
- int entitled_proc_capacity; /* number of processor units this partition is entitled to receive */
- int max_proc_capacity; /* maximum number of processor units this partition can ever have */
- int min_proc_capacity; /* minimum number of processor units this partition must have */
- int proc_capacity_increment; /* increment value to the entitled capacity */
- int unalloc_proc_capacity; /* number of processor units currently unallocated in the shared processor pool this partition belongs to */
- int var_proc_capacity_weight; /* partition priority weight to receive extra capacity */
- int unalloc_var_proc_capacity_weight; /* number of variable processor capacity weight units currently unallocated in the shared processor pool this partition belongs to */
- int online_phys_cpus_sys; /* number of physical CPUs currently active in the system containing this partition */
- int max_phys_cpus_sys; /* maximum possible number of physical CPUs in the system containing this partition */
- int phys_cpus_pool; /* number of the physical CPUs currently in the shared processor pool this partition belong to */
- u_longlong_t puser; /* raw number of physical processor tics in user mode */
- u_longlong_t psys; /* raw number of physical processor tics in system mode */
- u_longlong_t pidle; /* raw number of physical processor tics idle */
- u_longlong_t pwait; /* raw number of physical processor tics waiting for I/O */
- u_longlong_t pool_idle_time; /* number of clock tics a processor in the shared pool was idle */
- u_longlong_t phantintrs; /* number of phantom interrupts received by the partition */
- u_longlong_t invol_virt_cswitch; /* number involuntary virtual CPU context switches */
- u_longlong_t vol_virt_cswitch; /* number voluntary virtual CPU context switches */
- u_longlong_t timebase_last; /* most recently cpu time base */
- u_longlong_t reserved_pages; /* Currently number of 16GB pages. Cannot participate in DR operations */
- u_longlong_t reserved_pagesize; /* Currently 16GB pagesize Cannot participate in DR operations */
- u_longlong_t idle_donated_purr; /* number of idle cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_donated_spurr; /* number of idle spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_purr; /* number of busy cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_spurr; /* number of busy spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_stolen_purr; /* number of idle cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t idle_stolen_spurr; /* number of idle spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_purr; /* number of busy cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_spurr; /* number of busy spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t shcpus_in_sys; /* Number of physical processors allocated for shared processor use */
- u_longlong_t max_pool_capacity; /* Maximum processor capacity of partitions pool */
- u_longlong_t entitled_pool_capacity; /* Entitled processor capacity of partitions pool */
- u_longlong_t pool_max_time; /* Summation of maximum time that could be consumed by the pool (nano seconds) */
- u_longlong_t pool_busy_time; /* Summation of busy (non-idle) time accumulated across all partitions in the pool (nano seconds) */
- u_longlong_t pool_scaled_busy_time; /* Scaled summation of busy (non-idle) time accumulated across all partitions in the pool (nano seconds) */
- u_longlong_t shcpu_tot_time; /* Summation of total time across all physical processors allocated for shared processor use (nano seconds) */
- u_longlong_t shcpu_busy_time; /* Summation of busy (non-idle) time accumulated across all shared processor partitions (nano seconds) */
- u_longlong_t shcpu_scaled_busy_time; /* Scaled summation of busy time accumulated across all shared processor partitions (nano seconds) */
- int ams_pool_id; /* AMS pool id of the pool the LPAR belongs to */
- int var_mem_weight; /* variable memory capacity weight */
- u_longlong_t iome; /* I/O memory entitlement of the partition in bytes*/
- u_longlong_t pmem; /* Physical memory currently backing the partition's logical memory in bytes*/
- u_longlong_t hpi; /* number of hypervisor page-ins */
- u_longlong_t hpit; /* Time spent in hypervisor page-ins (in nanoseconds)*/
- u_longlong_t hypv_pagesize; /* Hypervisor page size in KB*/
-} perfstat_partition_total_t_53;
-
-typedef struct { /* partition total information AIX 6.1|5.3 > TL09 */
- char name[IDENTIFIER_LENGTH]; /* name of the logical partition */
- perfstat_partition_type_t type; /* set of bits describing the partition */
- int lpar_id; /* logical partition identifier */
- int group_id; /* identifier of the LPAR group this partition is a member of */
- int pool_id; /* identifier of the shared pool of physical processors this partition is a member of */
- int online_cpus; /* number of virtual CPUs currently online on the partition */
- int max_cpus; /* maximum number of virtual CPUs this partition can ever have */
- int min_cpus; /* minimum number of virtual CPUs this partition must have */
- u_longlong_t online_memory; /* amount of memory currently online */
- u_longlong_t max_memory; /* maximum amount of memory this partition can ever have */
- u_longlong_t min_memory; /* minimum amount of memory this partition must have */
- int entitled_proc_capacity; /* number of processor units this partition is entitled to receive */
- int max_proc_capacity; /* maximum number of processor units this partition can ever have */
- int min_proc_capacity; /* minimum number of processor units this partition must have */
- int proc_capacity_increment; /* increment value to the entitled capacity */
- int unalloc_proc_capacity; /* number of processor units currently unallocated in the shared processor pool this partition belongs to */
- int var_proc_capacity_weight; /* partition priority weight to receive extra capacity */
- int unalloc_var_proc_capacity_weight; /* number of variable processor capacity weight units currently unallocated in the shared processor pool this partition belongs to */
- int online_phys_cpus_sys; /* number of physical CPUs currently active in the system containing this partition */
- int max_phys_cpus_sys; /* maximum possible number of physical CPUs in the system containing this partition */
- int phys_cpus_pool; /* number of the physical CPUs currently in the shared processor pool this partition belong to */
- u_longlong_t puser; /* raw number of physical processor tics in user mode */
- u_longlong_t psys; /* raw number of physical processor tics in system mode */
- u_longlong_t pidle; /* raw number of physical processor tics idle */
- u_longlong_t pwait; /* raw number of physical processor tics waiting for I/O */
- u_longlong_t pool_idle_time; /* number of clock tics a processor in the shared pool was idle */
- u_longlong_t phantintrs; /* number of phantom interrupts received by the partition */
- u_longlong_t invol_virt_cswitch; /* number involuntary virtual CPU context switches */
- u_longlong_t vol_virt_cswitch; /* number voluntary virtual CPU context switches */
- u_longlong_t timebase_last; /* most recently cpu time base */
- u_longlong_t reserved_pages; /* Currently number of 16GB pages. Cannot participate in DR operations */
- u_longlong_t reserved_pagesize; /* Currently 16GB pagesize Cannot participate in DR operations */
- u_longlong_t idle_donated_purr; /* number of idle cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_donated_spurr; /* number of idle spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_purr; /* number of busy cycles donated by a dedicated partition enabled for donation */
- u_longlong_t busy_donated_spurr; /* number of busy spurr cycles donated by a dedicated partition enabled for donation */
- u_longlong_t idle_stolen_purr; /* number of idle cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t idle_stolen_spurr; /* number of idle spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_purr; /* number of busy cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t busy_stolen_spurr; /* number of busy spurr cycles stolen by the hypervisor from a dedicated partition */
- u_longlong_t shcpus_in_sys; /* Number of physical processors allocated for shared processor use */
- u_longlong_t max_pool_capacity; /* Maximum processor capacity of partitions pool */
- u_longlong_t entitled_pool_capacity; /* Entitled processor capacity of partitions pool */
- u_longlong_t pool_max_time; /* Summation of maximum time that could be consumed by the pool (nano seconds) */
- u_longlong_t pool_busy_time; /* Summation of busy (non-idle) time accumulated across all partitions in the pool (nano seconds) */
- u_longlong_t pool_scaled_busy_time; /* Scaled summation of busy (non-idle) time accumulated across all partitions in the pool (nano seconds) */
- u_longlong_t shcpu_tot_time; /* Summation of total time across all physical processors allocated for shared processor use (nano seconds) */
- u_longlong_t shcpu_busy_time; /* Summation of busy (non-idle) time accumulated across all shared processor partitions (nano seconds) */
- u_longlong_t shcpu_scaled_busy_time; /* Scaled summation of busy time accumulated across all shared processor partitions (nano seconds) */
- int ams_pool_id; /* AMS pool id of the pool the LPAR belongs to */
- int var_mem_weight; /* variable memory capacity weight */
- u_longlong_t iome; /* I/O memory entitlement of the partition in bytes*/
- u_longlong_t pmem; /* Physical memory currently backing the partition's logical memory in bytes*/
- u_longlong_t hpi; /* number of hypervisor page-ins */
- u_longlong_t hpit; /* Time spent in hypervisor page-ins (in nanoseconds)*/
- u_longlong_t hypv_pagesize; /* Hypervisor page size in KB*/
- uint online_lcpus; /* number of online logical cpus */
- uint smt_thrds; /* number of hardware threads that are running */
- u_longlong_t puser_spurr; /* number of spurr cycles spent in user mode */
- u_longlong_t psys_spurr; /* number of spurr cycles spent in kernel mode */
- u_longlong_t pidle_spurr; /* number of spurr cycles spent in idle mode */
- u_longlong_t pwait_spurr; /* number of spurr cycles spent in wait mode */
- int spurrflag; /* set if running in spurr mode */
-} perfstat_partition_total_t_61;
-
typedef struct { /* partition total information AIX 7.1 */
char name[IDENTIFIER_LENGTH]; /* name of the logical partition */
perfstat_partition_type_t type; /* set of bits describing the partition */
@@ -941,17 +610,6 @@ typedef union { /* WPAR Type & Flags */
} b;
} perfstat_wpar_type_t;
-typedef struct { /* Workload partition Information AIX 5.3 & 6.1*/
- char name[MAXCORRALNAMELEN+1]; /* name of the Workload Partition */
- perfstat_wpar_type_t type; /* set of bits describing the wpar */
- cid_t wpar_id; /* workload partition identifier */
- uint online_cpus; /* Number of Virtual CPUs in partition rset or number of virtual CPUs currently online on the Global partition*/
- int cpu_limit; /* CPU limit in 100ths of % - 1..10000 */
- int mem_limit; /* Memory limit in 100ths of % - 1..10000 */
- u_longlong_t online_memory; /* amount of memory currently online in Global Partition */
- int entitled_proc_capacity; /* number of processor units this partition is entitled to receive */
-} perfstat_wpar_total_t_61;
-
typedef struct { /* Workload partition Information AIX 7.1*/
char name[MAXCORRALNAMELEN+1]; /* name of the Workload Partition */
perfstat_wpar_type_t type; /* set of bits describing the wpar */
@@ -983,7 +641,7 @@ typedef struct { /* WPAR identifier */
-// end: libperfstat.h (AIX 5.2, 5.3, 6.1, 7.1)
+// end: libperfstat.h (AIX 7.1)
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
#define PERFSTAT_PARTITON_TOTAL_T_LATEST perfstat_partition_total_t_71_1 /* latest perfstat_partition_total_t structure */
diff --git a/src/hotspot/os/aix/os_aix.cpp b/src/hotspot/os/aix/os_aix.cpp
index adaab8914a7..5d5ea364b66 100644
--- a/src/hotspot/os/aix/os_aix.cpp
+++ b/src/hotspot/os/aix/os_aix.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2012, 2023 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -29,7 +29,6 @@
// no precompiled headers
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/compileBroker.hpp"
#include "interpreter/interpreter.hpp"
@@ -117,6 +116,10 @@
#include
#include
+#ifndef _LARGE_FILES
+#error Hotspot on AIX must be compiled with -D_LARGE_FILES
+#endif
+
// Missing prototypes for various system APIs.
extern "C"
int mread_real_time(timebasestruct_t *t, size_t size_of_timebasestruct_t);
@@ -273,6 +276,22 @@ julong os::Aix::available_memory() {
}
}
+jlong os::total_swap_space() {
+ perfstat_memory_total_t memory_info;
+ if (libperfstat::perfstat_memory_total(NULL, &memory_info, sizeof(perfstat_memory_total_t), 1) == -1) {
+ return -1;
+ }
+ return (jlong)(memory_info.pgsp_total * 4 * K);
+}
+
+jlong os::free_swap_space() {
+ perfstat_memory_total_t memory_info;
+ if (libperfstat::perfstat_memory_total(NULL, &memory_info, sizeof(perfstat_memory_total_t), 1) == -1) {
+ return -1;
+ }
+ return (jlong)(memory_info.pgsp_free * 4 * K);
+}
+
julong os::physical_memory() {
return Aix::physical_memory();
}
@@ -1584,7 +1603,7 @@ static char* reserve_shmated_memory (size_t bytes, char* requested_addr) {
// Now attach the shared segment.
// Note that we deliberately *don't* pass SHM_RND. The contract of os::attempt_reserve_memory_at() -
- // which invokes this function with a request address != NULL - is to map at the specified address
+ // which invokes this function with a request address != nullptr - is to map at the specified address
// excactly, or to fail. If the caller passed us an address that is not usable (aka not a valid segment
// boundary), shmat should not round down the address, or think up a completely new one.
// (In places where this matters, e.g. when reserving the heap, we take care of passing segment-aligned
@@ -1886,6 +1905,10 @@ void os::pd_realign_memory(char *addr, size_t bytes, size_t alignment_hint) {
void os::pd_free_memory(char *addr, size_t bytes, size_t alignment_hint) {
}
+size_t os::pd_pretouch_memory(void* first, void* last, size_t page_size) {
+ return page_size;
+}
+
void os::numa_make_global(char *addr, size_t bytes) {
}
@@ -2506,10 +2529,10 @@ int os::open(const char *path, int oflag, int mode) {
// IV90804: OPENING A FILE IN AFS WITH O_CLOEXEC FAILS WITH AN EINVAL ERROR APPLIES TO AIX 7100-04 17/04/14 PTF PECHANGE
int oflag_with_o_cloexec = oflag | O_CLOEXEC;
- int fd = ::open64(path, oflag_with_o_cloexec, mode);
+ int fd = ::open(path, oflag_with_o_cloexec, mode);
if (fd == -1) {
// we might fail in the open call when O_CLOEXEC is set, so try again without (see IV90804)
- fd = ::open64(path, oflag, mode);
+ fd = ::open(path, oflag, mode);
if (fd == -1) {
return -1;
}
@@ -2517,8 +2540,8 @@ int os::open(const char *path, int oflag, int mode) {
// If the open succeeded, the file might still be a directory.
{
- struct stat64 buf64;
- int ret = ::fstat64(fd, &buf64);
+ struct stat buf64;
+ int ret = ::fstat(fd, &buf64);
int st_mode = buf64.st_mode;
if (ret != -1) {
@@ -2572,17 +2595,17 @@ int os::open(const char *path, int oflag, int mode) {
int os::create_binary_file(const char* path, bool rewrite_existing) {
int oflags = O_WRONLY | O_CREAT;
oflags |= rewrite_existing ? O_TRUNC : O_EXCL;
- return ::open64(path, oflags, S_IREAD | S_IWRITE);
+ return ::open(path, oflags, S_IREAD | S_IWRITE);
}
// return current position of file pointer
jlong os::current_file_offset(int fd) {
- return (jlong)::lseek64(fd, (off64_t)0, SEEK_CUR);
+ return (jlong)::lseek(fd, (off_t)0, SEEK_CUR);
}
// move file pointer to the specified offset
jlong os::seek_to_file_offset(int fd, jlong offset) {
- return (jlong)::lseek64(fd, (off64_t)offset, SEEK_SET);
+ return (jlong)::lseek(fd, (off_t)offset, SEEK_SET);
}
// Map a block of memory.
@@ -3027,4 +3050,3 @@ void os::print_memory_mappings(char* addr, size_t bytes, outputStream* st) {}
void os::jfr_report_memory_info() {}
#endif // INCLUDE_JFR
-
diff --git a/src/hotspot/os/aix/os_perf_aix.cpp b/src/hotspot/os/aix/os_perf_aix.cpp
index e1719df48c3..b5ae1a6a725 100644
--- a/src/hotspot/os/aix/os_perf_aix.cpp
+++ b/src/hotspot/os/aix/os_perf_aix.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2022, IBM Corp.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2022, 2024, IBM Corp.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -87,6 +87,7 @@ static bool read_psinfo(const u_longlong_t& pid, psinfo_t& psinfo) {
}
len = fread(&psinfo, 1, sizeof(psinfo_t), fp);
+ fclose(fp);
return len == sizeof(psinfo_t);
}
diff --git a/src/hotspot/os/bsd/os_bsd.cpp b/src/hotspot/os/bsd/os_bsd.cpp
index 958372e20a5..dbc18794cfe 100644
--- a/src/hotspot/os/bsd/os_bsd.cpp
+++ b/src/hotspot/os/bsd/os_bsd.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -24,7 +24,6 @@
// no precompiled headers
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/compileBroker.hpp"
#include "compiler/disassembler.hpp"
@@ -181,6 +180,32 @@ void os::Bsd::print_uptime_info(outputStream* st) {
}
}
+jlong os::total_swap_space() {
+#if defined(__APPLE__)
+ struct xsw_usage vmusage;
+ size_t size = sizeof(vmusage);
+ if (sysctlbyname("vm.swapusage", &vmusage, &size, NULL, 0) != 0) {
+ return -1;
+ }
+ return (jlong)vmusage.xsu_total;
+#else
+ return -1;
+#endif
+}
+
+jlong os::free_swap_space() {
+#if defined(__APPLE__)
+ struct xsw_usage vmusage;
+ size_t size = sizeof(vmusage);
+ if (sysctlbyname("vm.swapusage", &vmusage, &size, NULL, 0) != 0) {
+ return -1;
+ }
+ return (jlong)vmusage.xsu_avail;
+#else
+ return -1;
+#endif
+}
+
julong os::physical_memory() {
return Bsd::physical_memory();
}
@@ -985,6 +1010,13 @@ void *os::Bsd::dlopen_helper(const char *filename, int mode, char *ebuf, int ebu
if (!ieee_handling) {
Events::log_dll_message(nullptr, "IEEE subnormal handling check failed before loading %s", filename);
log_info(os)("IEEE subnormal handling check failed before loading %s", filename);
+ if (CheckJNICalls) {
+ tty->print_cr("WARNING: IEEE subnormal handling check failed before loading %s", filename);
+ Thread* current = Thread::current();
+ if (current->is_Java_thread()) {
+ JavaThread::cast(current)->print_jni_stack();
+ }
+ }
}
// Save and restore the floating-point environment around dlopen().
@@ -1038,6 +1070,13 @@ void *os::Bsd::dlopen_helper(const char *filename, int mode, char *ebuf, int ebu
} else {
Events::log_dll_message(nullptr, "IEEE subnormal handling could not be corrected after loading %s", filename);
log_info(os)("IEEE subnormal handling could not be corrected after loading %s", filename);
+ if (CheckJNICalls) {
+ tty->print_cr("WARNING: IEEE subnormal handling could not be corrected after loading %s", filename);
+ Thread* current = Thread::current();
+ if (current->is_Java_thread()) {
+ JavaThread::cast(current)->print_jni_stack();
+ }
+ }
assert(false, "fesetenv didn't work");
}
}
@@ -1229,7 +1268,8 @@ void * os::dll_load(const char *filename, char *ebuf, int ebuflen) {
}
#endif // !__APPLE__
-int _print_dll_info_cb(const char * name, address base_address, address top_address, void * param) {
+static int _print_dll_info_cb(const char * name, address base_address,
+ address top_address, void * param) {
outputStream * out = (outputStream *) param;
out->print_cr(INTPTR_FORMAT " \t%s", (intptr_t)base_address, name);
return 0;
@@ -1623,6 +1663,10 @@ void os::pd_free_memory(char *addr, size_t bytes, size_t alignment_hint) {
::madvise(addr, bytes, MADV_DONTNEED);
}
+size_t os::pd_pretouch_memory(void* first, void* last, size_t page_size) {
+ return page_size;
+}
+
void os::numa_make_global(char *addr, size_t bytes) {
}
diff --git a/src/hotspot/os/linux/attachListener_linux.cpp b/src/hotspot/os/linux/attachListener_linux.cpp
index 41a24c45007..0e98bca0607 100644
--- a/src/hotspot/os/linux/attachListener_linux.cpp
+++ b/src/hotspot/os/linux/attachListener_linux.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -184,6 +184,8 @@ int LinuxAttachListener::init() {
char initial_path[UNIX_PATH_MAX]; // socket file during setup
int listener; // listener socket (file descriptor)
+ static_assert(sizeof(off_t) == 8, "Expected Large File Support in this file");
+
// register function to cleanup
if (!_atexit_registered) {
_atexit_registered = true;
@@ -446,14 +448,14 @@ AttachOperation* AttachListener::dequeue() {
void AttachListener::vm_start() {
char fn[UNIX_PATH_MAX];
- struct stat64 st;
+ struct stat st;
int ret;
int n = snprintf(fn, UNIX_PATH_MAX, "%s/.java_pid%d",
os::get_temp_directory(), os::current_process_id());
assert(n < (int)UNIX_PATH_MAX, "java_pid file name buffer overflow");
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == 0) {
ret = ::unlink(fn);
if (ret == -1) {
@@ -473,8 +475,8 @@ int AttachListener::pd_init() {
bool AttachListener::check_socket_file() {
int ret;
- struct stat64 st;
- ret = stat64(LinuxAttachListener::path(), &st);
+ struct stat st;
+ ret = stat(LinuxAttachListener::path(), &st);
if (ret == -1) { // need to restart attach listener.
log_debug(attach)("Socket file %s does not exist - Restart Attach Listener",
LinuxAttachListener::path());
@@ -513,14 +515,14 @@ bool AttachListener::is_init_trigger() {
}
char fn[PATH_MAX + 1];
int ret;
- struct stat64 st;
+ struct stat st;
os::snprintf_checked(fn, sizeof(fn), ".attach_pid%d", os::current_process_id());
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == -1) {
log_trace(attach)("Failed to find attach file: %s, trying alternate", fn);
snprintf(fn, sizeof(fn), "%s/.attach_pid%d",
os::get_temp_directory(), os::current_process_id());
- RESTARTABLE(::stat64(fn, &st), ret);
+ RESTARTABLE(::stat(fn, &st), ret);
if (ret == -1) {
log_debug(attach)("Failed to find attach file: %s", fn);
}
diff --git a/src/hotspot/os/linux/globals_linux.hpp b/src/hotspot/os/linux/globals_linux.hpp
index 15a0f7e97c9..068fd53b62d 100644
--- a/src/hotspot/os/linux/globals_linux.hpp
+++ b/src/hotspot/os/linux/globals_linux.hpp
@@ -130,7 +130,9 @@
develop(bool, DelayThreadStartALot, false, \
"Artificially delay thread starts randomly for testing.") \
\
-
+ product(bool, UseMadvPopulateWrite, true, DIAGNOSTIC, \
+ "Use MADV_POPULATE_WRITE in os::pd_pretouch_memory.") \
+ \
// end of RUNTIME_OS_FLAGS
diff --git a/src/hotspot/os/linux/hugepages.cpp b/src/hotspot/os/linux/hugepages.cpp
index 67645b91aa5..54e6c1adb7a 100644
--- a/src/hotspot/os/linux/hugepages.cpp
+++ b/src/hotspot/os/linux/hugepages.cpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2011, 2023, Red Hat Inc. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2011, 2024, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -36,15 +36,15 @@
#include
-StaticHugePageSupport::StaticHugePageSupport() :
+ExplicitHugePageSupport::ExplicitHugePageSupport() :
_initialized(false), _pagesizes(), _default_hugepage_size(SIZE_MAX), _inconsistent(false) {}
-os::PageSizes StaticHugePageSupport::pagesizes() const {
+os::PageSizes ExplicitHugePageSupport::pagesizes() const {
assert(_initialized, "Not initialized");
return _pagesizes;
}
-size_t StaticHugePageSupport::default_hugepage_size() const {
+size_t ExplicitHugePageSupport::default_hugepage_size() const {
assert(_initialized, "Not initialized");
return _default_hugepage_size;
}
@@ -132,9 +132,9 @@ static os::PageSizes scan_hugepages() {
return pagesizes;
}
-void StaticHugePageSupport::print_on(outputStream* os) {
+void ExplicitHugePageSupport::print_on(outputStream* os) {
if (_initialized) {
- os->print_cr("Static hugepage support:");
+ os->print_cr("Explicit hugepage support:");
for (size_t s = _pagesizes.smallest(); s != 0; s = _pagesizes.next_larger(s)) {
os->print_cr(" hugepage size: " EXACTFMT, EXACTFMTARGS(s));
}
@@ -143,18 +143,18 @@ void StaticHugePageSupport::print_on(outputStream* os) {
os->print_cr(" unknown.");
}
if (_inconsistent) {
- os->print_cr(" Support inconsistent. JVM will not use static hugepages.");
+ os->print_cr(" Support inconsistent. JVM will not use explicit hugepages.");
}
}
-void StaticHugePageSupport::scan_os() {
+void ExplicitHugePageSupport::scan_os() {
_default_hugepage_size = scan_default_hugepagesize();
if (_default_hugepage_size > 0) {
_pagesizes = scan_hugepages();
// See https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt: /proc/meminfo should match
// /sys/kernel/mm/hugepages/hugepages-xxxx. However, we may run on a broken kernel (e.g. on WSL)
// that only exposes /proc/meminfo but not /sys/kernel/mm/hugepages. In that case, we are not
- // sure about the state of hugepage support by the kernel, so we won't use static hugepages.
+ // sure about the state of hugepage support by the kernel, so we won't use explicit hugepages.
if (!_pagesizes.contains(_default_hugepage_size)) {
log_info(pagesize)("Unexpected configuration: default pagesize (" SIZE_FORMAT ") "
"has no associated directory in /sys/kernel/mm/hugepages..", _default_hugepage_size);
@@ -307,18 +307,18 @@ void ShmemTHPSupport::print_on(outputStream* os) {
}
}
-StaticHugePageSupport HugePages::_static_hugepage_support;
+ExplicitHugePageSupport HugePages::_explicit_hugepage_support;
THPSupport HugePages::_thp_support;
ShmemTHPSupport HugePages::_shmem_thp_support;
void HugePages::initialize() {
- _static_hugepage_support.scan_os();
+ _explicit_hugepage_support.scan_os();
_thp_support.scan_os();
_shmem_thp_support.scan_os();
}
void HugePages::print_on(outputStream* os) {
- _static_hugepage_support.print_on(os);
+ _explicit_hugepage_support.print_on(os);
_thp_support.print_on(os);
_shmem_thp_support.print_on(os);
}
diff --git a/src/hotspot/os/linux/hugepages.hpp b/src/hotspot/os/linux/hugepages.hpp
index ce9ab36edcc..2e61fabc5a5 100644
--- a/src/hotspot/os/linux/hugepages.hpp
+++ b/src/hotspot/os/linux/hugepages.hpp
@@ -1,6 +1,6 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2011, 2023, Red Hat Inc. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2011, 2024, Red Hat Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -34,13 +34,13 @@ class outputStream;
// Header contains the interface that reads OS information about
// available hugepage support:
-// - class StaticHugePageSupport - about static (non-THP) hugepages
+// - class ExplicitHugePageSupport - about explicit (non-THP) hugepages
// - class THPSupport - about transparent huge pages
// and:
// - class HugePages - a static umbrella wrapper
// Information about static (non-thp) hugepages
-class StaticHugePageSupport {
+class ExplicitHugePageSupport {
bool _initialized;
// All supported hugepage sizes (sizes for which entries exist
@@ -56,7 +56,7 @@ class StaticHugePageSupport {
bool _inconsistent;
public:
- StaticHugePageSupport();
+ ExplicitHugePageSupport();
void scan_os();
@@ -122,18 +122,18 @@ class ShmemTHPSupport {
// Umbrella static interface
class HugePages : public AllStatic {
- static StaticHugePageSupport _static_hugepage_support;
+ static ExplicitHugePageSupport _explicit_hugepage_support;
static THPSupport _thp_support;
static ShmemTHPSupport _shmem_thp_support;
public:
- static const StaticHugePageSupport& static_info() { return _static_hugepage_support; }
+ static const ExplicitHugePageSupport& explicit_hugepage_info() { return _explicit_hugepage_support; }
static const THPSupport& thp_info() { return _thp_support; }
static const ShmemTHPSupport& shmem_thp_info() { return _shmem_thp_support; }
- static size_t default_static_hugepage_size() { return _static_hugepage_support.default_hugepage_size(); }
- static bool supports_static_hugepages() { return default_static_hugepage_size() > 0 && !_static_hugepage_support.inconsistent(); }
+ static size_t default_explicit_hugepage_size() { return _explicit_hugepage_support.default_hugepage_size(); }
+ static bool supports_explicit_hugepages() { return default_explicit_hugepage_size() > 0 && !_explicit_hugepage_support.inconsistent(); }
static bool supports_thp() { return thp_mode() == THPMode::madvise || thp_mode() == THPMode::always; }
static THPMode thp_mode() { return _thp_support.mode(); }
diff --git a/src/hotspot/os/linux/osContainer_linux.cpp b/src/hotspot/os/linux/osContainer_linux.cpp
index ff2220bb33f..d3ba04742d1 100644
--- a/src/hotspot/os/linux/osContainer_linux.cpp
+++ b/src/hotspot/os/linux/osContainer_linux.cpp
@@ -167,7 +167,7 @@ jlong OSContainer::pids_current() {
void OSContainer::print_container_helper(outputStream* st, jlong j, const char* metrics) {
st->print("%s: ", metrics);
- if (j > 0) {
+ if (j >= 0) {
if (j >= 1024) {
st->print_cr(UINT64_FORMAT " k", uint64_t(j) / K);
} else {
diff --git a/src/hotspot/os/linux/os_linux.cpp b/src/hotspot/os/linux/os_linux.cpp
index 95b55b877a9..e1ef0443885 100644
--- a/src/hotspot/os/linux/os_linux.cpp
+++ b/src/hotspot/os/linux/os_linux.cpp
@@ -25,7 +25,6 @@
// no precompiled headers
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/compileBroker.hpp"
#include "compiler/disassembler.hpp"
@@ -292,6 +291,34 @@ julong os::Linux::free_memory() {
return free_mem;
}
+jlong os::total_swap_space() {
+ if (OSContainer::is_containerized()) {
+ if (OSContainer::memory_limit_in_bytes() > 0) {
+ return (jlong)(OSContainer::memory_and_swap_limit_in_bytes() - OSContainer::memory_limit_in_bytes());
+ }
+ }
+ struct sysinfo si;
+ int ret = sysinfo(&si);
+ if (ret != 0) {
+ return -1;
+ }
+ return (jlong)(si.totalswap * si.mem_unit);
+}
+
+jlong os::free_swap_space() {
+ if (OSContainer::is_containerized()) {
+ // TODO add a good implementation
+ return -1;
+ } else {
+ struct sysinfo si;
+ int ret = sysinfo(&si);
+ if (ret != 0) {
+ return -1;
+ }
+ return (jlong)(si.freeswap * si.mem_unit);
+ }
+}
+
julong os::physical_memory() {
jlong phys_mem = 0;
if (OSContainer::is_containerized()) {
@@ -420,7 +447,7 @@ pid_t os::Linux::gettid() {
julong os::Linux::host_swap() {
struct sysinfo si;
sysinfo(&si);
- return (julong)si.totalswap;
+ return (julong)(si.totalswap * si.mem_unit);
}
// Most versions of linux have a bug where the number of processors are
@@ -1816,6 +1843,13 @@ void * os::Linux::dlopen_helper(const char *filename, char *ebuf, int ebuflen) {
if (!ieee_handling) {
Events::log_dll_message(nullptr, "IEEE subnormal handling check failed before loading %s", filename);
log_info(os)("IEEE subnormal handling check failed before loading %s", filename);
+ if (CheckJNICalls) {
+ tty->print_cr("WARNING: IEEE subnormal handling check failed before loading %s", filename);
+ Thread* current = Thread::current();
+ if (current->is_Java_thread()) {
+ JavaThread::cast(current)->print_jni_stack();
+ }
+ }
}
// Save and restore the floating-point environment around dlopen().
@@ -1870,6 +1904,13 @@ void * os::Linux::dlopen_helper(const char *filename, char *ebuf, int ebuflen) {
} else {
Events::log_dll_message(nullptr, "IEEE subnormal handling could not be corrected after loading %s", filename);
log_info(os)("IEEE subnormal handling could not be corrected after loading %s", filename);
+ if (CheckJNICalls) {
+ tty->print_cr("WARNING: IEEE subnormal handling could not be corrected after loading %s", filename);
+ Thread* current = Thread::current();
+ if (current->is_Java_thread()) {
+ JavaThread::cast(current)->print_jni_stack();
+ }
+ }
assert(false, "fesetenv didn't work");
}
}
@@ -2845,6 +2886,8 @@ void os::jvm_path(char *buf, jint buflen) {
void linux_wrap_code(char* base, size_t size) {
static volatile jint cnt = 0;
+ static_assert(sizeof(off_t) == 8, "Expected Large File Support in this file");
+
if (!UseOprofile) {
return;
}
@@ -2969,6 +3012,15 @@ void os::pd_commit_memory_or_exit(char* addr, size_t size, bool exec,
#define MADV_HUGEPAGE 14
#endif
+// Define MADV_POPULATE_WRITE here so we can build HotSpot on old systems.
+#define MADV_POPULATE_WRITE_value 23
+#ifndef MADV_POPULATE_WRITE
+ #define MADV_POPULATE_WRITE MADV_POPULATE_WRITE_value
+#else
+ // Sanity-check our assumed default value if we build with a new enough libc.
+ static_assert(MADV_POPULATE_WRITE == MADV_POPULATE_WRITE_value);
+#endif
+
// Note that the value for MAP_FIXED_NOREPLACE differs between architectures, but all architectures
// supported by OpenJDK share the same flag value.
#define MAP_FIXED_NOREPLACE_value 0x100000
@@ -3028,6 +3080,31 @@ void os::pd_free_memory(char *addr, size_t bytes, size_t alignment_hint) {
}
}
+size_t os::pd_pretouch_memory(void* first, void* last, size_t page_size) {
+ const size_t len = pointer_delta(last, first, sizeof(char)) + page_size;
+ // Use madvise to pretouch on Linux when THP is used, and fallback to the
+ // common method if unsupported. THP can form right after madvise rather than
+ // being assembled later.
+ if (HugePages::thp_mode() == THPMode::always || UseTransparentHugePages) {
+ int err = 0;
+ if (UseMadvPopulateWrite &&
+ ::madvise(first, len, MADV_POPULATE_WRITE) == -1) {
+ err = errno;
+ }
+ if (!UseMadvPopulateWrite || err == EINVAL) { // Not to use or not supported
+ // When using THP we need to always pre-touch using small pages as the
+ // OS will initially always use small pages.
+ return os::vm_page_size();
+ } else if (err != 0) {
+ log_info(gc, os)("::madvise(" PTR_FORMAT ", " SIZE_FORMAT ", %d) failed; "
+ "error='%s' (errno=%d)", p2i(first), len,
+ MADV_POPULATE_WRITE, os::strerror(err), err);
+ }
+ return 0;
+ }
+ return page_size;
+}
+
void os::numa_make_global(char *addr, size_t bytes) {
Linux::numa_interleave_memory(addr, bytes);
}
@@ -3747,14 +3824,14 @@ bool os::unguard_memory(char* addr, size_t size) {
}
static int hugetlbfs_page_size_flag(size_t page_size) {
- if (page_size != HugePages::default_static_hugepage_size()) {
+ if (page_size != HugePages::default_explicit_hugepage_size()) {
return (exact_log2(page_size) << MAP_HUGE_SHIFT);
}
return 0;
}
static bool hugetlbfs_sanity_check(size_t page_size) {
- const os::PageSizes page_sizes = HugePages::static_info().pagesizes();
+ const os::PageSizes page_sizes = HugePages::explicit_hugepage_info().pagesizes();
assert(page_sizes.contains(page_size), "Invalid page sizes passed");
// Include the page size flag to ensure we sanity check the correct page size.
@@ -3934,8 +4011,8 @@ void os::Linux::large_page_init() {
return;
}
- // Check if the OS supports static hugepages.
- if (!UseTransparentHugePages && !HugePages::supports_static_hugepages()) {
+ // Check if the OS supports explicit hugepages.
+ if (!UseTransparentHugePages && !HugePages::supports_explicit_hugepages()) {
warn_no_large_pages_configured();
UseLargePages = false;
return;
@@ -3954,12 +4031,12 @@ void os::Linux::large_page_init() {
} else {
- // In static hugepage mode:
- // - os::large_page_size() is the default static hugepage size (/proc/meminfo "Hugepagesize")
+ // In explicit hugepage mode:
+ // - os::large_page_size() is the default explicit hugepage size (/proc/meminfo "Hugepagesize")
// - os::pagesizes() contains all hugepage sizes the kernel supports, regardless whether there
// are pages configured in the pool or not (from /sys/kernel/hugepages/hugepage-xxxx ...)
- os::PageSizes all_large_pages = HugePages::static_info().pagesizes();
- const size_t default_large_page_size = HugePages::default_static_hugepage_size();
+ os::PageSizes all_large_pages = HugePages::explicit_hugepage_info().pagesizes();
+ const size_t default_large_page_size = HugePages::default_explicit_hugepage_size();
// 3) Consistency check and post-processing
@@ -4043,7 +4120,7 @@ static bool commit_memory_special(size_t bytes,
char* req_addr,
bool exec) {
assert(UseLargePages, "Should only get here for huge pages");
- assert(!UseTransparentHugePages, "Should only get here for static hugepage mode");
+ assert(!UseTransparentHugePages, "Should only get here for explicit hugepage mode");
assert(is_aligned(bytes, page_size), "Unaligned size");
assert(is_aligned(req_addr, page_size), "Unaligned address");
assert(req_addr != nullptr, "Must have a requested address for special mappings");
@@ -4077,7 +4154,7 @@ static char* reserve_memory_special_huge_tlbfs(size_t bytes,
size_t page_size,
char* req_addr,
bool exec) {
- const os::PageSizes page_sizes = HugePages::static_info().pagesizes();
+ const os::PageSizes page_sizes = HugePages::explicit_hugepage_info().pagesizes();
assert(UseLargePages, "only for Huge TLBFS large pages");
assert(is_aligned(req_addr, alignment), "Must be");
assert(is_aligned(req_addr, page_size), "Must be");
@@ -4156,7 +4233,7 @@ size_t os::large_page_size() {
return _large_page_size;
}
-// static hugepages (hugetlbfs) allow application to commit large page memory
+// explicit hugepages (hugetlbfs) allow application to commit large page memory
// on demand.
// However, when committing memory with hugepages fails, the region
// that was supposed to be committed will lose the old reservation
@@ -4207,7 +4284,7 @@ char* os::pd_attempt_reserve_memory_at(char* requested_addr, size_t bytes, bool
size_t os::vm_min_address() {
// Determined by sysctl vm.mmap_min_addr. It exists as a safety zone to prevent
- // NULL pointer dereferences.
+ // null pointer dereferences.
// Most distros set this value to 64 KB. It *can* be zero, but rarely is. Here,
// we impose a minimum value if vm.mmap_min_addr is too low, for increased protection.
static size_t value = 0;
@@ -4335,7 +4412,7 @@ jlong os::Linux::fast_thread_cpu_time(clockid_t clockid) {
// the number of bytes written to out_fd is returned if transfer was successful
// otherwise, returns -1 that implies an error
jlong os::Linux::sendfile(int out_fd, int in_fd, jlong* offset, jlong count) {
- return ::sendfile64(out_fd, in_fd, (off64_t*)offset, (size_t)count);
+ return ::sendfile(out_fd, in_fd, (off_t*)offset, (size_t)count);
}
// Determine if the vmid is the parent pid for a child in a PID namespace.
@@ -4450,6 +4527,9 @@ void os::init(void) {
check_pax();
+ // Check the availability of MADV_POPULATE_WRITE.
+ FLAG_SET_DEFAULT(UseMadvPopulateWrite, (::madvise(0, 0, MADV_POPULATE_WRITE) == 0));
+
os::Posix::init();
}
@@ -5028,14 +5108,14 @@ int os::open(const char *path, int oflag, int mode) {
oflag |= O_CLOEXEC;
#endif
- int fd = ::open64(path, oflag, mode);
+ int fd = ::open(path, oflag, mode);
if (fd == -1) return -1;
//If the open succeeded, the file might still be a directory
{
- struct stat64 buf64;
- int ret = ::fstat64(fd, &buf64);
- int st_mode = buf64.st_mode;
+ struct stat buf;
+ int ret = ::fstat(fd, &buf);
+ int st_mode = buf.st_mode;
if (ret != -1) {
if ((st_mode & S_IFMT) == S_IFDIR) {
@@ -5072,17 +5152,17 @@ int os::open(const char *path, int oflag, int mode) {
int os::create_binary_file(const char* path, bool rewrite_existing) {
int oflags = O_WRONLY | O_CREAT;
oflags |= rewrite_existing ? O_TRUNC : O_EXCL;
- return ::open64(path, oflags, S_IREAD | S_IWRITE);
+ return ::open(path, oflags, S_IREAD | S_IWRITE);
}
// return current position of file pointer
jlong os::current_file_offset(int fd) {
- return (jlong)::lseek64(fd, (off64_t)0, SEEK_CUR);
+ return (jlong)::lseek(fd, (off_t)0, SEEK_CUR);
}
// move file pointer to the specified offset
jlong os::seek_to_file_offset(int fd, jlong offset) {
- return (jlong)::lseek64(fd, (off64_t)offset, SEEK_SET);
+ return (jlong)::lseek(fd, (off_t)offset, SEEK_SET);
}
// Map a block of memory.
diff --git a/src/hotspot/os/posix/os_posix.cpp b/src/hotspot/os/posix/os_posix.cpp
index 1158392fa49..339cc475b34 100644
--- a/src/hotspot/os/posix/os_posix.cpp
+++ b/src/hotspot/os/posix/os_posix.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -265,7 +265,7 @@ bool os::dir_is_empty(const char* path) {
return result;
}
-static char* reserve_mmapped_memory(size_t bytes, char* requested_addr) {
+static char* reserve_mmapped_memory(size_t bytes, char* requested_addr, MEMFLAGS flag) {
char * addr;
int flags = MAP_PRIVATE NOT_AIX( | MAP_NORESERVE ) | MAP_ANONYMOUS;
if (requested_addr != nullptr) {
@@ -280,13 +280,14 @@ static char* reserve_mmapped_memory(size_t bytes, char* requested_addr) {
flags, -1, 0);
if (addr != MAP_FAILED) {
- MemTracker::record_virtual_memory_reserve((address)addr, bytes, CALLER_PC);
+ MemTracker::record_virtual_memory_reserve((address)addr, bytes, CALLER_PC, flag);
return addr;
}
return nullptr;
}
static int util_posix_fallocate(int fd, off_t offset, off_t len) {
+ static_assert(sizeof(off_t) == 8, "Expected Large File Support in this file");
#ifdef __APPLE__
fstore_t store = { F_ALLOCATECONTIG, F_PEOFPOSMODE, 0, len };
// First we try to get a continuous chunk of disk space
@@ -392,7 +393,7 @@ char* os::reserve_memory_aligned(size_t size, size_t alignment, bool exec) {
return chop_extra_memory(size, alignment, extra_base, extra_size);
}
-char* os::map_memory_to_file_aligned(size_t size, size_t alignment, int file_desc) {
+char* os::map_memory_to_file_aligned(size_t size, size_t alignment, int file_desc, MEMFLAGS flag) {
size_t extra_size = calculate_aligned_extra_size(size, alignment);
// For file mapping, we do not call os:map_memory_to_file(size,fd) since:
// - we later chop away parts of the mapping using os::release_memory and that could fail if the
@@ -400,7 +401,7 @@ char* os::map_memory_to_file_aligned(size_t size, size_t alignment, int file_des
// - The memory API os::reserve_memory uses is an implementation detail. It may (and usually is)
// mmap but it also may System V shared memory which cannot be uncommitted as a whole, so
// chopping off and unmapping excess bits back and front (see below) would not work.
- char* extra_base = reserve_mmapped_memory(extra_size, nullptr);
+ char* extra_base = reserve_mmapped_memory(extra_size, nullptr, flag);
if (extra_base == nullptr) {
return nullptr;
}
@@ -752,11 +753,11 @@ void os::dll_unload(void *lib) {
}
jlong os::lseek(int fd, jlong offset, int whence) {
- return (jlong) BSD_ONLY(::lseek) NOT_BSD(::lseek64)(fd, offset, whence);
+ return (jlong) ::lseek(fd, offset, whence);
}
int os::ftruncate(int fd, jlong length) {
- return BSD_ONLY(::ftruncate) NOT_BSD(::ftruncate64)(fd, length);
+ return ::ftruncate(fd, length);
}
const char* os::get_current_directory(char *buf, size_t buflen) {
diff --git a/src/hotspot/os/posix/os_posix.hpp b/src/hotspot/os/posix/os_posix.hpp
index 8c71516f70b..d872a6dae89 100644
--- a/src/hotspot/os/posix/os_posix.hpp
+++ b/src/hotspot/os/posix/os_posix.hpp
@@ -31,7 +31,7 @@
// Note: the Posix API aims to capture functionality available on all Posix
// compliant platforms, but in practice the implementations may depend on
-// non-Posix functionality. For example, the use of lseek64 and ftruncate64.
+// non-Posix functionality.
// This use of non-Posix API's is made possible by compiling/linking in a mode
// that is not restricted to being fully Posix complaint, such as by declaring
// -D_GNU_SOURCE. But be aware that in doing so we may enable non-Posix
diff --git a/src/hotspot/os/posix/signals_posix.cpp b/src/hotspot/os/posix/signals_posix.cpp
index eaadb367315..6a958f8903b 100644
--- a/src/hotspot/os/posix/signals_posix.cpp
+++ b/src/hotspot/os/posix/signals_posix.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -340,7 +340,7 @@ static const struct {
////////////////////////////////////////////////////////////////////////////////
// sun.misc.Signal and BREAK_SIGNAL support
-void jdk_misc_signal_init() {
+static void jdk_misc_signal_init() {
// Initialize signal structures
::memset((void*)pending_signals, 0, sizeof(pending_signals));
@@ -380,7 +380,7 @@ int os::signal_wait() {
////////////////////////////////////////////////////////////////////////////////
// signal chaining support
-struct sigaction* get_chained_signal_action(int sig) {
+static struct sigaction* get_chained_signal_action(int sig) {
struct sigaction *actp = nullptr;
if (libjsig_is_loaded) {
@@ -1245,7 +1245,7 @@ int os::get_signal_number(const char* signal_name) {
return -1;
}
-void set_signal_handler(int sig) {
+static void set_signal_handler(int sig) {
// Check for overwrite.
struct sigaction oldAct;
sigaction(sig, (struct sigaction*)nullptr, &oldAct);
@@ -1292,7 +1292,7 @@ void set_signal_handler(int sig) {
// install signal handlers for signals that HotSpot needs to
// handle in order to support Java-level exception handling.
-void install_signal_handlers() {
+static void install_signal_handlers() {
// signal-chaining
typedef void (*signal_setting_t)();
signal_setting_t begin_signal_setting = nullptr;
@@ -1723,7 +1723,7 @@ static void SR_handler(int sig, siginfo_t* siginfo, void* context) {
errno = old_errno;
}
-int SR_initialize() {
+static int SR_initialize() {
struct sigaction act;
char *s;
// Get signal number to use for suspend/resume
diff --git a/src/hotspot/os/windows/attachListener_windows.cpp b/src/hotspot/os/windows/attachListener_windows.cpp
index da3b6c56739..7e455cf0a49 100644
--- a/src/hotspot/os/windows/attachListener_windows.cpp
+++ b/src/hotspot/os/windows/attachListener_windows.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -248,10 +248,13 @@ Win32AttachOperation* Win32AttachListener::dequeue() {
DWORD res = ::WaitForSingleObject(enqueued_ops_semaphore(), INFINITE);
// returning from WaitForSingleObject will have decreased
// the current count of the semaphore by 1.
- guarantee(res == WAIT_OBJECT_0, "wait failed");
+ guarantee(res != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ guarantee(res == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", res);
res = ::WaitForSingleObject(mutex(), INFINITE);
- guarantee(res == WAIT_OBJECT_0, "wait failed");
+ guarantee(res != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ guarantee(res == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", res);
+
Win32AttachOperation* op = head();
if (op != nullptr) {
@@ -338,6 +341,9 @@ void Win32AttachOperation::complete(jint result, bufferedStream* result_stream)
}
DWORD res = ::WaitForSingleObject(Win32AttachListener::mutex(), INFINITE);
+ assert(res != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(res == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", res);
+
if (res == WAIT_OBJECT_0) {
// put the operation back on the available list
diff --git a/src/hotspot/os/windows/os_windows.cpp b/src/hotspot/os/windows/os_windows.cpp
index 1b5c7850bf2..f9c3f23f0a6 100644
--- a/src/hotspot/os/windows/os_windows.cpp
+++ b/src/hotspot/os/windows/os_windows.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -27,7 +27,6 @@
// no precompiled headers
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/compileBroker.hpp"
@@ -840,6 +839,20 @@ julong os::win32::available_memory() {
return (julong)ms.ullAvailPhys;
}
+jlong os::total_swap_space() {
+ MEMORYSTATUSEX ms;
+ ms.dwLength = sizeof(ms);
+ GlobalMemoryStatusEx(&ms);
+ return (jlong) ms.ullTotalPageFile;
+}
+
+jlong os::free_swap_space() {
+ MEMORYSTATUSEX ms;
+ ms.dwLength = sizeof(ms);
+ GlobalMemoryStatusEx(&ms);
+ return (jlong) ms.ullAvailPageFile;
+}
+
julong os::physical_memory() {
return win32::physical_memory();
}
@@ -2469,9 +2482,9 @@ LONG Handle_IDiv_Exception(struct _EXCEPTION_POINTERS* exceptionInfo) {
#elif defined(_M_AMD64)
PCONTEXT ctx = exceptionInfo->ContextRecord;
address pc = (address)ctx->Rip;
- guarantee(pc[0] >= Assembler::REX && pc[0] <= Assembler::REX_WRXB && pc[1] == 0xF7 || pc[0] == 0xF7,
+ guarantee((pc[0] >= Assembler::REX && pc[0] <= Assembler::REX_WRXB && pc[1] == 0xF7) || pc[0] == 0xF7,
"not an idiv opcode, pc[0] = 0x%x and pc[1] = 0x%x", pc[0], pc[1]);
- guarantee(pc[0] >= Assembler::REX && pc[0] <= Assembler::REX_WRXB && (pc[2] & ~0x7) == 0xF8 || (pc[1] & ~0x7) == 0xF8,
+ guarantee((pc[0] >= Assembler::REX && pc[0] <= Assembler::REX_WRXB && (pc[2] & ~0x7) == 0xF8) || (pc[1] & ~0x7) == 0xF8,
"cannot handle non-register operands, pc[0] = 0x%x, pc[1] = 0x%x and pc[2] = 0x%x", pc[0], pc[1], pc[2]);
if (pc[0] == 0xF7) {
// set correct result values and continue after idiv instruction
@@ -3330,7 +3343,7 @@ char* os::replace_existing_mapping_with_file_mapping(char* base, size_t size, in
// Multiple threads can race in this code but it's not possible to unmap small sections of
// virtual space to get requested alignment, like posix-like os's.
// Windows prevents multiple thread from remapping over each other so this loop is thread-safe.
-static char* map_or_reserve_memory_aligned(size_t size, size_t alignment, int file_desc) {
+static char* map_or_reserve_memory_aligned(size_t size, size_t alignment, int file_desc, MEMFLAGS flag = mtNone) {
assert(is_aligned(alignment, os::vm_allocation_granularity()),
"Alignment must be a multiple of allocation granularity (page size)");
assert(is_aligned(size, os::vm_allocation_granularity()),
@@ -3343,8 +3356,8 @@ static char* map_or_reserve_memory_aligned(size_t size, size_t alignment, int fi
static const int max_attempts = 20;
for (int attempt = 0; attempt < max_attempts && aligned_base == nullptr; attempt ++) {
- char* extra_base = file_desc != -1 ? os::map_memory_to_file(extra_size, file_desc) :
- os::reserve_memory(extra_size);
+ char* extra_base = file_desc != -1 ? os::map_memory_to_file(extra_size, file_desc, flag) :
+ os::reserve_memory(extra_size, false, flag);
if (extra_base == nullptr) {
return nullptr;
}
@@ -3360,8 +3373,8 @@ static char* map_or_reserve_memory_aligned(size_t size, size_t alignment, int fi
// Attempt to map, into the just vacated space, the slightly smaller aligned area.
// Which may fail, hence the loop.
- aligned_base = file_desc != -1 ? os::attempt_map_memory_to_file_at(aligned_base, size, file_desc) :
- os::attempt_reserve_memory_at(aligned_base, size);
+ aligned_base = file_desc != -1 ? os::attempt_map_memory_to_file_at(aligned_base, size, file_desc, flag) :
+ os::attempt_reserve_memory_at(aligned_base, size, false, flag);
}
assert(aligned_base != nullptr, "Did not manage to re-map after %d attempts?", max_attempts);
@@ -3374,8 +3387,8 @@ char* os::reserve_memory_aligned(size_t size, size_t alignment, bool exec) {
return map_or_reserve_memory_aligned(size, alignment, -1 /* file_desc */);
}
-char* os::map_memory_to_file_aligned(size_t size, size_t alignment, int fd) {
- return map_or_reserve_memory_aligned(size, alignment, fd);
+char* os::map_memory_to_file_aligned(size_t size, size_t alignment, int fd, MEMFLAGS flag) {
+ return map_or_reserve_memory_aligned(size, alignment, fd, flag);
}
char* os::pd_reserve_memory(size_t bytes, bool exec) {
@@ -3796,6 +3809,11 @@ bool os::unguard_memory(char* addr, size_t bytes) {
void os::pd_realign_memory(char *addr, size_t bytes, size_t alignment_hint) { }
void os::pd_free_memory(char *addr, size_t bytes, size_t alignment_hint) { }
+
+size_t os::pd_pretouch_memory(void* first, void* last, size_t page_size) {
+ return page_size;
+}
+
void os::numa_make_global(char *addr, size_t bytes) { }
void os::numa_make_local(char *addr, size_t bytes, int lgrp_hint) { }
bool os::numa_topology_changed() { return false; }
@@ -5354,7 +5372,8 @@ int PlatformEvent::park(jlong Millis) {
phri = new HighResolutionInterval(prd);
}
rv = ::WaitForSingleObject(_ParkHandle, prd);
- assert(rv == WAIT_OBJECT_0 || rv == WAIT_TIMEOUT, "WaitForSingleObject failed");
+ assert(rv != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(rv == WAIT_OBJECT_0 || rv == WAIT_TIMEOUT, "WaitForSingleObject failed with return value: %lu", rv);
if (rv == WAIT_TIMEOUT) {
Millis -= prd;
}
@@ -5393,7 +5412,8 @@ void PlatformEvent::park() {
// spin attempts by this thread.
while (_Event < 0) {
DWORD rv = ::WaitForSingleObject(_ParkHandle, INFINITE);
- assert(rv == WAIT_OBJECT_0, "WaitForSingleObject failed");
+ assert(rv != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(rv == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", rv);
}
// Usually we'll find _Event == 0 at this point, but as
@@ -5456,16 +5476,25 @@ void Parker::park(bool isAbsolute, jlong time) {
JavaThread* thread = JavaThread::current();
// Don't wait if interrupted or already triggered
- if (thread->is_interrupted(false) ||
- WaitForSingleObject(_ParkHandle, 0) == WAIT_OBJECT_0) {
+ if (thread->is_interrupted(false)) {
ResetEvent(_ParkHandle);
return;
} else {
- ThreadBlockInVM tbivm(thread);
- OSThreadWaitState osts(thread->osthread(), false /* not Object.wait() */);
+ DWORD rv = WaitForSingleObject(_ParkHandle, 0);
+ assert(rv != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(rv == WAIT_OBJECT_0 || rv == WAIT_TIMEOUT, "WaitForSingleObject failed with return value: %lu", rv);
+ if (rv == WAIT_OBJECT_0) {
+ ResetEvent(_ParkHandle);
+ return;
+ } else {
+ ThreadBlockInVM tbivm(thread);
+ OSThreadWaitState osts(thread->osthread(), false /* not Object.wait() */);
- WaitForSingleObject(_ParkHandle, time);
- ResetEvent(_ParkHandle);
+ rv = WaitForSingleObject(_ParkHandle, time);
+ assert(rv != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(rv == WAIT_OBJECT_0 || rv == WAIT_TIMEOUT, "WaitForSingleObject failed with return value: %lu", rv);
+ ResetEvent(_ParkHandle);
+ }
}
}
@@ -5541,7 +5570,9 @@ int os::fork_and_exec(const char* cmd) {
if (rslt) {
// Wait until child process exits.
- WaitForSingleObject(pi.hProcess, INFINITE);
+ DWORD rv = WaitForSingleObject(pi.hProcess, INFINITE);
+ assert(rv != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(rv == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", rv);
GetExitCodeProcess(pi.hProcess, &exit_code);
diff --git a/src/hotspot/os/windows/threadCritical_windows.cpp b/src/hotspot/os/windows/threadCritical_windows.cpp
index f781a422484..c85143f8093 100644
--- a/src/hotspot/os/windows/threadCritical_windows.cpp
+++ b/src/hotspot/os/windows/threadCritical_windows.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -64,7 +64,8 @@ ThreadCritical::ThreadCritical() {
if (lock_owner != current_thread) {
// Grab the lock before doing anything.
DWORD ret = WaitForSingleObject(lock_event, INFINITE);
- assert(ret == WAIT_OBJECT_0, "unexpected return value from WaitForSingleObject");
+ assert(ret != WAIT_FAILED, "WaitForSingleObject failed with error code: %lu", GetLastError());
+ assert(ret == WAIT_OBJECT_0, "WaitForSingleObject failed with return value: %lu", ret);
lock_owner = current_thread;
}
// Atomicity isn't required. Bump the recursion count.
diff --git a/src/hotspot/os_cpu/aix_ppc/os_aix_ppc.cpp b/src/hotspot/os_cpu/aix_ppc/os_aix_ppc.cpp
index 5e0086521aa..242042d4247 100644
--- a/src/hotspot/os_cpu/aix_ppc/os_aix_ppc.cpp
+++ b/src/hotspot/os_cpu/aix_ppc/os_aix_ppc.cpp
@@ -28,7 +28,6 @@
#include "asm/assembler.inline.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/bsd_aarch64/os_bsd_aarch64.cpp b/src/hotspot/os_cpu/bsd_aarch64/os_bsd_aarch64.cpp
index fbd7c4eccd4..4750ed88056 100644
--- a/src/hotspot/os_cpu/bsd_aarch64/os_bsd_aarch64.cpp
+++ b/src/hotspot/os_cpu/bsd_aarch64/os_bsd_aarch64.cpp
@@ -29,7 +29,6 @@
#include "classfile/classLoader.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/bsd_x86/os_bsd_x86.cpp b/src/hotspot/os_cpu/bsd_x86/os_bsd_x86.cpp
index 37b92bc7ffd..c73e83996ff 100644
--- a/src/hotspot/os_cpu/bsd_x86/os_bsd_x86.cpp
+++ b/src/hotspot/os_cpu/bsd_x86/os_bsd_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -26,7 +26,6 @@
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
@@ -351,7 +350,7 @@ frame os::get_sender_for_C_frame(frame* fr) {
return frame(fr->sender_sp(), fr->link(), fr->sender_pc());
}
-intptr_t* _get_previous_fp() {
+static intptr_t* _get_previous_fp() {
#if defined(__clang__) || defined(__llvm__)
intptr_t **ebp;
__asm__("mov %%" SPELL_REG_FP ", %0":"=r"(ebp));
diff --git a/src/hotspot/os_cpu/bsd_zero/os_bsd_zero.cpp b/src/hotspot/os_cpu/bsd_zero/os_bsd_zero.cpp
index 012f85ac0ff..0fc9484ce23 100644
--- a/src/hotspot/os_cpu/bsd_zero/os_bsd_zero.cpp
+++ b/src/hotspot/os_cpu/bsd_zero/os_bsd_zero.cpp
@@ -27,7 +27,6 @@
#include "asm/assembler.inline.hpp"
#include "atomic_bsd_zero.hpp"
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/linux_aarch64/os_linux_aarch64.cpp b/src/hotspot/os_cpu/linux_aarch64/os_linux_aarch64.cpp
index 4835eb9405a..3698896abb7 100644
--- a/src/hotspot/os_cpu/linux_aarch64/os_linux_aarch64.cpp
+++ b/src/hotspot/os_cpu/linux_aarch64/os_linux_aarch64.cpp
@@ -27,7 +27,6 @@
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "code/nativeInst.hpp"
#include "interpreter/interpreter.hpp"
diff --git a/src/hotspot/os_cpu/linux_arm/os_linux_arm.cpp b/src/hotspot/os_cpu/linux_arm/os_linux_arm.cpp
index 86e8ed25618..55127058843 100644
--- a/src/hotspot/os_cpu/linux_arm/os_linux_arm.cpp
+++ b/src/hotspot/os_cpu/linux_arm/os_linux_arm.cpp
@@ -25,7 +25,6 @@
// no precompiled headers
#include "asm/assembler.inline.hpp"
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/linux_ppc/os_linux_ppc.cpp b/src/hotspot/os_cpu/linux_ppc/os_linux_ppc.cpp
index b570e3b6d7f..0b666f29c31 100644
--- a/src/hotspot/os_cpu/linux_ppc/os_linux_ppc.cpp
+++ b/src/hotspot/os_cpu/linux_ppc/os_linux_ppc.cpp
@@ -28,7 +28,6 @@
#include "asm/assembler.inline.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/linux_riscv/os_linux_riscv.cpp b/src/hotspot/os_cpu/linux_riscv/os_linux_riscv.cpp
index 282467bc9e0..9f13e2bdd2c 100644
--- a/src/hotspot/os_cpu/linux_riscv/os_linux_riscv.cpp
+++ b/src/hotspot/os_cpu/linux_riscv/os_linux_riscv.cpp
@@ -27,7 +27,6 @@
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
@@ -39,6 +38,7 @@
#include "prims/jvm_misc.hpp"
#include "runtime/arguments.hpp"
#include "runtime/frame.inline.hpp"
+#include "runtime/globals.hpp"
#include "runtime/interfaceSupport.inline.hpp"
#include "runtime/java.hpp"
#include "runtime/javaCalls.hpp"
@@ -406,6 +406,14 @@ static inline void atomic_copy64(const volatile void *src, volatile void *dst) {
extern "C" {
int SpinPause() {
+ if (UseZihintpause) {
+ // PAUSE is encoded as a FENCE instruction with pred=W, succ=0, fm=0, rd=x0, and rs1=x0.
+ // To do: __asm__ volatile("pause " : : : );
+ // Since we're currently not passing '-march=..._zihintpause' to the compiler,
+ // it will not recognize the "pause" instruction, hence the hard-coded instruction.
+ __asm__ volatile(".word 0x0100000f " : : : );
+ return 1;
+ }
return 0;
}
diff --git a/src/hotspot/os_cpu/linux_riscv/riscv_hwprobe.cpp b/src/hotspot/os_cpu/linux_riscv/riscv_hwprobe.cpp
index 243c4b850ee..df4a2e347cc 100644
--- a/src/hotspot/os_cpu/linux_riscv/riscv_hwprobe.cpp
+++ b/src/hotspot/os_cpu/linux_riscv/riscv_hwprobe.cpp
@@ -49,6 +49,36 @@
#define RISCV_HWPROBE_EXT_ZBA (1 << 3)
#define RISCV_HWPROBE_EXT_ZBB (1 << 4)
#define RISCV_HWPROBE_EXT_ZBS (1 << 5)
+#define RISCV_HWPROBE_EXT_ZICBOZ (1 << 6)
+#define RISCV_HWPROBE_EXT_ZBC (1 << 7)
+#define RISCV_HWPROBE_EXT_ZBKB (1 << 8)
+#define RISCV_HWPROBE_EXT_ZBKC (1 << 9)
+#define RISCV_HWPROBE_EXT_ZBKX (1 << 10)
+#define RISCV_HWPROBE_EXT_ZKND (1 << 11)
+#define RISCV_HWPROBE_EXT_ZKNE (1 << 12)
+#define RISCV_HWPROBE_EXT_ZKNH (1 << 13)
+#define RISCV_HWPROBE_EXT_ZKSED (1 << 14)
+#define RISCV_HWPROBE_EXT_ZKSH (1 << 15)
+#define RISCV_HWPROBE_EXT_ZKT (1 << 16)
+#define RISCV_HWPROBE_EXT_ZVBB (1 << 17)
+#define RISCV_HWPROBE_EXT_ZVBC (1 << 18)
+#define RISCV_HWPROBE_EXT_ZVKB (1 << 19)
+#define RISCV_HWPROBE_EXT_ZVKG (1 << 20)
+#define RISCV_HWPROBE_EXT_ZVKNED (1 << 21)
+#define RISCV_HWPROBE_EXT_ZVKNHA (1 << 22)
+#define RISCV_HWPROBE_EXT_ZVKNHB (1 << 23)
+#define RISCV_HWPROBE_EXT_ZVKSED (1 << 24)
+#define RISCV_HWPROBE_EXT_ZVKSH (1 << 25)
+#define RISCV_HWPROBE_EXT_ZVKT (1 << 26)
+#define RISCV_HWPROBE_EXT_ZFH (1 << 27)
+#define RISCV_HWPROBE_EXT_ZFHMIN (1 << 28)
+#define RISCV_HWPROBE_EXT_ZIHINTNTL (1 << 29)
+#define RISCV_HWPROBE_EXT_ZVFH (1 << 30)
+#define RISCV_HWPROBE_EXT_ZVFHMIN (1 << 31)
+#define RISCV_HWPROBE_EXT_ZFA (1ULL << 32)
+#define RISCV_HWPROBE_EXT_ZTSO (1ULL << 33)
+#define RISCV_HWPROBE_EXT_ZACAS (1ULL << 34)
+#define RISCV_HWPROBE_EXT_ZICOND (1ULL << 35)
#define RISCV_HWPROBE_KEY_CPUPERF_0 5
#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)
@@ -145,6 +175,9 @@ void RiscvHwprobe::add_features_from_query_result() {
if (is_set(RISCV_HWPROBE_KEY_IMA_EXT_0, RISCV_HWPROBE_EXT_ZBS)) {
VM_Version::ext_Zbs.enable_feature();
}
+ if (is_set(RISCV_HWPROBE_KEY_IMA_EXT_0, RISCV_HWPROBE_EXT_ZFH)) {
+ VM_Version::ext_Zfh.enable_feature();
+ }
if (is_valid(RISCV_HWPROBE_KEY_CPUPERF_0)) {
VM_Version::unaligned_access.enable_feature(
query[RISCV_HWPROBE_KEY_CPUPERF_0].value & RISCV_HWPROBE_MISALIGNED_MASK);
diff --git a/src/hotspot/os_cpu/linux_riscv/vm_version_linux_riscv.cpp b/src/hotspot/os_cpu/linux_riscv/vm_version_linux_riscv.cpp
index 5230a06e43f..c7d61b33829 100644
--- a/src/hotspot/os_cpu/linux_riscv/vm_version_linux_riscv.cpp
+++ b/src/hotspot/os_cpu/linux_riscv/vm_version_linux_riscv.cpp
@@ -169,13 +169,13 @@ void VM_Version::os_aux_features() {
}
VM_Version::VM_MODE VM_Version::parse_satp_mode(const char* vm_mode) {
- if (!strcmp(vm_mode, "sv39")) {
+ if (!strncmp(vm_mode, "sv39", sizeof "sv39" - 1)) {
return VM_SV39;
- } else if (!strcmp(vm_mode, "sv48")) {
+ } else if (!strncmp(vm_mode, "sv48", sizeof "sv48" - 1)) {
return VM_SV48;
- } else if (!strcmp(vm_mode, "sv57")) {
+ } else if (!strncmp(vm_mode, "sv57", sizeof "sv57" - 1)) {
return VM_SV57;
- } else if (!strcmp(vm_mode, "sv64")) {
+ } else if (!strncmp(vm_mode, "sv64", sizeof "sv64" - 1)) {
return VM_SV64;
} else {
return VM_MBARE;
@@ -197,7 +197,7 @@ char* VM_Version::os_uarch_additional_features() {
if ((p = strchr(buf, ':')) != nullptr) {
if (mode == VM_NOTSET) {
if (strncmp(buf, "mmu", sizeof "mmu" - 1) == 0) {
- mode = VM_Version::parse_satp_mode(p);
+ mode = VM_Version::parse_satp_mode(p + 2);
}
}
if (ret == nullptr) {
@@ -244,6 +244,8 @@ void VM_Version::rivos_features() {
ext_Zfh.enable_feature();
+ ext_Zacas.enable_feature();
+ ext_Zicboz.enable_feature();
ext_Zicsr.enable_feature();
ext_Zifencei.enable_feature();
ext_Zic64b.enable_feature();
diff --git a/src/hotspot/os_cpu/linux_s390/os_linux_s390.cpp b/src/hotspot/os_cpu/linux_s390/os_linux_s390.cpp
index 033ea14ead6..5aa65e705d9 100644
--- a/src/hotspot/os_cpu/linux_s390/os_linux_s390.cpp
+++ b/src/hotspot/os_cpu/linux_s390/os_linux_s390.cpp
@@ -28,7 +28,6 @@
// no precompiled headers
#include "asm/assembler.inline.hpp"
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/nativeInst.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/disassembler.hpp"
diff --git a/src/hotspot/os_cpu/linux_x86/os_linux_x86.cpp b/src/hotspot/os_cpu/linux_x86/os_linux_x86.cpp
index b211330409d..4dcaedf71da 100644
--- a/src/hotspot/os_cpu/linux_x86/os_linux_x86.cpp
+++ b/src/hotspot/os_cpu/linux_x86/os_linux_x86.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -26,7 +26,6 @@
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
@@ -165,7 +164,7 @@ frame os::get_sender_for_C_frame(frame* fr) {
return frame(fr->sender_sp(), fr->link(), fr->sender_pc());
}
-intptr_t* _get_previous_fp() {
+static intptr_t* _get_previous_fp() {
#if defined(__clang__)
intptr_t **ebp;
__asm__ __volatile__ ("mov %%" SPELL_REG_FP ", %0":"=r"(ebp):);
diff --git a/src/hotspot/os_cpu/linux_zero/os_linux_zero.cpp b/src/hotspot/os_cpu/linux_zero/os_linux_zero.cpp
index 1ce73588524..d593c46d15d 100644
--- a/src/hotspot/os_cpu/linux_zero/os_linux_zero.cpp
+++ b/src/hotspot/os_cpu/linux_zero/os_linux_zero.cpp
@@ -27,7 +27,6 @@
#include "asm/assembler.inline.hpp"
#include "atomic_linux_zero.hpp"
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/os_cpu/windows_aarch64/os_windows_aarch64.cpp b/src/hotspot/os_cpu/windows_aarch64/os_windows_aarch64.cpp
index 46f718a9cd0..78e98609b6b 100644
--- a/src/hotspot/os_cpu/windows_aarch64/os_windows_aarch64.cpp
+++ b/src/hotspot/os_cpu/windows_aarch64/os_windows_aarch64.cpp
@@ -27,7 +27,6 @@
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "code/nativeInst.hpp"
#include "interpreter/interpreter.hpp"
diff --git a/src/hotspot/os_cpu/windows_x86/os_windows_x86.cpp b/src/hotspot/os_cpu/windows_x86/os_windows_x86.cpp
index 4e18334315a..7e0814c014b 100644
--- a/src/hotspot/os_cpu/windows_x86/os_windows_x86.cpp
+++ b/src/hotspot/os_cpu/windows_x86/os_windows_x86.cpp
@@ -25,7 +25,6 @@
// no precompiled headers
#include "asm/macroAssembler.hpp"
#include "classfile/vmSymbols.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "interpreter/interpreter.hpp"
#include "jvm.h"
diff --git a/src/hotspot/share/adlc/main.cpp b/src/hotspot/share/adlc/main.cpp
index 6d921cf5e7b..4d1c5491044 100644
--- a/src/hotspot/share/adlc/main.cpp
+++ b/src/hotspot/share/adlc/main.cpp
@@ -216,7 +216,6 @@ int main(int argc, char *argv[])
AD.addInclude(AD._CPP_file, "code/nativeInst.hpp");
AD.addInclude(AD._CPP_file, "code/vmreg.inline.hpp");
AD.addInclude(AD._CPP_file, "gc/shared/collectedHeap.inline.hpp");
- AD.addInclude(AD._CPP_file, "oops/compiledICHolder.hpp");
AD.addInclude(AD._CPP_file, "oops/compressedOops.hpp");
AD.addInclude(AD._CPP_file, "oops/markWord.hpp");
AD.addInclude(AD._CPP_file, "oops/method.hpp");
diff --git a/src/hotspot/share/asm/assembler.hpp b/src/hotspot/share/asm/assembler.hpp
index 7b7dbd4ede7..a533b963844 100644
--- a/src/hotspot/share/asm/assembler.hpp
+++ b/src/hotspot/share/asm/assembler.hpp
@@ -359,6 +359,7 @@ class AbstractAssembler : public ResourceObj {
}
static bool is_uimm12(uint64_t x) { return is_uimm(x, 12); }
+ static bool is_uimm32(uint64_t x) { return is_uimm(x, 32); }
// Accessors
CodeSection* code_section() const { return _code_section; }
diff --git a/src/hotspot/share/asm/codeBuffer.cpp b/src/hotspot/share/asm/codeBuffer.cpp
index 7a0a31abf59..5b1e113f15d 100644
--- a/src/hotspot/share/asm/codeBuffer.cpp
+++ b/src/hotspot/share/asm/codeBuffer.cpp
@@ -24,6 +24,7 @@
#include "precompiled.hpp"
#include "asm/codeBuffer.hpp"
+#include "code/compiledIC.hpp"
#include "code/oopRecorder.inline.hpp"
#include "compiler/disassembler.hpp"
#include "logging/log.hpp"
diff --git a/src/hotspot/share/asm/codeBuffer.hpp b/src/hotspot/share/asm/codeBuffer.hpp
index 9ef1ed1f37c..48476bedfe2 100644
--- a/src/hotspot/share/asm/codeBuffer.hpp
+++ b/src/hotspot/share/asm/codeBuffer.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -209,7 +209,7 @@ class CodeSection {
}
void set_locs_point(address pc) {
assert(pc >= locs_point(), "relocation addr may not decrease");
- assert(allocates2(pc), "relocation addr must be in this section");
+ assert(allocates2(pc), "relocation addr " INTPTR_FORMAT " must be in this section from " INTPTR_FORMAT " to " INTPTR_FORMAT, p2i(pc), p2i(_start), p2i(_limit));
_locs_point = pc;
}
diff --git a/src/hotspot/share/asm/codeBuffer.inline.hpp b/src/hotspot/share/asm/codeBuffer.inline.hpp
index 838447ad882..06ec9174b34 100644
--- a/src/hotspot/share/asm/codeBuffer.inline.hpp
+++ b/src/hotspot/share/asm/codeBuffer.inline.hpp
@@ -48,7 +48,7 @@ bool emit_shared_stubs_to_interp(CodeBuffer* cb, SharedStubToInterpRequests* sha
shared_stub_to_interp_requests->sort(by_shared_method);
MacroAssembler masm(cb);
for (int i = 0; i < shared_stub_to_interp_requests->length();) {
- address stub = __ start_a_stub(CompiledStaticCall::to_interp_stub_size());
+ address stub = __ start_a_stub(CompiledDirectCall::to_interp_stub_size());
if (stub == nullptr) {
return false;
}
diff --git a/src/hotspot/share/c1/c1_Canonicalizer.cpp b/src/hotspot/share/c1/c1_Canonicalizer.cpp
index dab11d4f70e..0dbf29300d1 100644
--- a/src/hotspot/share/c1/c1_Canonicalizer.cpp
+++ b/src/hotspot/share/c1/c1_Canonicalizer.cpp
@@ -469,9 +469,11 @@ void Canonicalizer::do_CompareOp (CompareOp* x) {
void Canonicalizer::do_IfOp(IfOp* x) {
- // Caution: do not use do_Op2(x) here for now since
- // we map the condition to the op for now!
- move_const_to_right(x);
+ // Currently, Canonicalizer is only used by GraphBuilder,
+ // and IfOp is not created by GraphBuilder but only later
+ // when eliminating conditional expressions with CE_Eliminator,
+ // so this method will not be called.
+ ShouldNotReachHere();
}
diff --git a/src/hotspot/share/c1/c1_GraphBuilder.cpp b/src/hotspot/share/c1/c1_GraphBuilder.cpp
index 6e5fb99242c..396c83c6ab9 100644
--- a/src/hotspot/share/c1/c1_GraphBuilder.cpp
+++ b/src/hotspot/share/c1/c1_GraphBuilder.cpp
@@ -523,7 +523,7 @@ inline bool BlockListBuilder::is_successor(BlockBegin* block, BlockBegin* sux) {
#ifndef PRODUCT
-int compare_depth_first(BlockBegin** a, BlockBegin** b) {
+static int compare_depth_first(BlockBegin** a, BlockBegin** b) {
return (*a)->depth_first_number() - (*b)->depth_first_number();
}
diff --git a/src/hotspot/share/c1/c1_LIRAssembler.cpp b/src/hotspot/share/c1/c1_LIRAssembler.cpp
index 1744aa44481..51fb851d00c 100644
--- a/src/hotspot/share/c1/c1_LIRAssembler.cpp
+++ b/src/hotspot/share/c1/c1_LIRAssembler.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2000, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -606,13 +606,14 @@ void LIR_Assembler::emit_op0(LIR_Op0* op) {
Unimplemented();
break;
- case lir_std_entry:
+ case lir_std_entry: {
// init offsets
offsets()->set_value(CodeOffsets::OSR_Entry, _masm->offset());
- _masm->align(CodeEntryAlignment);
if (needs_icache(compilation()->method())) {
- check_icache();
+ int offset = check_icache();
+ offsets()->set_value(CodeOffsets::Entry, offset);
}
+ _masm->align(CodeEntryAlignment);
offsets()->set_value(CodeOffsets::Verified_Entry, _masm->offset());
_masm->verified_entry(compilation()->directive()->BreakAtExecuteOption);
if (needs_clinit_barrier_on_entry(compilation()->method())) {
@@ -621,6 +622,7 @@ void LIR_Assembler::emit_op0(LIR_Op0* op) {
build_frame();
offsets()->set_value(CodeOffsets::Frame_Complete, _masm->offset());
break;
+ }
case lir_osr_entry:
offsets()->set_value(CodeOffsets::OSR_Entry, _masm->offset());
@@ -841,8 +843,6 @@ void LIR_Assembler::verify_oop_map(CodeEmitInfo* info) {
if (v.is_oop()) {
VMReg r = v.reg();
if (!r->is_stack()) {
- stringStream st;
- st.print("bad oop %s at %d", r->as_Register()->name(), _masm->offset());
_masm->verify_oop(r->as_Register());
} else {
_masm->verify_stack_oop(r->reg2stack() * VMRegImpl::stack_slot_size);
diff --git a/src/hotspot/share/c1/c1_LinearScan.cpp b/src/hotspot/share/c1/c1_LinearScan.cpp
index 9e9195a0d60..a4d955e52a0 100644
--- a/src/hotspot/share/c1/c1_LinearScan.cpp
+++ b/src/hotspot/share/c1/c1_LinearScan.cpp
@@ -1446,12 +1446,12 @@ int LinearScan::interval_cmp(Interval** a, Interval** b) {
}
}
-#ifndef PRODUCT
-int interval_cmp(Interval* const& l, Interval* const& r) {
+#ifdef ASSERT
+static int interval_cmp(Interval* const& l, Interval* const& r) {
return l->from() - r->from();
}
-bool find_interval(Interval* interval, IntervalArray* intervals) {
+static bool find_interval(Interval* interval, IntervalArray* intervals) {
bool found;
int idx = intervals->find_sorted(interval, found);
@@ -2303,11 +2303,11 @@ void assert_no_register_values(GrowableArray* values) {
}
}
-void assert_equal(Location l1, Location l2) {
+static void assert_equal(Location l1, Location l2) {
assert(l1.where() == l2.where() && l1.type() == l2.type() && l1.offset() == l2.offset(), "");
}
-void assert_equal(ScopeValue* v1, ScopeValue* v2) {
+static void assert_equal(ScopeValue* v1, ScopeValue* v2) {
if (v1->is_location()) {
assert(v2->is_location(), "");
assert_equal(((LocationValue*)v1)->location(), ((LocationValue*)v2)->location());
@@ -2328,12 +2328,12 @@ void assert_equal(ScopeValue* v1, ScopeValue* v2) {
}
}
-void assert_equal(MonitorValue* m1, MonitorValue* m2) {
+static void assert_equal(MonitorValue* m1, MonitorValue* m2) {
assert_equal(m1->owner(), m2->owner());
assert_equal(m1->basic_lock(), m2->basic_lock());
}
-void assert_equal(IRScopeDebugInfo* d1, IRScopeDebugInfo* d2) {
+static void assert_equal(IRScopeDebugInfo* d1, IRScopeDebugInfo* d2) {
assert(d1->scope() == d2->scope(), "not equal");
assert(d1->bci() == d2->bci(), "not equal");
@@ -2375,7 +2375,7 @@ void assert_equal(IRScopeDebugInfo* d1, IRScopeDebugInfo* d2) {
}
}
-void check_stack_depth(CodeEmitInfo* info, int stack_end) {
+static void check_stack_depth(CodeEmitInfo* info, int stack_end) {
if (info->stack()->bci() != SynchronizationEntryBCI && !info->scope()->method()->is_native()) {
Bytecodes::Code code = info->scope()->method()->java_code_at_bci(info->stack()->bci());
switch (code) {
diff --git a/src/hotspot/share/c1/c1_MacroAssembler.hpp b/src/hotspot/share/c1/c1_MacroAssembler.hpp
index 6a8304bd405..1e193ce0869 100644
--- a/src/hotspot/share/c1/c1_MacroAssembler.hpp
+++ b/src/hotspot/share/c1/c1_MacroAssembler.hpp
@@ -38,7 +38,6 @@ class C1_MacroAssembler: public MacroAssembler {
//----------------------------------------------------
void explicit_null_check(Register base);
- void inline_cache_check(Register receiver, Register iCache);
void build_frame(int frame_size_in_bytes, int bang_size_in_bytes);
void remove_frame(int frame_size_in_bytes);
diff --git a/src/hotspot/share/c1/c1_Optimizer.cpp b/src/hotspot/share/c1/c1_Optimizer.cpp
index 50009f9425a..dd428a5895b 100644
--- a/src/hotspot/share/c1/c1_Optimizer.cpp
+++ b/src/hotspot/share/c1/c1_Optimizer.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -335,7 +335,7 @@ void Optimizer::eliminate_conditional_expressions() {
}
// This removes others' relation to block, but doesn't empty block's lists
-void disconnect_from_graph(BlockBegin* block) {
+static void disconnect_from_graph(BlockBegin* block) {
for (int p = 0; p < block->number_of_preds(); p++) {
BlockBegin* pred = block->pred_at(p);
int idx;
@@ -714,6 +714,8 @@ class NullCheckEliminator: public ValueVisitor {
void handle_Phi (Phi* x);
void handle_ProfileCall (ProfileCall* x);
void handle_ProfileReturnType (ProfileReturnType* x);
+ void handle_Constant (Constant* x);
+ void handle_IfOp (IfOp* x);
};
@@ -728,7 +730,7 @@ class NullCheckEliminator: public ValueVisitor {
// that in for safety, otherwise should think more about it.
void NullCheckVisitor::do_Phi (Phi* x) { nce()->handle_Phi(x); }
void NullCheckVisitor::do_Local (Local* x) {}
-void NullCheckVisitor::do_Constant (Constant* x) { /* FIXME: handle object constants */ }
+void NullCheckVisitor::do_Constant (Constant* x) { nce()->handle_Constant(x); }
void NullCheckVisitor::do_LoadField (LoadField* x) { nce()->handle_AccessField(x); }
void NullCheckVisitor::do_StoreField (StoreField* x) { nce()->handle_AccessField(x); }
void NullCheckVisitor::do_ArrayLength (ArrayLength* x) { nce()->handle_ArrayLength(x); }
@@ -739,7 +741,7 @@ void NullCheckVisitor::do_ArithmeticOp (ArithmeticOp* x) { if (x->can_trap(
void NullCheckVisitor::do_ShiftOp (ShiftOp* x) {}
void NullCheckVisitor::do_LogicOp (LogicOp* x) {}
void NullCheckVisitor::do_CompareOp (CompareOp* x) {}
-void NullCheckVisitor::do_IfOp (IfOp* x) {}
+void NullCheckVisitor::do_IfOp (IfOp* x) { nce()->handle_IfOp(x); }
void NullCheckVisitor::do_Convert (Convert* x) {}
void NullCheckVisitor::do_NullCheck (NullCheck* x) { nce()->handle_NullCheck(x); }
void NullCheckVisitor::do_TypeCast (TypeCast* x) {}
@@ -882,7 +884,9 @@ void NullCheckEliminator::iterate_one(BlockBegin* block) {
// visiting instructions which are references in other blocks or
// visiting instructions more than once.
mark_visitable(instr);
- if (instr->is_pinned() || instr->can_trap() || (instr->as_NullCheck() != nullptr)) {
+ if (instr->is_pinned() || instr->can_trap() || (instr->as_NullCheck() != nullptr)
+ || (instr->as_Constant() != nullptr && instr->as_Constant()->type()->is_object())
+ || (instr->as_IfOp() != nullptr)) {
mark_visited(instr);
instr->input_values_do(this);
instr->visit(&_visitor);
@@ -1198,6 +1202,28 @@ void NullCheckEliminator::handle_ProfileReturnType(ProfileReturnType* x) {
x->set_needs_null_check(!set_contains(x->ret()));
}
+void NullCheckEliminator::handle_Constant(Constant *x) {
+ ObjectType* ot = x->type()->as_ObjectType();
+ if (ot != nullptr && ot->is_loaded()) {
+ ObjectConstant* oc = ot->as_ObjectConstant();
+ if (oc == nullptr || !oc->value()->is_null_object()) {
+ set_put(x);
+ if (PrintNullCheckElimination) {
+ tty->print_cr("Constant %d is non-null", x->id());
+ }
+ }
+ }
+}
+
+void NullCheckEliminator::handle_IfOp(IfOp *x) {
+ if (x->type()->is_object() && set_contains(x->tval()) && set_contains(x->fval())) {
+ set_put(x);
+ if (PrintNullCheckElimination) {
+ tty->print_cr("IfOp %d is non-null", x->id());
+ }
+ }
+}
+
void Optimizer::eliminate_null_checks() {
ResourceMark rm;
diff --git a/src/hotspot/share/cds/filemap.cpp b/src/hotspot/share/cds/filemap.cpp
index c2bfec3dfeb..f6e02594b5c 100644
--- a/src/hotspot/share/cds/filemap.cpp
+++ b/src/hotspot/share/cds/filemap.cpp
@@ -1664,9 +1664,9 @@ void FileMapInfo::close() {
/*
* Same as os::map_memory() but also pretouches if AlwaysPreTouch is enabled.
*/
-char* map_memory(int fd, const char* file_name, size_t file_offset,
- char *addr, size_t bytes, bool read_only,
- bool allow_exec, MEMFLAGS flags = mtNone) {
+static char* map_memory(int fd, const char* file_name, size_t file_offset,
+ char *addr, size_t bytes, bool read_only,
+ bool allow_exec, MEMFLAGS flags = mtNone) {
char* mem = os::map_memory(fd, file_name, file_offset, addr, bytes,
AlwaysPreTouch ? false : read_only,
allow_exec, flags);
@@ -2156,7 +2156,7 @@ bool FileMapInfo::map_heap_region_impl() {
if (_heap_pointers_need_patching) {
char* bitmap_base = map_bitmap_region();
- if (bitmap_base == NULL) {
+ if (bitmap_base == nullptr) {
log_info(cds)("CDS heap cannot be used because bitmap region cannot be mapped");
dealloc_heap_region();
unmap_region(MetaspaceShared::hp);
diff --git a/src/hotspot/share/ci/ciField.cpp b/src/hotspot/share/ci/ciField.cpp
index 0d4487fdf2f..0eddd87200a 100644
--- a/src/hotspot/share/ci/ciField.cpp
+++ b/src/hotspot/share/ci/ciField.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -35,7 +35,7 @@
#include "oops/oop.inline.hpp"
#include "runtime/fieldDescriptor.inline.hpp"
#include "runtime/handles.inline.hpp"
-#include "runtime/reflectionUtils.hpp"
+#include "runtime/reflection.hpp"
// ciField
//
diff --git a/src/hotspot/share/ci/ciMethodData.cpp b/src/hotspot/share/ci/ciMethodData.cpp
index dc7082c15ca..5abb342d031 100644
--- a/src/hotspot/share/ci/ciMethodData.cpp
+++ b/src/hotspot/share/ci/ciMethodData.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -31,6 +31,7 @@
#include "memory/allocation.inline.hpp"
#include "memory/resourceArea.hpp"
#include "oops/klass.inline.hpp"
+#include "oops/methodData.inline.hpp"
#include "runtime/deoptimization.hpp"
#include "utilities/copy.hpp"
@@ -87,8 +88,18 @@ class PrepareExtraDataClosure : public CleanExtraDataClosure {
// Preparation finished iff all Methods* were already cached.
return true;
}
- // Holding locks through safepoints is bad practice.
- MutexUnlocker mu(_mdo->extra_data_lock());
+ // We are currently holding the extra_data_lock and ensuring
+ // no safepoint breaks the lock.
+ _mdo->check_extra_data_locked();
+
+ // We now want to cache some method data. This could cause a safepoint.
+ // We temporarily release the lock and allow safepoints, and revert that
+ // at the end of the scope. This is safe, since we currently do not hold
+ // any extra_method_data: finish is called only after clean_extra_data,
+ // and the outer scope that first aquired the lock should not hold any
+ // extra_method_data while cleaning is performed, as the offsets can change.
+ MutexUnlocker mu(_mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
for (int i = 0; i < _uncached_methods.length(); ++i) {
if (has_safepointed()) {
// The metadata in the growable array might contain stale
@@ -123,7 +134,10 @@ void ciMethodData::prepare_metadata() {
void ciMethodData::load_remaining_extra_data() {
MethodData* mdo = get_MethodData();
- MutexLocker ml(mdo->extra_data_lock());
+
+ // Lock to read ProfileData, and ensure lock is not unintentionally broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
// Deferred metadata cleaning due to concurrent class unloading.
prepare_metadata();
// After metadata preparation, there is no stale metadata,
@@ -562,6 +576,9 @@ void ciMethodData::set_argument_type(int bci, int i, ciKlass* k) {
VM_ENTRY_MARK;
MethodData* mdo = get_MethodData();
if (mdo != nullptr) {
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
ProfileData* data = mdo->bci_to_data(bci);
if (data != nullptr) {
if (data->is_CallTypeData()) {
@@ -586,6 +603,9 @@ void ciMethodData::set_return_type(int bci, ciKlass* k) {
VM_ENTRY_MARK;
MethodData* mdo = get_MethodData();
if (mdo != nullptr) {
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
ProfileData* data = mdo->bci_to_data(bci);
if (data != nullptr) {
if (data->is_CallTypeData()) {
diff --git a/src/hotspot/share/classfile/altHashing.cpp b/src/hotspot/share/classfile/altHashing.cpp
index 158a8a232a7..1d43d6ebf1e 100644
--- a/src/hotspot/share/classfile/altHashing.cpp
+++ b/src/hotspot/share/classfile/altHashing.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -120,7 +120,7 @@ static void halfsiphash_init64(uint32_t v[4], uint64_t seed) {
v[1] ^= 0xee;
}
-uint32_t halfsiphash_finish32(uint32_t v[4], int rounds) {
+static uint32_t halfsiphash_finish32(uint32_t v[4], int rounds) {
v[2] ^= 0xff;
halfsiphash_rounds(v, rounds);
return (v[1] ^ v[3]);
diff --git a/src/hotspot/share/classfile/classLoader.cpp b/src/hotspot/share/classfile/classLoader.cpp
index 43aa82b67f8..9ce49b71734 100644
--- a/src/hotspot/share/classfile/classLoader.cpp
+++ b/src/hotspot/share/classfile/classLoader.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -134,7 +134,8 @@ ClassPathEntry* ClassLoader::_last_module_path_entry = nullptr;
#endif
// helper routines
-bool string_starts_with(const char* str, const char* str_to_find) {
+#if INCLUDE_CDS
+static bool string_starts_with(const char* str, const char* str_to_find) {
size_t str_len = strlen(str);
size_t str_to_find_len = strlen(str_to_find);
if (str_to_find_len > str_len) {
@@ -142,6 +143,7 @@ bool string_starts_with(const char* str, const char* str_to_find) {
}
return (strncmp(str, str_to_find, str_to_find_len) == 0);
}
+#endif
static const char* get_jimage_version_string() {
static char version_string[10] = "";
@@ -1009,8 +1011,8 @@ const char* ClassLoader::file_name_for_class_name(const char* class_name,
return file_name;
}
-ClassPathEntry* find_first_module_cpe(ModuleEntry* mod_entry,
- const GrowableArray* const module_list) {
+static ClassPathEntry* find_first_module_cpe(ModuleEntry* mod_entry,
+ const GrowableArray* const module_list) {
int num_of_entries = module_list->length();
const Symbol* class_module_name = mod_entry->name();
@@ -1355,7 +1357,7 @@ void ClassLoader::initialize(TRAPS) {
setup_bootstrap_search_path(THREAD);
}
-char* lookup_vm_resource(JImageFile *jimage, const char *jimage_version, const char *path) {
+static char* lookup_vm_resource(JImageFile *jimage, const char *jimage_version, const char *path) {
jlong size;
JImageLocationRef location = (*JImageFindResource)(jimage, "java.base", jimage_version, path, &size);
if (location == 0)
diff --git a/src/hotspot/share/classfile/classLoaderData.cpp b/src/hotspot/share/classfile/classLoaderData.cpp
index e9a72195ecf..0e0e6b77d70 100644
--- a/src/hotspot/share/classfile/classLoaderData.cpp
+++ b/src/hotspot/share/classfile/classLoaderData.cpp
@@ -843,10 +843,10 @@ OopHandle ClassLoaderData::add_handle(Handle h) {
void ClassLoaderData::remove_handle(OopHandle h) {
assert(!is_unloading(), "Do not remove a handle for a CLD that is unloading");
- oop* ptr = h.ptr_raw();
- if (ptr != nullptr) {
- assert(_handles.owner_of(ptr), "Got unexpected handle " PTR_FORMAT, p2i(ptr));
- NativeAccess<>::oop_store(ptr, oop(nullptr));
+ if (!h.is_empty()) {
+ assert(_handles.owner_of(h.ptr_raw()),
+ "Got unexpected handle " PTR_FORMAT, p2i(h.ptr_raw()));
+ h.replace(oop(nullptr));
}
}
@@ -1008,7 +1008,11 @@ void ClassLoaderData::print_on(outputStream* out) const {
_holder.print_on(out);
out->print_cr("");
}
- out->print_cr(" - class loader " INTPTR_FORMAT, p2i(_class_loader.ptr_raw()));
+ if (!_unloading) {
+ out->print_cr(" - class loader " INTPTR_FORMAT, p2i(_class_loader.peek()));
+ } else {
+ out->print_cr(" - class loader ");
+ }
out->print_cr(" - metaspace " INTPTR_FORMAT, p2i(_metaspace));
out->print_cr(" - unloading %s", _unloading ? "true" : "false");
out->print_cr(" - class mirror holder %s", _has_class_mirror_holder ? "true" : "false");
diff --git a/src/hotspot/share/classfile/loaderConstraints.cpp b/src/hotspot/share/classfile/loaderConstraints.cpp
index 4206fc10d4f..99d0c07ed42 100644
--- a/src/hotspot/share/classfile/loaderConstraints.cpp
+++ b/src/hotspot/share/classfile/loaderConstraints.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -296,8 +296,8 @@ void LoaderConstraintTable::purge_loader_constraints() {
_loader_constraint_table->unlink(&purge);
}
-void log_ldr_constraint_msg(Symbol* class_name, const char* reason,
- ClassLoaderData* loader1, ClassLoaderData* loader2) {
+static void log_ldr_constraint_msg(Symbol* class_name, const char* reason,
+ ClassLoaderData* loader1, ClassLoaderData* loader2) {
LogTarget(Info, class, loader, constraints) lt;
if (lt.is_enabled()) {
ResourceMark rm;
@@ -387,7 +387,7 @@ bool LoaderConstraintTable::add_entry(Symbol* class_name,
} else if (pp1 == nullptr) {
pp2->extend_loader_constraint(class_name, loader1, klass);
} else if (pp2 == nullptr) {
- pp1->extend_loader_constraint(class_name, loader1, klass);
+ pp1->extend_loader_constraint(class_name, loader2, klass);
} else {
merge_loader_constraints(class_name, pp1, pp2, klass);
}
diff --git a/src/hotspot/share/classfile/modules.cpp b/src/hotspot/share/classfile/modules.cpp
index bd4be93b868..4664d9565d3 100644
--- a/src/hotspot/share/classfile/modules.cpp
+++ b/src/hotspot/share/classfile/modules.cpp
@@ -1,5 +1,5 @@
/*
-* Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+* Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -259,7 +259,7 @@ static void define_javabase_module(Handle module_handle, jstring version, jstrin
}
// Caller needs ResourceMark.
-void throw_dup_pkg_exception(const char* module_name, PackageEntry* package, TRAPS) {
+static void throw_dup_pkg_exception(const char* module_name, PackageEntry* package, TRAPS) {
const char* package_name = package->name()->as_C_string();
if (package->module()->is_named()) {
THROW_MSG(vmSymbols::java_lang_IllegalStateException(),
diff --git a/src/hotspot/share/classfile/placeholders.cpp b/src/hotspot/share/classfile/placeholders.cpp
index 1bb5f878704..a6a86473ea7 100644
--- a/src/hotspot/share/classfile/placeholders.cpp
+++ b/src/hotspot/share/classfile/placeholders.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -198,8 +198,8 @@ void PlaceholderEntry::set_supername(Symbol* supername) {
// All threads examining the placeholder table must hold the
// SystemDictionary_lock, so we don't need special precautions
// on store ordering here.
-PlaceholderEntry* add_entry(Symbol* class_name, ClassLoaderData* loader_data,
- Symbol* supername){
+static PlaceholderEntry* add_entry(Symbol* class_name, ClassLoaderData* loader_data,
+ Symbol* supername){
assert_locked_or_safepoint(SystemDictionary_lock);
assert(class_name != nullptr, "adding nullptr obj");
@@ -213,7 +213,7 @@ PlaceholderEntry* add_entry(Symbol* class_name, ClassLoaderData* loader_data,
}
// Remove a placeholder object.
-void remove_entry(Symbol* class_name, ClassLoaderData* loader_data) {
+static void remove_entry(Symbol* class_name, ClassLoaderData* loader_data) {
assert_locked_or_safepoint(SystemDictionary_lock);
PlaceholderKey key(class_name, loader_data);
diff --git a/src/hotspot/share/classfile/stringTable.cpp b/src/hotspot/share/classfile/stringTable.cpp
index 9e96340d82b..be2971288ef 100644
--- a/src/hotspot/share/classfile/stringTable.cpp
+++ b/src/hotspot/share/classfile/stringTable.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -122,7 +122,7 @@ volatile bool _alt_hash = false;
static bool _rehashed = false;
static uint64_t _alt_hash_seed = 0;
-unsigned int hash_string(const jchar* s, int len, bool useAlt) {
+static unsigned int hash_string(const jchar* s, int len, bool useAlt) {
return useAlt ?
AltHashing::halfsiphash_32(_alt_hash_seed, s, len) :
java_lang_String::hash_code(s, len);
diff --git a/src/hotspot/share/classfile/systemDictionary.cpp b/src/hotspot/share/classfile/systemDictionary.cpp
index e635c825633..f2a88f00e0d 100644
--- a/src/hotspot/share/classfile/systemDictionary.cpp
+++ b/src/hotspot/share/classfile/systemDictionary.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -212,13 +212,13 @@ void SystemDictionary::set_platform_loader(ClassLoaderData *cld) {
// ----------------------------------------------------------------------------
// Parallel class loading check
-bool is_parallelCapable(Handle class_loader) {
+static bool is_parallelCapable(Handle class_loader) {
if (class_loader.is_null()) return true;
return java_lang_ClassLoader::parallelCapable(class_loader());
}
// ----------------------------------------------------------------------------
// ParallelDefineClass flag does not apply to bootclass loader
-bool is_parallelDefine(Handle class_loader) {
+static bool is_parallelDefine(Handle class_loader) {
if (class_loader.is_null()) return false;
if (AllowParallelDefineClass && java_lang_ClassLoader::parallelCapable(class_loader())) {
return true;
@@ -280,7 +280,7 @@ Symbol* SystemDictionary::class_name_symbol(const char* name, Symbol* exception,
#ifdef ASSERT
// Used to verify that class loading succeeded in adding k to the dictionary.
-void verify_dictionary_entry(Symbol* class_name, InstanceKlass* k) {
+static void verify_dictionary_entry(Symbol* class_name, InstanceKlass* k) {
MutexLocker mu(SystemDictionary_lock);
ClassLoaderData* loader_data = k->class_loader_data();
Dictionary* dictionary = loader_data->dictionary();
@@ -1111,14 +1111,13 @@ InstanceKlass* SystemDictionary::load_shared_lambda_proxy_class(InstanceKlass* i
if (loaded_ik != nullptr) {
assert(shared_nest_host->is_same_class_package(ik),
"lambda proxy class and its nest host must be in the same package");
+ // The lambda proxy class and its nest host have the same class loader and class loader data,
+ // as verified in SystemDictionaryShared::add_lambda_proxy_class()
+ assert(shared_nest_host->class_loader() == class_loader(), "mismatched class loader");
+ assert(shared_nest_host->class_loader_data() == class_loader_data(class_loader), "mismatched class loader data");
+ ik->set_nest_host(shared_nest_host);
}
- // The lambda proxy class and its nest host have the same class loader and class loader data,
- // as verified in SystemDictionaryShared::add_lambda_proxy_class()
- assert(shared_nest_host->class_loader() == class_loader(), "mismatched class loader");
- assert(shared_nest_host->class_loader_data() == class_loader_data(class_loader), "mismatched class loader data");
- ik->set_nest_host(shared_nest_host);
-
return loaded_ik;
}
diff --git a/src/hotspot/share/classfile/systemDictionaryShared.cpp b/src/hotspot/share/classfile/systemDictionaryShared.cpp
index 29748352684..44d7da5c4a4 100644
--- a/src/hotspot/share/classfile/systemDictionaryShared.cpp
+++ b/src/hotspot/share/classfile/systemDictionaryShared.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1348,7 +1348,7 @@ void SystemDictionaryShared::update_shared_entry(InstanceKlass* k, int id) {
info->_id = id;
}
-const char* class_loader_name_for_shared(Klass* k) {
+static const char* class_loader_name_for_shared(Klass* k) {
assert(k != nullptr, "Sanity");
assert(k->is_shared(), "Must be");
assert(k->is_instance_klass(), "Must be");
diff --git a/src/hotspot/share/classfile/vmIntrinsics.cpp b/src/hotspot/share/classfile/vmIntrinsics.cpp
index 6c2eecf7ae7..6e72b4cbac3 100644
--- a/src/hotspot/share/classfile/vmIntrinsics.cpp
+++ b/src/hotspot/share/classfile/vmIntrinsics.cpp
@@ -222,7 +222,6 @@ bool vmIntrinsics::disabled_by_jvm_flags(vmIntrinsics::ID id) {
case vmIntrinsics::_compareToLU:
case vmIntrinsics::_compareToUL:
case vmIntrinsics::_equalsL:
- case vmIntrinsics::_equalsU:
case vmIntrinsics::_equalsC:
case vmIntrinsics::_vectorizedHashCode:
case vmIntrinsics::_getCharStringU:
@@ -532,7 +531,6 @@ bool vmIntrinsics::disabled_by_jvm_flags(vmIntrinsics::ID id) {
if (!SpecialStringIndexOf) return true;
break;
case vmIntrinsics::_equalsL:
- case vmIntrinsics::_equalsU:
if (!SpecialStringEquals) return true;
break;
case vmIntrinsics::_vectorizedHashCode:
diff --git a/src/hotspot/share/classfile/vmIntrinsics.hpp b/src/hotspot/share/classfile/vmIntrinsics.hpp
index c9bc4acbeff..6104fb5683b 100644
--- a/src/hotspot/share/classfile/vmIntrinsics.hpp
+++ b/src/hotspot/share/classfile/vmIntrinsics.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -402,7 +402,6 @@ class methodHandle;
do_signature(indexOfI_signature, "([BI[BII)I") \
do_signature(indexOfChar_signature, "([BIII)I") \
do_intrinsic(_equalsL, java_lang_StringLatin1,equals_name, equalsB_signature, F_S) \
- do_intrinsic(_equalsU, java_lang_StringUTF16, equals_name, equalsB_signature, F_S) \
\
do_intrinsic(_isDigit, java_lang_CharacterDataLatin1, isDigit_name, int_bool_signature, F_R) \
do_name( isDigit_name, "isDigit") \
@@ -596,8 +595,8 @@ class methodHandle;
do_intrinsic(_notifyJvmtiVThreadEnd, java_lang_VirtualThread, notifyJvmtiEnd_name, void_method_signature, F_RN) \
do_intrinsic(_notifyJvmtiVThreadMount, java_lang_VirtualThread, notifyJvmtiMount_name, bool_void_signature, F_RN) \
do_intrinsic(_notifyJvmtiVThreadUnmount, java_lang_VirtualThread, notifyJvmtiUnmount_name, bool_void_signature, F_RN) \
- do_intrinsic(_notifyJvmtiVThreadHideFrames, java_lang_VirtualThread, notifyJvmtiHideFrames_name, bool_void_signature, F_RN) \
- do_intrinsic(_notifyJvmtiVThreadDisableSuspend, java_lang_VirtualThread, notifyJvmtiDisableSuspend_name, bool_void_signature, F_RN) \
+ do_intrinsic(_notifyJvmtiVThreadHideFrames, java_lang_VirtualThread, notifyJvmtiHideFrames_name, bool_void_signature, F_SN) \
+ do_intrinsic(_notifyJvmtiVThreadDisableSuspend, java_lang_VirtualThread, notifyJvmtiDisableSuspend_name, bool_void_signature, F_SN) \
\
/* support for UnsafeConstants */ \
do_class(jdk_internal_misc_UnsafeConstants, "jdk/internal/misc/UnsafeConstants") \
diff --git a/src/hotspot/share/code/codeBlob.cpp b/src/hotspot/share/code/codeBlob.cpp
index 5a2c9456468..d24e29c288d 100644
--- a/src/hotspot/share/code/codeBlob.cpp
+++ b/src/hotspot/share/code/codeBlob.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,7 +25,6 @@
#include "precompiled.hpp"
#include "code/codeBlob.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/relocInfo.hpp"
#include "code/vtableStubs.hpp"
#include "compiler/disassembler.hpp"
@@ -236,9 +235,9 @@ const ImmutableOopMap* CodeBlob::oop_map_for_return_address(address return_addre
return _oop_maps->find_map_at_offset((intptr_t) return_address - (intptr_t) code_begin());
}
-void CodeBlob::print_code() {
+void CodeBlob::print_code_on(outputStream* st) {
ResourceMark m;
- Disassembler::decode(this, tty);
+ Disassembler::decode(this, st);
}
//----------------------------------------------------------------------------------------------------
@@ -649,11 +648,6 @@ void CodeBlob::dump_for_addr(address addr, outputStream* st, bool verbose) const
st->print_cr(INTPTR_FORMAT " is pointing to an (unnamed) stub routine", p2i(addr));
return;
}
- // the InlineCacheBuffer is using stubs generated into a buffer blob
- if (InlineCacheBuffer::contains(addr)) {
- st->print_cr(INTPTR_FORMAT " is pointing into InlineCacheBuffer", p2i(addr));
- return;
- }
VtableStub* v = VtableStubs::stub_containing(addr);
if (v != nullptr) {
st->print_cr(INTPTR_FORMAT " is at entry_point+%d in a vtable stub", p2i(addr), (int)(addr - v->entry_point()));
diff --git a/src/hotspot/share/code/codeBlob.hpp b/src/hotspot/share/code/codeBlob.hpp
index 6aae1acd84a..56caa906ecb 100644
--- a/src/hotspot/share/code/codeBlob.hpp
+++ b/src/hotspot/share/code/codeBlob.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -240,7 +240,7 @@ class CodeBlob {
virtual void print_on(outputStream* st) const;
virtual void print_value_on(outputStream* st) const;
void dump_for_addr(address addr, outputStream* st, bool verbose) const;
- void print_code();
+ void print_code_on(outputStream* st);
// Print to stream, any comments associated with offset.
virtual void print_block_comment(outputStream* stream, address block_begin) const {
diff --git a/src/hotspot/share/code/codeCache.cpp b/src/hotspot/share/code/codeCache.cpp
index bf8c1d84e71..cdc9f3f50dc 100644
--- a/src/hotspot/share/code/codeCache.cpp
+++ b/src/hotspot/share/code/codeCache.cpp
@@ -29,7 +29,6 @@
#include "code/compiledIC.hpp"
#include "code/dependencies.hpp"
#include "code/dependencyContext.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "code/pcDesc.hpp"
#include "compiler/compilationPolicy.hpp"
@@ -913,23 +912,6 @@ void CodeCache::verify_clean_inline_caches() {
#endif
}
-void CodeCache::verify_icholder_relocations() {
-#ifdef ASSERT
- // make sure that we aren't leaking icholders
- int count = 0;
- FOR_ALL_HEAPS(heap) {
- FOR_ALL_BLOBS(cb, *heap) {
- CompiledMethod *nm = cb->as_compiled_method_or_null();
- if (nm != nullptr) {
- count += nm->verify_icholder_relocations();
- }
- }
- }
- assert(count + InlineCacheBuffer::pending_icholder_count() + CompiledICHolder::live_not_claimed_count() ==
- CompiledICHolder::live_count(), "must agree");
-#endif
-}
-
// Defer freeing of concurrently cleaned ExceptionCache entries until
// after a global handshake operation.
void CodeCache::release_exception_cache(ExceptionCache* entry) {
diff --git a/src/hotspot/share/code/codeCache.hpp b/src/hotspot/share/code/codeCache.hpp
index 103268c8ffc..d1c91727bf1 100644
--- a/src/hotspot/share/code/codeCache.hpp
+++ b/src/hotspot/share/code/codeCache.hpp
@@ -294,7 +294,6 @@ class CodeCache : AllStatic {
}
static void verify_clean_inline_caches();
- static void verify_icholder_relocations();
// Deoptimization
private:
diff --git a/src/hotspot/share/code/compiledIC.cpp b/src/hotspot/share/code/compiledIC.cpp
index c5063560ee5..250ef063a2a 100644
--- a/src/hotspot/share/code/compiledIC.cpp
+++ b/src/hotspot/share/code/compiledIC.cpp
@@ -26,27 +26,19 @@
#include "code/codeBehaviours.hpp"
#include "code/codeCache.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "code/vtableStubs.hpp"
-#include "interpreter/interpreter.hpp"
-#include "interpreter/linkResolver.hpp"
-#include "memory/metadataFactory.hpp"
-#include "memory/oopFactory.hpp"
#include "memory/resourceArea.hpp"
#include "memory/universe.hpp"
+#include "oops/compressedKlass.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
-#include "oops/oop.inline.hpp"
-#include "oops/symbol.hpp"
+#include "runtime/atomic.hpp"
#include "runtime/continuationEntry.hpp"
#include "runtime/handles.inline.hpp"
-#include "runtime/icache.hpp"
-#include "runtime/safepoint.hpp"
+#include "runtime/interfaceSupport.inline.hpp"
#include "runtime/sharedRuntime.hpp"
-#include "runtime/stubRoutines.hpp"
#include "sanitizers/leak.hpp"
-#include "utilities/events.hpp"
// Every time a compiled IC is changed or its type is being accessed,
@@ -75,191 +67,175 @@ bool CompiledICLocker::is_safe(address code) {
return CompiledICProtectionBehaviour::current()->is_safe(cm);
}
-//-----------------------------------------------------------------------------
-// Low-level access to an inline cache. Private, since they might not be
-// MT-safe to use.
+CompiledICData::CompiledICData()
+ : _speculated_method(),
+ _speculated_klass(),
+ _itable_defc_klass(),
+ _itable_refc_klass(),
+ _is_initialized() {}
-void* CompiledIC::cached_value() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- assert (!is_optimized(), "an optimized virtual call does not have a cached metadata");
-
- if (!is_in_transition_state()) {
- void* data = get_data();
- // If we let the metadata value here be initialized to zero...
- assert(data != nullptr || Universe::non_oop_word() == nullptr,
- "no raw nulls in CompiledIC metadatas, because of patching races");
- return (data == (void*)Universe::non_oop_word()) ? nullptr : data;
+// Inline cache callsite info is initialized once the first time it is resolved
+void CompiledICData::initialize(CallInfo* call_info, Klass* receiver_klass) {
+ _speculated_method = call_info->selected_method();
+ if (UseCompressedClassPointers) {
+ _speculated_klass = (uintptr_t)CompressedKlassPointers::encode_not_null(receiver_klass);
} else {
- return InlineCacheBuffer::cached_value_for((CompiledIC *)this);
+ _speculated_klass = (uintptr_t)receiver_klass;
}
+ if (call_info->call_kind() == CallInfo::itable_call) {
+ _itable_defc_klass = call_info->resolved_method()->method_holder();
+ _itable_refc_klass = call_info->resolved_klass();
+ }
+ _is_initialized = true;
}
+bool CompiledICData::is_speculated_klass_unloaded() const {
+ return is_initialized() && _speculated_klass == 0;
+}
-void CompiledIC::internal_set_ic_destination(address entry_point, bool is_icstub, void* cache, bool is_icholder) {
- assert(entry_point != nullptr, "must set legal entry point");
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- assert (!is_optimized() || cache == nullptr, "an optimized virtual call does not have a cached metadata");
- assert (cache == nullptr || cache != (Metadata*)badOopVal, "invalid metadata");
-
- assert(!is_icholder || is_icholder_entry(entry_point), "must be");
-
- // Don't use ic_destination for this test since that forwards
- // through ICBuffer instead of returning the actual current state of
- // the CompiledIC.
- if (is_icholder_entry(_call->destination())) {
- // When patching for the ICStub case the cached value isn't
- // overwritten until the ICStub copied into the CompiledIC during
- // the next safepoint. Make sure that the CompiledICHolder* is
- // marked for release at this point since it won't be identifiable
- // once the entry point is overwritten.
- InlineCacheBuffer::queue_for_release((CompiledICHolder*)get_data());
+void CompiledICData::clean_metadata() {
+ if (!is_initialized() || is_speculated_klass_unloaded()) {
+ return;
}
- if (TraceCompiledIC) {
- tty->print(" ");
- print_compiled_ic();
- tty->print(" changing destination to " INTPTR_FORMAT, p2i(entry_point));
- if (!is_optimized()) {
- tty->print(" changing cached %s to " INTPTR_FORMAT, is_icholder ? "icholder" : "metadata", p2i((address)cache));
- }
- if (is_icstub) {
- tty->print(" (icstub)");
- }
- tty->cr();
+ // GC cleaning doesn't need to change the state of the inline cache,
+ // only nuke stale speculated metadata if it gets unloaded. If the
+ // inline cache is monomorphic, the unverified entries will miss, and
+ // subsequent miss handlers will upgrade the callsite to megamorphic,
+ // which makes sense as it obviously is megamorphic then.
+ if (!speculated_klass()->is_loader_alive()) {
+ Atomic::store(&_speculated_klass, (uintptr_t)0);
+ Atomic::store(&_speculated_method, (Method*)nullptr);
}
-#ifdef ASSERT
- {
- CodeBlob* cb = CodeCache::find_blob(_call->instruction_address());
- assert(cb != nullptr && cb->is_compiled(), "must be compiled");
- }
-#endif
- _call->set_destination_mt_safe(entry_point);
+ assert(_speculated_method == nullptr || _speculated_method->method_holder()->is_loader_alive(),
+ "Speculated method is not unloaded despite class being unloaded");
+}
- if (is_optimized() || is_icstub) {
- // Optimized call sites don't have a cache value and ICStub call
- // sites only change the entry point. Changing the value in that
- // case could lead to MT safety issues.
- assert(cache == nullptr, "must be null");
+void CompiledICData::metadata_do(MetadataClosure* cl) {
+ if (!is_initialized()) {
return;
}
- if (cache == nullptr) cache = Universe::non_oop_word();
-
- set_data((intptr_t)cache);
-}
-
-
-void CompiledIC::set_ic_destination(ICStub* stub) {
- internal_set_ic_destination(stub->code_begin(), true, nullptr, false);
+ if (!is_speculated_klass_unloaded()) {
+ cl->do_metadata(_speculated_method);
+ cl->do_metadata(speculated_klass());
+ }
+ if (_itable_refc_klass != nullptr) {
+ cl->do_metadata(_itable_refc_klass);
+ }
+ if (_itable_defc_klass != nullptr) {
+ cl->do_metadata(_itable_defc_klass);
+ }
}
+Klass* CompiledICData::speculated_klass() const {
+ if (is_speculated_klass_unloaded()) {
+ return nullptr;
+ }
-
-address CompiledIC::ic_destination() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- if (!is_in_transition_state()) {
- return _call->destination();
+ if (UseCompressedClassPointers) {
+ return CompressedKlassPointers::decode_not_null((narrowKlass)_speculated_klass);
} else {
- return InlineCacheBuffer::ic_destination_for((CompiledIC *)this);
+ return (Klass*)_speculated_klass;
}
}
+//-----------------------------------------------------------------------------
+// High-level access to an inline cache. Guaranteed to be MT-safe.
-bool CompiledIC::is_in_transition_state() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- return InlineCacheBuffer::contains(_call->destination());;
+CompiledICData* CompiledIC::data() const {
+ return _data;
}
+CompiledICData* data_from_reloc_iter(RelocIterator* iter) {
+ assert(iter->type() == relocInfo::virtual_call_type, "wrong reloc. info");
+
+ virtual_call_Relocation* r = iter->virtual_call_reloc();
+ NativeMovConstReg* value = nativeMovConstReg_at(r->cached_value());
+
+ return (CompiledICData*)value->data();
+}
-bool CompiledIC::is_icholder_call() const {
+CompiledIC::CompiledIC(RelocIterator* iter)
+ : _method(iter->code()),
+ _data(data_from_reloc_iter(iter)),
+ _call(nativeCall_at(iter->addr()))
+{
+ assert(_method != nullptr, "must pass compiled method");
+ assert(_method->contains(iter->addr()), "must be in compiled method");
assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- return !_is_optimized && is_icholder_entry(ic_destination());
}
-// Returns native address of 'call' instruction in inline-cache. Used by
-// the InlineCacheBuffer when it needs to find the stub.
-address CompiledIC::stub_address() const {
- assert(is_in_transition_state(), "should only be called when we are in a transition state");
- return _call->destination();
+CompiledIC* CompiledIC_before(CompiledMethod* nm, address return_addr) {
+ address call_site = nativeCall_before(return_addr)->instruction_address();
+ return CompiledIC_at(nm, call_site);
}
-// Clears the IC stub if the compiled IC is in transition state
-void CompiledIC::clear_ic_stub() {
- if (is_in_transition_state()) {
- ICStub* stub = ICStub::from_destination_address(stub_address());
- stub->clear();
- }
+CompiledIC* CompiledIC_at(CompiledMethod* nm, address call_site) {
+ RelocIterator iter(nm, call_site, call_site + 1);
+ iter.next();
+ return CompiledIC_at(&iter);
}
-//-----------------------------------------------------------------------------
-// High-level access to an inline cache. Guaranteed to be MT-safe.
+CompiledIC* CompiledIC_at(Relocation* call_reloc) {
+ address call_site = call_reloc->addr();
+ CompiledMethod* cm = CodeCache::find_blob(call_reloc->addr())->as_compiled_method();
+ return CompiledIC_at(cm, call_site);
+}
-void CompiledIC::initialize_from_iter(RelocIterator* iter) {
- assert(iter->addr() == _call->instruction_address(), "must find ic_call");
+CompiledIC* CompiledIC_at(RelocIterator* reloc_iter) {
+ CompiledIC* c_ic = new CompiledIC(reloc_iter);
+ c_ic->verify();
+ return c_ic;
+}
- if (iter->type() == relocInfo::virtual_call_type) {
- virtual_call_Relocation* r = iter->virtual_call_reloc();
- _is_optimized = false;
- _value = _call->get_load_instruction(r);
- } else {
- assert(iter->type() == relocInfo::opt_virtual_call_type, "must be a virtual call");
- _is_optimized = true;
- _value = nullptr;
+void CompiledIC::ensure_initialized(CallInfo* call_info, Klass* receiver_klass) {
+ if (!_data->is_initialized()) {
+ _data->initialize(call_info, receiver_klass);
}
}
-CompiledIC::CompiledIC(CompiledMethod* cm, NativeCall* call)
- : _method(cm)
-{
- _call = _method->call_wrapper_at((address) call);
- address ic_call = _call->instruction_address();
-
- assert(ic_call != nullptr, "ic_call address must be set");
- assert(cm != nullptr, "must pass compiled method");
- assert(cm->contains(ic_call), "must be in compiled method");
-
- // Search for the ic_call at the given address.
- RelocIterator iter(cm, ic_call, ic_call+1);
- bool ret = iter.next();
- assert(ret == true, "relocInfo must exist at this address");
- assert(iter.addr() == ic_call, "must find ic_call");
-
- initialize_from_iter(&iter);
+void CompiledIC::set_to_clean() {
+ log_debug(inlinecache)("IC@" INTPTR_FORMAT ": set to clean", p2i(_call->instruction_address()));
+ _call->set_destination_mt_safe(SharedRuntime::get_resolve_virtual_call_stub());
}
-CompiledIC::CompiledIC(RelocIterator* iter)
- : _method(iter->code())
-{
- _call = _method->call_wrapper_at(iter->addr());
- address ic_call = _call->instruction_address();
+void CompiledIC::set_to_monomorphic() {
+ assert(data()->is_initialized(), "must be initialized");
+ Method* method = data()->speculated_method();
+ CompiledMethod* code = method->code();
+ address entry;
+ bool to_compiled = code != nullptr && code->is_in_use() && !code->is_unloading();
+
+ if (to_compiled) {
+ entry = code->entry_point();
+ } else {
+ entry = method->get_c2i_unverified_entry();
+ }
- CompiledMethod* nm = iter->code();
- assert(ic_call != nullptr, "ic_call address must be set");
- assert(nm != nullptr, "must pass compiled method");
- assert(nm->contains(ic_call), "must be in compiled method");
+ log_trace(inlinecache)("IC@" INTPTR_FORMAT ": monomorphic to %s: %s",
+ p2i(_call->instruction_address()),
+ to_compiled ? "compiled" : "interpreter",
+ method->print_value_string());
- initialize_from_iter(iter);
+ _call->set_destination_mt_safe(entry);
}
-// This function may fail for two reasons: either due to running out of vtable
-// stubs, or due to running out of IC stubs in an attempted transition to a
-// transitional state. The needs_ic_stub_refill value will be set if the failure
-// was due to running out of IC stubs, in which case the caller will refill IC
-// stubs and retry.
-bool CompiledIC::set_to_megamorphic(CallInfo* call_info, Bytecodes::Code bytecode,
- bool& needs_ic_stub_refill, TRAPS) {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- assert(!is_optimized(), "cannot set an optimized virtual call to megamorphic");
- assert(is_call_to_compiled() || is_call_to_interpreted(), "going directly to megamorphic?");
+void CompiledIC::set_to_megamorphic(CallInfo* call_info) {
+ assert(data()->is_initialized(), "must be initialized");
address entry;
- if (call_info->call_kind() == CallInfo::itable_call) {
- assert(bytecode == Bytecodes::_invokeinterface, "");
+ if (call_info->call_kind() == CallInfo::direct_call) {
+ // C1 sometimes compiles a callsite before the target method is loaded, resulting in
+ // dynamically bound callsites that should really be statically bound. However, the
+ // target method might not have a vtable or itable. We just wait for better code to arrive
+ return;
+ } else if (call_info->call_kind() == CallInfo::itable_call) {
int itable_index = call_info->itable_index();
entry = VtableStubs::find_itable_stub(itable_index);
if (entry == nullptr) {
- return false;
+ return;
}
#ifdef ASSERT
int index = call_info->resolved_method()->itable_index();
@@ -267,401 +243,151 @@ bool CompiledIC::set_to_megamorphic(CallInfo* call_info, Bytecodes::Code bytecod
InstanceKlass* k = call_info->resolved_method()->method_holder();
assert(k->verify_itable_index(itable_index), "sanity check");
#endif //ASSERT
- CompiledICHolder* holder = new CompiledICHolder(call_info->resolved_method()->method_holder(),
- call_info->resolved_klass(), false);
- holder->claim();
- if (!InlineCacheBuffer::create_transition_stub(this, holder, entry)) {
- delete holder;
- needs_ic_stub_refill = true;
- return false;
- }
- // LSan appears unable to follow malloc-based memory consistently when embedded as an immediate
- // in generated machine code. So we have to ignore it.
- LSAN_IGNORE_OBJECT(holder);
} else {
- assert(call_info->call_kind() == CallInfo::vtable_call, "either itable or vtable");
+ assert(call_info->call_kind() == CallInfo::vtable_call, "what else?");
// Can be different than selected_method->vtable_index(), due to package-private etc.
int vtable_index = call_info->vtable_index();
assert(call_info->resolved_klass()->verify_vtable_index(vtable_index), "sanity check");
entry = VtableStubs::find_vtable_stub(vtable_index);
if (entry == nullptr) {
- return false;
+ return;
}
- if (!InlineCacheBuffer::create_transition_stub(this, nullptr, entry)) {
- needs_ic_stub_refill = true;
- return false;
- }
- }
-
- {
- ResourceMark rm;
- assert(call_info->selected_method() != nullptr, "Unexpected null selected method");
- log_trace(inlinecache)("IC@" INTPTR_FORMAT ": to megamorphic %s entry: " INTPTR_FORMAT,
- p2i(instruction_address()), call_info->selected_method()->print_value_string(), p2i(entry));
}
- // We can't check this anymore. With lazy deopt we could have already
- // cleaned this IC entry before we even return. This is possible if
- // we ran out of space in the inline cache buffer trying to do the
- // set_next and we safepointed to free up space. This is a benign
- // race because the IC entry was complete when we safepointed so
- // cleaning it immediately is harmless.
- // assert(is_megamorphic(), "sanity check");
- return true;
-}
-
+ log_trace(inlinecache)("IC@" INTPTR_FORMAT ": to megamorphic %s entry: " INTPTR_FORMAT,
+ p2i(_call->instruction_address()), call_info->selected_method()->print_value_string(), p2i(entry));
-// true if destination is megamorphic stub
-bool CompiledIC::is_megamorphic() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- assert(!is_optimized(), "an optimized call cannot be megamorphic");
-
- // Cannot rely on cached_value. It is either an interface or a method.
- return VtableStubs::entry_point(ic_destination()) != nullptr;
+ _call->set_destination_mt_safe(entry);
+ assert(is_megamorphic(), "sanity check");
}
-bool CompiledIC::is_call_to_compiled() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
-
- CodeBlob* cb = CodeCache::find_blob(ic_destination());
- bool is_monomorphic = (cb != nullptr && cb->is_compiled());
- // Check that the cached_value is a klass for non-optimized monomorphic calls
- // This assertion is invalid for compiler1: a call that does not look optimized (no static stub) can be used
- // for calling directly to vep without using the inline cache (i.e., cached_value == nullptr).
- // For JVMCI this occurs because CHA is only used to improve inlining so call sites which could be optimized
- // virtuals because there are no currently loaded subclasses of a type are left as virtual call sites.
-#ifdef ASSERT
- CodeBlob* caller = CodeCache::find_blob(instruction_address());
- bool is_c1_or_jvmci_method = caller->is_compiled_by_c1() || caller->is_compiled_by_jvmci();
- assert( is_c1_or_jvmci_method ||
- !is_monomorphic ||
- is_optimized() ||
- (cached_metadata() != nullptr && cached_metadata()->is_klass()), "sanity check");
-#endif // ASSERT
- return is_monomorphic;
-}
+void CompiledIC::update(CallInfo* call_info, Klass* receiver_klass) {
+ // If this is the first time we fix the inline cache, we ensure it's initialized
+ ensure_initialized(call_info, receiver_klass);
+ if (is_megamorphic()) {
+ // Terminal state for the inline cache
+ return;
+ }
-bool CompiledIC::is_call_to_interpreted() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- // Call to interpreter if destination is either calling to a stub (if it
- // is optimized), or calling to an I2C blob
- bool is_call_to_interpreted = false;
- if (!is_optimized()) {
- CodeBlob* cb = CodeCache::find_blob(ic_destination());
- is_call_to_interpreted = (cb != nullptr && cb->is_adapter_blob());
- assert(!is_call_to_interpreted || (is_icholder_call() && cached_icholder() != nullptr), "sanity check");
+ if (is_speculated_klass(receiver_klass)) {
+ // If the speculated class matches the receiver klass, we can speculate that will
+ // continue to be the case with a monomorphic inline cache
+ set_to_monomorphic();
} else {
- // Check if we are calling into our own codeblob (i.e., to a stub)
- address dest = ic_destination();
-#ifdef ASSERT
- {
- _call->verify_resolve_call(dest);
- }
-#endif /* ASSERT */
- is_call_to_interpreted = _call->is_call_to_interpreted(dest);
+ // If the dynamic type speculation fails, we try to transform to a megamorphic state
+ // for the inline cache using stubs to dispatch in tables
+ set_to_megamorphic(call_info);
}
- return is_call_to_interpreted;
}
-bool CompiledIC::set_to_clean(bool in_use) {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- if (TraceInlineCacheClearing) {
- tty->print_cr("IC@" INTPTR_FORMAT ": set to clean", p2i(instruction_address()));
- print();
- }
- log_trace(inlinecache)("IC@" INTPTR_FORMAT ": set to clean", p2i(instruction_address()));
-
- address entry = _call->get_resolve_call_stub(is_optimized());
-
- bool safe_transition = _call->is_safe_for_patching() || !in_use || is_optimized() || SafepointSynchronize::is_at_safepoint();
+bool CompiledIC::is_clean() const {
+ return destination() == SharedRuntime::get_resolve_virtual_call_stub();
+}
- if (safe_transition) {
- // Kill any leftover stub we might have too
- clear_ic_stub();
- if (is_optimized()) {
- set_ic_destination(entry);
- } else {
- set_ic_destination_and_value(entry, (void*)nullptr);
- }
- } else {
- // Unsafe transition - create stub.
- if (!InlineCacheBuffer::create_transition_stub(this, nullptr, entry)) {
- return false;
- }
- }
- // We can't check this anymore. With lazy deopt we could have already
- // cleaned this IC entry before we even return. This is possible if
- // we ran out of space in the inline cache buffer trying to do the
- // set_next and we safepointed to free up space. This is a benign
- // race because the IC entry was complete when we safepointed so
- // cleaning it immediately is harmless.
- // assert(is_clean(), "sanity check");
- return true;
+bool CompiledIC::is_monomorphic() const {
+ return !is_clean() && !is_megamorphic();
}
-bool CompiledIC::is_clean() const {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- bool is_clean = false;
- address dest = ic_destination();
- is_clean = dest == _call->get_resolve_call_stub(is_optimized());
- assert(!is_clean || is_optimized() || cached_value() == nullptr, "sanity check");
- return is_clean;
+bool CompiledIC::is_megamorphic() const {
+ return VtableStubs::entry_point(destination()) != nullptr;;
}
-bool CompiledIC::set_to_monomorphic(CompiledICInfo& info) {
- assert(CompiledICLocker::is_safe(_method), "mt unsafe call");
- // Updating a cache to the wrong entry can cause bugs that are very hard
- // to track down - if cache entry gets invalid - we just clean it. In
- // this way it is always the same code path that is responsible for
- // updating and resolving an inline cache
- //
- // The above is no longer true. SharedRuntime::fixup_callers_callsite will change optimized
- // callsites. In addition ic_miss code will update a site to monomorphic if it determines
- // that an monomorphic call to the interpreter can now be monomorphic to compiled code.
- //
- // In both of these cases the only thing being modified is the jump/call target and these
- // transitions are mt_safe
-
- Thread *thread = Thread::current();
- if (info.to_interpreter()) {
- // Call to interpreter
- if (info.is_optimized() && is_optimized()) {
- assert(is_clean(), "unsafe IC path");
- // the call analysis (callee structure) specifies that the call is optimized
- // (either because of CHA or the static target is final)
- // At code generation time, this call has been emitted as static call
- // Call via stub
- assert(info.cached_metadata() != nullptr && info.cached_metadata()->is_method(), "sanity check");
- methodHandle method (thread, (Method*)info.cached_metadata());
- _call->set_to_interpreted(method, info);
-
- {
- ResourceMark rm(thread);
- log_trace(inlinecache)("IC@" INTPTR_FORMAT ": monomorphic to interpreter: %s",
- p2i(instruction_address()),
- method->print_value_string());
- }
- } else {
- // Call via method-klass-holder
- CompiledICHolder* holder = info.claim_cached_icholder();
- if (!InlineCacheBuffer::create_transition_stub(this, holder, info.entry())) {
- delete holder;
- return false;
- }
- // LSan appears unable to follow malloc-based memory consistently when embedded as an
- // immediate in generated machine code. So we have to ignore it.
- LSAN_IGNORE_OBJECT(holder);
- {
- ResourceMark rm(thread);
- log_trace(inlinecache)("IC@" INTPTR_FORMAT ": monomorphic to interpreter via icholder ", p2i(instruction_address()));
- }
- }
- } else {
- // Call to compiled code
- bool static_bound = info.is_optimized() || (info.cached_metadata() == nullptr);
-#ifdef ASSERT
- CodeBlob* cb = CodeCache::find_blob(info.entry());
- assert (cb != nullptr && cb->is_compiled(), "must be compiled!");
-#endif /* ASSERT */
-
- // This is MT safe if we come from a clean-cache and go through a
- // non-verified entry point
- bool safe = SafepointSynchronize::is_at_safepoint() ||
- (!is_in_transition_state() && (info.is_optimized() || static_bound || is_clean()));
-
- if (!safe) {
- if (!InlineCacheBuffer::create_transition_stub(this, info.cached_metadata(), info.entry())) {
- return false;
- }
- } else {
- if (is_optimized()) {
- set_ic_destination(info.entry());
- } else {
- set_ic_destination_and_value(info.entry(), info.cached_metadata());
- }
- }
+bool CompiledIC::is_speculated_klass(Klass* receiver_klass) {
+ return data()->speculated_klass() == receiver_klass;
+}
- {
- ResourceMark rm(thread);
- assert(info.cached_metadata() == nullptr || info.cached_metadata()->is_klass(), "must be");
- log_trace(inlinecache)("IC@" INTPTR_FORMAT ": monomorphic to compiled (rcvr klass = %s) %s",
- p2i(instruction_address()),
- (info.cached_metadata() != nullptr) ? ((Klass*)info.cached_metadata())->print_value_string() : "nullptr",
- (safe) ? "" : " via stub");
- }
- }
- // We can't check this anymore. With lazy deopt we could have already
- // cleaned this IC entry before we even return. This is possible if
- // we ran out of space in the inline cache buffer trying to do the
- // set_next and we safepointed to free up space. This is a benign
- // race because the IC entry was complete when we safepointed so
- // cleaning it immediately is harmless.
- // assert(is_call_to_compiled() || is_call_to_interpreted(), "sanity check");
- return true;
-}
-
-
-// is_optimized: Compiler has generated an optimized call (i.e. fixed, no inline cache)
-// static_bound: The call can be static bound. If it isn't also optimized, the property
-// wasn't provable at time of compilation. An optimized call will have any necessary
-// null check, while a static_bound won't. A static_bound (but not optimized) must
-// therefore use the unverified entry point.
-void CompiledIC::compute_monomorphic_entry(const methodHandle& method,
- Klass* receiver_klass,
- bool is_optimized,
- bool static_bound,
- bool caller_is_nmethod,
- CompiledICInfo& info,
- TRAPS) {
- CompiledMethod* method_code = method->code();
-
- address entry = nullptr;
- if (method_code != nullptr && method_code->is_in_use() && !method_code->is_unloading()) {
- assert(method_code->is_compiled(), "must be compiled");
- // Call to compiled code
- //
- // Note: the following problem exists with Compiler1:
- // - at compile time we may or may not know if the destination is final
- // - if we know that the destination is final (is_optimized), we will emit
- // an optimized virtual call (no inline cache), and need a Method* to make
- // a call to the interpreter
- // - if we don't know if the destination is final, we emit a standard
- // virtual call, and use CompiledICHolder to call interpreted code
- // (no static call stub has been generated)
- // - In the case that we here notice the call is static bound we
- // convert the call into what looks to be an optimized virtual call,
- // but we must use the unverified entry point (since there will be no
- // null check on a call when the target isn't loaded).
- // This causes problems when verifying the IC because
- // it looks vanilla but is optimized. Code in is_call_to_interpreted
- // is aware of this and weakens its asserts.
- if (is_optimized) {
- entry = method_code->verified_entry_point();
- } else {
- entry = method_code->entry_point();
- }
- }
- if (entry != nullptr) {
- // Call to near compiled code.
- info.set_compiled_entry(entry, is_optimized ? nullptr : receiver_klass, is_optimized);
- } else {
- if (is_optimized) {
- // Use stub entry
- info.set_interpreter_entry(method()->get_c2i_entry(), method());
- } else {
- // Use icholder entry
- assert(method_code == nullptr || method_code->is_compiled(), "must be compiled");
- CompiledICHolder* holder = new CompiledICHolder(method(), receiver_klass);
- info.set_icholder_entry(method()->get_c2i_unverified_entry(), holder);
- }
- }
- assert(info.is_optimized() == is_optimized, "must agree");
+// GC support
+void CompiledIC::clean_metadata() {
+ data()->clean_metadata();
}
+void CompiledIC::metadata_do(MetadataClosure* cl) {
+ data()->metadata_do(cl);
+}
-bool CompiledIC::is_icholder_entry(address entry) {
- CodeBlob* cb = CodeCache::find_blob(entry);
- if (cb == nullptr) {
- return false;
- }
- if (cb->is_adapter_blob()) {
- return true;
- } else if (cb->is_vtable_blob()) {
- return VtableStubs::is_icholder_entry(entry);
- }
- return false;
+#ifndef PRODUCT
+void CompiledIC::print() {
+ tty->print("Inline cache at " INTPTR_FORMAT ", calling " INTPTR_FORMAT " cached_value " INTPTR_FORMAT,
+ p2i(instruction_address()), p2i(destination()), p2i(data()));
+ tty->cr();
}
-bool CompiledIC::is_icholder_call_site(virtual_call_Relocation* call_site, const CompiledMethod* cm) {
- // This call site might have become stale so inspect it carefully.
- address dest = cm->call_wrapper_at(call_site->addr())->destination();
- return is_icholder_entry(dest);
+void CompiledIC::verify() {
+ _call->verify();
}
+#endif
// ----------------------------------------------------------------------------
-bool CompiledStaticCall::set_to_clean(bool in_use) {
+void CompiledDirectCall::set_to_clean() {
// in_use is unused but needed to match template function in CompiledMethod
assert(CompiledICLocker::is_safe(instruction_address()), "mt unsafe call");
// Reset call site
- set_destination_mt_safe(resolve_call_stub());
+ RelocIterator iter((nmethod*)nullptr, instruction_address(), instruction_address() + 1);
+ while (iter.next()) {
+ switch(iter.type()) {
+ case relocInfo::static_call_type:
+ _call->set_destination_mt_safe(SharedRuntime::get_resolve_static_call_stub());
+ break;
+ case relocInfo::opt_virtual_call_type:
+ _call->set_destination_mt_safe(SharedRuntime::get_resolve_opt_virtual_call_stub());
+ break;
+ default:
+ ShouldNotReachHere();
+ }
+ }
+ assert(is_clean(), "should be clean after cleaning");
- // Do not reset stub here: It is too expensive to call find_stub.
- // Instead, rely on caller (nmethod::clear_inline_caches) to clear
- // both the call and its stub.
- return true;
+ log_debug(inlinecache)("DC@" INTPTR_FORMAT ": set to clean", p2i(_call->instruction_address()));
}
-bool CompiledStaticCall::is_clean() const {
- return destination() == resolve_call_stub();
-}
+void CompiledDirectCall::set(const methodHandle& callee_method) {
+ CompiledMethod* code = callee_method->code();
+ CompiledMethod* caller = CodeCache::find_compiled(instruction_address());
-bool CompiledStaticCall::is_call_to_compiled() const {
- return CodeCache::contains(destination());
-}
+ bool to_interp_cont_enter = caller->method()->is_continuation_enter_intrinsic() &&
+ ContinuationEntry::is_interpreted_call(instruction_address());
-bool CompiledDirectStaticCall::is_call_to_interpreted() const {
- // It is a call to interpreted, if it calls to a stub. Hence, the destination
- // must be in the stub part of the nmethod that contains the call
- CompiledMethod* cm = CodeCache::find_compiled(instruction_address());
- return cm->stub_contains(destination());
-}
+ bool to_compiled = !to_interp_cont_enter && code != nullptr && code->is_in_use() && !code->is_unloading();
-void CompiledStaticCall::set_to_compiled(address entry) {
- {
- ResourceMark rm;
- log_trace(inlinecache)("%s@" INTPTR_FORMAT ": set_to_compiled " INTPTR_FORMAT,
- name(),
- p2i(instruction_address()),
- p2i(entry));
+ if (to_compiled) {
+ _call->set_destination_mt_safe(code->verified_entry_point());
+ assert(is_call_to_compiled(), "should be compiled after set to compiled");
+ } else {
+ // Patch call site to C2I adapter if code is deoptimized or unloaded.
+ // We also need to patch the static call stub to set the rmethod register
+ // to the callee_method so the c2i adapter knows how to build the frame
+ set_to_interpreted(callee_method, callee_method->get_c2i_entry());
+ assert(is_call_to_interpreted(), "should be interpreted after set to interpreted");
}
- // Call to compiled code
- assert(CodeCache::contains(entry), "wrong entry point");
- set_destination_mt_safe(entry);
+
+ log_trace(inlinecache)("DC@" INTPTR_FORMAT ": set to %s: %s: " INTPTR_FORMAT,
+ p2i(_call->instruction_address()),
+ to_compiled ? "compiled" : "interpreter",
+ callee_method->print_value_string(),
+ p2i(_call->destination()));
}
-void CompiledStaticCall::set(const StaticCallInfo& info) {
- assert(CompiledICLocker::is_safe(instruction_address()), "mt unsafe call");
- // Updating a cache to the wrong entry can cause bugs that are very hard
- // to track down - if cache entry gets invalid - we just clean it. In
- // this way it is always the same code path that is responsible for
- // updating and resolving an inline cache
- assert(is_clean(), "do not update a call entry - use clean");
-
- if (info._to_interpreter) {
- // Call to interpreted code
- set_to_interpreted(info.callee(), info.entry());
- } else {
- set_to_compiled(info.entry());
- }
+bool CompiledDirectCall::is_clean() const {
+ return destination() == SharedRuntime::get_resolve_static_call_stub() ||
+ destination() == SharedRuntime::get_resolve_opt_virtual_call_stub();
}
-// Compute settings for a CompiledStaticCall. Since we might have to set
-// the stub when calling to the interpreter, we need to return arguments.
-void CompiledStaticCall::compute_entry(const methodHandle& m, bool caller_is_nmethod, StaticCallInfo& info) {
- CompiledMethod* m_code = m->code();
- info._callee = m;
- if (m_code != nullptr && m_code->is_in_use() && !m_code->is_unloading()) {
- info._to_interpreter = false;
- info._entry = m_code->verified_entry_point();
- } else {
- // Callee is interpreted code. In any case entering the interpreter
- // puts a converter-frame on the stack to save arguments.
- assert(!m->is_method_handle_intrinsic(), "Compiled code should never call interpreter MH intrinsics");
- info._to_interpreter = true;
- info._entry = m()->get_c2i_entry();
- }
+bool CompiledDirectCall::is_call_to_interpreted() const {
+ // It is a call to interpreted, if it calls to a stub. Hence, the destination
+ // must be in the stub part of the nmethod that contains the call
+ CompiledMethod* cm = CodeCache::find_compiled(instruction_address());
+ return cm->stub_contains(destination());
}
-void CompiledStaticCall::compute_entry_for_continuation_entry(const methodHandle& m, StaticCallInfo& info) {
- if (ContinuationEntry::is_interpreted_call(instruction_address())) {
- info._to_interpreter = true;
- info._entry = m()->get_c2i_entry();
- }
+bool CompiledDirectCall::is_call_to_compiled() const {
+ CompiledMethod* caller = CodeCache::find_compiled(instruction_address());
+ CodeBlob* dest_cb = CodeCache::find_blob(destination());
+ return !caller->stub_contains(destination()) && dest_cb->is_compiled();
}
-address CompiledDirectStaticCall::find_stub_for(address instruction) {
+address CompiledDirectCall::find_stub_for(address instruction) {
// Find reloc. information containing this call-site
RelocIterator iter((nmethod*)nullptr, instruction);
while (iter.next()) {
@@ -673,8 +399,6 @@ address CompiledDirectStaticCall::find_stub_for(address instruction) {
// from the CompiledIC implementation
case relocInfo::opt_virtual_call_type:
return iter.opt_virtual_call_reloc()->static_stub();
- case relocInfo::poll_type:
- case relocInfo::poll_return_type: // A safepoint can't overlap a call.
default:
ShouldNotReachHere();
}
@@ -683,36 +407,13 @@ address CompiledDirectStaticCall::find_stub_for(address instruction) {
return nullptr;
}
-address CompiledDirectStaticCall::find_stub() {
- return CompiledDirectStaticCall::find_stub_for(instruction_address());
+address CompiledDirectCall::find_stub() {
+ return find_stub_for(instruction_address());
}
-address CompiledDirectStaticCall::resolve_call_stub() const {
- return SharedRuntime::get_resolve_static_call_stub();
-}
-
-//-----------------------------------------------------------------------------
-// Non-product mode code
#ifndef PRODUCT
-
-void CompiledIC::verify() {
- _call->verify();
- assert(is_clean() || is_call_to_compiled() || is_call_to_interpreted()
- || is_optimized() || is_megamorphic(), "sanity check");
-}
-
-void CompiledIC::print() {
- print_compiled_ic();
- tty->cr();
-}
-
-void CompiledIC::print_compiled_ic() {
- tty->print("Inline cache at " INTPTR_FORMAT ", calling %s " INTPTR_FORMAT " cached_value " INTPTR_FORMAT,
- p2i(instruction_address()), is_call_to_interpreted() ? "interpreted " : "", p2i(ic_destination()), p2i(is_optimized() ? nullptr : cached_value()));
-}
-
-void CompiledDirectStaticCall::print() {
- tty->print("static call at " INTPTR_FORMAT " -> ", p2i(instruction_address()));
+void CompiledDirectCall::print() {
+ tty->print("direct call at " INTPTR_FORMAT " to " INTPTR_FORMAT " -> ", p2i(instruction_address()), p2i(destination()));
if (is_clean()) {
tty->print("clean");
} else if (is_call_to_compiled()) {
@@ -723,9 +424,10 @@ void CompiledDirectStaticCall::print() {
tty->cr();
}
-void CompiledDirectStaticCall::verify_mt_safe(const methodHandle& callee, address entry,
- NativeMovConstReg* method_holder,
- NativeJump* jump) {
+void CompiledDirectCall::verify_mt_safe(const methodHandle& callee, address entry,
+ NativeMovConstReg* method_holder,
+ NativeJump* jump) {
+ _call->verify();
// A generated lambda form might be deleted from the Lambdaform
// cache in MethodTypeForm. If a jit compiled lambdaform method
// becomes not entrant and the cache access returns null, the new
@@ -743,4 +445,4 @@ void CompiledDirectStaticCall::verify_mt_safe(const methodHandle& callee, addres
|| old_method->is_old(), // may be race patching deoptimized nmethod due to redefinition.
"b) MT-unsafe modification of inline cache");
}
-#endif // !PRODUCT
+#endif
diff --git a/src/hotspot/share/code/compiledIC.hpp b/src/hotspot/share/code/compiledIC.hpp
index 17586fc57a0..321bf280ed4 100644
--- a/src/hotspot/share/code/compiledIC.hpp
+++ b/src/hotspot/share/code/compiledIC.hpp
@@ -27,42 +27,19 @@
#include "code/nativeInst.hpp"
#include "interpreter/linkResolver.hpp"
-#include "oops/compiledICHolder.hpp"
#include "runtime/safepointVerifiers.hpp"
//-----------------------------------------------------------------------------
// The CompiledIC represents a compiled inline cache.
//
-// In order to make patching of the inline cache MT-safe, we only allow the following
-// transitions (when not at a safepoint):
-//
-//
-// [1] --<-- Clean -->--- [1]
-// / (null) \
-// / \ /-<-\
-// / [2] \ / \
-// Interpreted ---------> Monomorphic | [3]
-// (CompiledICHolder*) (Klass*) |
-// \ / \ /
-// [4] \ / [4] \->-/
-// \->- Megamorphic -<-/
-// (CompiledICHolder*)
-//
-// The text in parentheses () refers to the value of the inline cache receiver (mov instruction)
-//
-// The numbers in square brackets refer to the kind of transition:
-// [1]: Initial fixup. Receiver it found from debug information
-// [2]: Compilation of a method
-// [3]: Recompilation of a method (note: only entry is changed. The Klass* must stay the same)
-// [4]: Inline cache miss. We go directly to megamorphic call.
-//
-// The class automatically inserts transition stubs (using the InlineCacheBuffer) when an MT-unsafe
-// transition is made to a stub.
+// It's safe to transition from any state to any state. Typically an inline cache starts
+// in the clean state, meaning it will resolve the call when called. Then it typically
+// transitions to monomorphic, assuming the first dynamic receiver will be the only one
+// observed. If that speculation fails, we transition to megamorphic.
//
class CompiledIC;
class CompiledICProtectionBehaviour;
class CompiledMethod;
-class ICStub;
class CompiledICLocker: public StackObj {
CompiledMethod* _method;
@@ -77,237 +54,105 @@ class CompiledICLocker: public StackObj {
static bool is_safe(address code);
};
-class CompiledICInfo : public StackObj {
- private:
- address _entry; // entry point for call
- void* _cached_value; // Value of cached_value (either in stub or inline cache)
- bool _is_icholder; // Is the cached value a CompiledICHolder*
- bool _is_optimized; // it is an optimized virtual call (i.e., can be statically bound)
- bool _to_interpreter; // Call it to interpreter
- bool _release_icholder;
+// A CompiledICData is a helper object for the inline cache implementation.
+// It comprises:
+// (1) The first receiver klass and its selected method
+// (2) Itable call metadata
+
+class CompiledICData : public CHeapObj {
+ friend class VMStructs;
+ friend class JVMCIVMStructs;
+
+ Method* volatile _speculated_method;
+ uintptr_t volatile _speculated_klass;
+ Klass* _itable_defc_klass;
+ Klass* _itable_refc_klass;
+ bool _is_initialized;
+
+ bool is_speculated_klass_unloaded() const;
+
public:
- address entry() const { return _entry; }
- Metadata* cached_metadata() const { assert(!_is_icholder, ""); return (Metadata*)_cached_value; }
- CompiledICHolder* claim_cached_icholder() {
- assert(_is_icholder, "");
- assert(_cached_value != nullptr, "must be non-null");
- _release_icholder = false;
- CompiledICHolder* icholder = (CompiledICHolder*)_cached_value;
- icholder->claim();
- return icholder;
- }
- bool is_optimized() const { return _is_optimized; }
- bool to_interpreter() const { return _to_interpreter; }
-
- void set_compiled_entry(address entry, Klass* klass, bool is_optimized) {
- _entry = entry;
- _cached_value = (void*)klass;
- _to_interpreter = false;
- _is_icholder = false;
- _is_optimized = is_optimized;
- _release_icholder = false;
- }
+ // Constructor
+ CompiledICData();
- void set_interpreter_entry(address entry, Method* method) {
- _entry = entry;
- _cached_value = (void*)method;
- _to_interpreter = true;
- _is_icholder = false;
- _is_optimized = true;
- _release_icholder = false;
- }
+ // accessors
+ Klass* speculated_klass() const;
+ Method* speculated_method() const { return _speculated_method; }
+ Klass* itable_defc_klass() const { return _itable_defc_klass; }
+ Klass* itable_refc_klass() const { return _itable_refc_klass; }
- void set_icholder_entry(address entry, CompiledICHolder* icholder) {
- _entry = entry;
- _cached_value = (void*)icholder;
- _to_interpreter = true;
- _is_icholder = true;
- _is_optimized = false;
- _release_icholder = true;
- }
+ static ByteSize speculated_method_offset() { return byte_offset_of(CompiledICData, _speculated_method); }
+ static ByteSize speculated_klass_offset() { return byte_offset_of(CompiledICData, _speculated_klass); }
- CompiledICInfo(): _entry(nullptr), _cached_value(nullptr), _is_icholder(false),
- _is_optimized(false), _to_interpreter(false), _release_icholder(false) {
- }
- ~CompiledICInfo() {
- // In rare cases the info is computed but not used, so release any
- // CompiledICHolder* that was created
- if (_release_icholder) {
- assert(_is_icholder, "must be");
- CompiledICHolder* icholder = (CompiledICHolder*)_cached_value;
- icholder->claim();
- delete icholder;
- }
- }
-};
+ static ByteSize itable_defc_klass_offset() { return byte_offset_of(CompiledICData, _itable_defc_klass); }
+ static ByteSize itable_refc_klass_offset() { return byte_offset_of(CompiledICData, _itable_refc_klass); }
-class NativeCallWrapper: public ResourceObj {
-public:
- virtual address destination() const = 0;
- virtual address instruction_address() const = 0;
- virtual address next_instruction_address() const = 0;
- virtual address return_address() const = 0;
- virtual address get_resolve_call_stub(bool is_optimized) const = 0;
- virtual void set_destination_mt_safe(address dest) = 0;
- virtual void set_to_interpreted(const methodHandle& method, CompiledICInfo& info) = 0;
- virtual void verify() const = 0;
- virtual void verify_resolve_call(address dest) const = 0;
-
- virtual bool is_call_to_interpreted(address dest) const = 0;
- virtual bool is_safe_for_patching() const = 0;
-
- virtual NativeInstruction* get_load_instruction(virtual_call_Relocation* r) const = 0;
-
- virtual void *get_data(NativeInstruction* instruction) const = 0;
- virtual void set_data(NativeInstruction* instruction, intptr_t data) = 0;
+ void initialize(CallInfo* call_info, Klass* receiver_klass);
+
+ bool is_initialized() const { return _is_initialized; }
+
+ // GC Support
+ void clean_metadata();
+ void metadata_do(MetadataClosure* cl);
};
class CompiledIC: public ResourceObj {
- friend class InlineCacheBuffer;
- friend class ICStub;
-
- private:
- NativeCallWrapper* _call;
- NativeInstruction* _value; // patchable value cell for this IC
- bool _is_optimized; // an optimized virtual call (i.e., no compiled IC)
+private:
CompiledMethod* _method;
+ CompiledICData* _data;
+ NativeCall* _call;
- CompiledIC(CompiledMethod* cm, NativeCall* ic_call);
CompiledIC(RelocIterator* iter);
- void initialize_from_iter(RelocIterator* iter);
+ // CompiledICData wrappers
+ void ensure_initialized(CallInfo* call_info, Klass* receiver_klass);
+ bool is_speculated_klass(Klass* receiver_klass);
- static bool is_icholder_entry(address entry);
+ // Inline cache states
+ void set_to_monomorphic();
+ void set_to_megamorphic(CallInfo* call_info);
- // low-level inline-cache manipulation. Cannot be accessed directly, since it might not be MT-safe
- // to change an inline-cache. These changes the underlying inline-cache directly. They *newer* make
- // changes to a transition stub.
- void internal_set_ic_destination(address entry_point, bool is_icstub, void* cache, bool is_icholder);
- void set_ic_destination(ICStub* stub);
- void set_ic_destination(address entry_point) {
- assert(_is_optimized, "use set_ic_destination_and_value instead");
- internal_set_ic_destination(entry_point, false, nullptr, false);
- }
- // This only for use by ICStubs where the type of the value isn't known
- void set_ic_destination_and_value(address entry_point, void* value) {
- internal_set_ic_destination(entry_point, false, value, is_icholder_entry(entry_point));
- }
- void set_ic_destination_and_value(address entry_point, Metadata* value) {
- internal_set_ic_destination(entry_point, false, value, false);
- }
- void set_ic_destination_and_value(address entry_point, CompiledICHolder* value) {
- internal_set_ic_destination(entry_point, false, value, true);
- }
-
- // Reads the location of the transition stub. This will fail with an assertion, if no transition stub is
- // associated with the inline cache.
- address stub_address() const;
- bool is_in_transition_state() const; // Use InlineCacheBuffer
-
- public:
+public:
// conversion (machine PC to CompiledIC*)
friend CompiledIC* CompiledIC_before(CompiledMethod* nm, address return_addr);
friend CompiledIC* CompiledIC_at(CompiledMethod* nm, address call_site);
friend CompiledIC* CompiledIC_at(Relocation* call_site);
friend CompiledIC* CompiledIC_at(RelocIterator* reloc_iter);
- static bool is_icholder_call_site(virtual_call_Relocation* call_site, const CompiledMethod* cm);
-
- // Return the cached_metadata/destination associated with this inline cache. If the cache currently points
- // to a transition stub, it will read the values from the transition stub.
- void* cached_value() const;
- CompiledICHolder* cached_icholder() const {
- assert(is_icholder_call(), "must be");
- return (CompiledICHolder*) cached_value();
- }
- Metadata* cached_metadata() const {
- assert(!is_icholder_call(), "must be");
- return (Metadata*) cached_value();
- }
-
- void* get_data() const {
- return _call->get_data(_value);
- }
-
- void set_data(intptr_t data) {
- _call->set_data(_value, data);
- }
-
- address ic_destination() const;
-
- bool is_optimized() const { return _is_optimized; }
+ CompiledICData* data() const;
// State
- bool is_clean() const;
+ bool is_clean() const;
+ bool is_monomorphic() const;
bool is_megamorphic() const;
- bool is_call_to_compiled() const;
- bool is_call_to_interpreted() const;
-
- bool is_icholder_call() const;
- address end_of_call() const { return _call->return_address(); }
+ address end_of_call() const { return _call->return_address(); }
- // MT-safe patching of inline caches. Note: Only safe to call is_xxx when holding the CompiledIC_ock
+ // MT-safe patching of inline caches. Note: Only safe to call is_xxx when holding the CompiledICLocker
// so you are guaranteed that no patching takes place. The same goes for verify.
- //
- // Note: We do not provide any direct access to the stub code, to prevent parts of the code
- // to manipulate the inline cache in MT-unsafe ways.
- //
- // They all takes a TRAP argument, since they can cause a GC if the inline-cache buffer is full.
- //
- bool set_to_clean(bool in_use = true);
- bool set_to_monomorphic(CompiledICInfo& info);
- void clear_ic_stub();
-
- // Returns true if successful and false otherwise. The call can fail if memory
- // allocation in the code cache fails, or ic stub refill is required.
- bool set_to_megamorphic(CallInfo* call_info, Bytecodes::Code bytecode, bool& needs_ic_stub_refill, TRAPS);
-
- static void compute_monomorphic_entry(const methodHandle& method, Klass* receiver_klass,
- bool is_optimized, bool static_bound, bool caller_is_nmethod,
- CompiledICInfo& info, TRAPS);
+ void set_to_clean();
+ void update(CallInfo* call_info, Klass* receiver_klass);
+
+ // GC support
+ void clean_metadata();
+ void metadata_do(MetadataClosure* cl);
// Location
address instruction_address() const { return _call->instruction_address(); }
+ address destination() const { return _call->destination(); }
// Misc
void print() PRODUCT_RETURN;
- void print_compiled_ic() PRODUCT_RETURN;
void verify() PRODUCT_RETURN;
};
-inline CompiledIC* CompiledIC_before(CompiledMethod* nm, address return_addr) {
- CompiledIC* c_ic = new CompiledIC(nm, nativeCall_before(return_addr));
- c_ic->verify();
- return c_ic;
-}
-
-inline CompiledIC* CompiledIC_at(CompiledMethod* nm, address call_site) {
- CompiledIC* c_ic = new CompiledIC(nm, nativeCall_at(call_site));
- c_ic->verify();
- return c_ic;
-}
-
-inline CompiledIC* CompiledIC_at(Relocation* call_site) {
- assert(call_site->type() == relocInfo::virtual_call_type ||
- call_site->type() == relocInfo::opt_virtual_call_type, "wrong reloc. info");
- CompiledIC* c_ic = new CompiledIC(call_site->code(), nativeCall_at(call_site->addr()));
- c_ic->verify();
- return c_ic;
-}
-
-inline CompiledIC* CompiledIC_at(RelocIterator* reloc_iter) {
- assert(reloc_iter->type() == relocInfo::virtual_call_type ||
- reloc_iter->type() == relocInfo::opt_virtual_call_type, "wrong reloc. info");
- CompiledIC* c_ic = new CompiledIC(reloc_iter);
- c_ic->verify();
- return c_ic;
-}
+CompiledIC* CompiledIC_before(CompiledMethod* nm, address return_addr);
+CompiledIC* CompiledIC_at(CompiledMethod* nm, address call_site);
+CompiledIC* CompiledIC_at(Relocation* call_site);
+CompiledIC* CompiledIC_at(RelocIterator* reloc_iter);
//-----------------------------------------------------------------------------
-// The CompiledStaticCall represents a call to a static method in the compiled
-//
-// Transition diagram of a static call site is somewhat simpler than for an inlined cache:
+// The CompiledDirectCall represents a call to a method in the compiled code
//
//
// -----<----- Clean ----->-----
@@ -321,63 +166,7 @@ inline CompiledIC* CompiledIC_at(RelocIterator* reloc_iter) {
//
//
-class StaticCallInfo {
- private:
- address _entry; // Entrypoint
- methodHandle _callee; // Callee (used when calling interpreter)
- bool _to_interpreter; // call to interpreted method (otherwise compiled)
-
- friend class CompiledStaticCall;
- friend class CompiledDirectStaticCall;
- friend class CompiledPltStaticCall;
- public:
- address entry() const { return _entry; }
- methodHandle callee() const { return _callee; }
-};
-
-class CompiledStaticCall : public ResourceObj {
- public:
- // Code
-
- // Returns null if CodeBuffer::expand fails
- static address emit_to_interp_stub(CodeBuffer &cbuf, address mark = nullptr);
- static int to_interp_stub_size();
- static int to_trampoline_stub_size();
- static int reloc_to_interp_stub();
-
- // Compute entry point given a method
- static void compute_entry(const methodHandle& m, bool caller_is_nmethod, StaticCallInfo& info);
- void compute_entry_for_continuation_entry(const methodHandle& m, StaticCallInfo& info);
-
-public:
- // Clean static call (will force resolving on next use)
- virtual address destination() const = 0;
-
- // Clean static call (will force resolving on next use)
- bool set_to_clean(bool in_use = true);
-
- // Set state. The entry must be the same, as computed by compute_entry.
- // Computation and setting is split up, since the actions are separate during
- // a OptoRuntime::resolve_xxx.
- void set(const StaticCallInfo& info);
-
- // State
- bool is_clean() const;
- bool is_call_to_compiled() const;
- virtual bool is_call_to_interpreted() const = 0;
-
- virtual address instruction_address() const = 0;
- virtual address end_of_call() const = 0;
-protected:
- virtual address resolve_call_stub() const = 0;
- virtual void set_destination_mt_safe(address dest) = 0;
- virtual void set_to_interpreted(const methodHandle& callee, address entry) = 0;
- virtual const char* name() const = 0;
-
- void set_to_compiled(address entry);
-};
-
-class CompiledDirectStaticCall : public CompiledStaticCall {
+class CompiledDirectCall : public ResourceObj {
private:
friend class CompiledIC;
friend class DirectNativeCallWrapper;
@@ -392,22 +181,28 @@ class CompiledDirectStaticCall : public CompiledStaticCall {
NativeCall* _call;
- CompiledDirectStaticCall(NativeCall* call) : _call(call) {}
+ CompiledDirectCall(NativeCall* call) : _call(call) {}
public:
- static inline CompiledDirectStaticCall* before(address return_addr) {
- CompiledDirectStaticCall* st = new CompiledDirectStaticCall(nativeCall_before(return_addr));
+ // Returns null if CodeBuffer::expand fails
+ static address emit_to_interp_stub(CodeBuffer &cbuf, address mark = nullptr);
+ static int to_interp_stub_size();
+ static int to_trampoline_stub_size();
+ static int reloc_to_interp_stub();
+
+ static inline CompiledDirectCall* before(address return_addr) {
+ CompiledDirectCall* st = new CompiledDirectCall(nativeCall_before(return_addr));
st->verify();
return st;
}
- static inline CompiledDirectStaticCall* at(address native_call) {
- CompiledDirectStaticCall* st = new CompiledDirectStaticCall(nativeCall_at(native_call));
+ static inline CompiledDirectCall* at(address native_call) {
+ CompiledDirectCall* st = new CompiledDirectCall(nativeCall_at(native_call));
st->verify();
return st;
}
- static inline CompiledDirectStaticCall* at(Relocation* call_site) {
+ static inline CompiledDirectCall* at(Relocation* call_site) {
return at(call_site->addr());
}
@@ -415,8 +210,15 @@ class CompiledDirectStaticCall : public CompiledStaticCall {
address destination() const { return _call->destination(); }
address end_of_call() const { return _call->return_address(); }
+ // Clean static call (will force resolving on next use)
+ void set_to_clean();
+
+ void set(const methodHandle& callee_method);
+
// State
- virtual bool is_call_to_interpreted() const;
+ bool is_clean() const;
+ bool is_call_to_interpreted() const;
+ bool is_call_to_compiled() const;
// Stub support
static address find_stub_for(address instruction);
@@ -426,10 +228,6 @@ class CompiledDirectStaticCall : public CompiledStaticCall {
// Misc.
void print() PRODUCT_RETURN;
void verify() PRODUCT_RETURN;
-
- protected:
- virtual address resolve_call_stub() const;
- virtual const char* name() const { return "CompiledDirectStaticCall"; }
};
#endif // SHARE_CODE_COMPILEDIC_HPP
diff --git a/src/hotspot/share/code/compiledMethod.cpp b/src/hotspot/share/code/compiledMethod.cpp
index 30b4e3617d6..6553d6f7934 100644
--- a/src/hotspot/share/code/compiledMethod.cpp
+++ b/src/hotspot/share/code/compiledMethod.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -28,7 +28,6 @@
#include "code/exceptionHandlerTable.hpp"
#include "code/scopeDesc.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetNMethod.hpp"
#include "gc/shared/gcBehaviours.hpp"
@@ -36,7 +35,6 @@
#include "logging/log.hpp"
#include "logging/logTag.hpp"
#include "memory/resourceArea.hpp"
-#include "oops/compiledICHolder.inline.hpp"
#include "oops/klass.inline.hpp"
#include "oops/methodData.hpp"
#include "oops/method.inline.hpp"
@@ -335,28 +333,6 @@ address CompiledMethod::oops_reloc_begin() const {
return low_boundary;
}
-int CompiledMethod::verify_icholder_relocations() {
- ResourceMark rm;
- int count = 0;
-
- RelocIterator iter(this);
- while(iter.next()) {
- if (iter.type() == relocInfo::virtual_call_type) {
- if (CompiledIC::is_icholder_call_site(iter.virtual_call_reloc(), this)) {
- CompiledIC *ic = CompiledIC_at(&iter);
- if (TraceCompiledIC) {
- tty->print("noticed icholder " INTPTR_FORMAT " ", p2i(ic->cached_icholder()));
- ic->print();
- }
- assert(ic->cached_icholder() != nullptr, "must be non-nullptr");
- count++;
- }
- }
- }
-
- return count;
-}
-
// Method that knows how to preserve outgoing arguments at call. This method must be
// called with a frame corresponding to a Java invoke
void CompiledMethod::preserve_callee_argument_oops(frame fr, const RegisterMap *reg_map, OopClosure* f) {
@@ -431,20 +407,6 @@ void CompiledMethod::clear_inline_caches() {
}
}
-// Clear IC callsites, releasing ICStubs of all compiled ICs
-// as well as any associated CompiledICHolders.
-void CompiledMethod::clear_ic_callsites() {
- assert(CompiledICLocker::is_safe(this), "mt unsafe call");
- ResourceMark rm;
- RelocIterator iter(this);
- while(iter.next()) {
- if (iter.type() == relocInfo::virtual_call_type) {
- CompiledIC* ic = CompiledIC_at(&iter);
- ic->set_to_clean(false);
- }
- }
-}
-
#ifdef ASSERT
// Check class_loader is alive for this bit of metadata.
class CheckClass : public MetadataClosure {
@@ -466,70 +428,22 @@ class CheckClass : public MetadataClosure {
#endif // ASSERT
-bool CompiledMethod::clean_ic_if_metadata_is_dead(CompiledIC *ic) {
- if (ic->is_clean()) {
- return true;
- }
- if (ic->is_icholder_call()) {
- // The only exception is compiledICHolder metadata which may
- // yet be marked below. (We check this further below).
- CompiledICHolder* cichk_metdata = ic->cached_icholder();
-
- if (cichk_metdata->is_loader_alive()) {
- return true;
- }
- } else {
- Metadata* ic_metdata = ic->cached_metadata();
- if (ic_metdata != nullptr) {
- if (ic_metdata->is_klass()) {
- if (((Klass*)ic_metdata)->is_loader_alive()) {
- return true;
- }
- } else if (ic_metdata->is_method()) {
- Method* method = (Method*)ic_metdata;
- assert(!method->is_old(), "old method should have been cleaned");
- if (method->method_holder()->is_loader_alive()) {
- return true;
- }
- } else {
- ShouldNotReachHere();
- }
- } else {
- // This inline cache is a megamorphic vtable call. Those ICs never hold
- // any Metadata and should therefore never be cleaned by this function.
- return true;
- }
- }
-
- return ic->set_to_clean();
+static void clean_ic_if_metadata_is_dead(CompiledIC *ic) {
+ ic->clean_metadata();
}
// Clean references to unloaded nmethods at addr from this one, which is not unloaded.
-template
-static bool clean_if_nmethod_is_unloaded(CompiledICorStaticCall *ic, address addr, CompiledMethod* from,
+template
+static void clean_if_nmethod_is_unloaded(CallsiteT* callsite, CompiledMethod* from,
bool clean_all) {
- CodeBlob *cb = CodeCache::find_blob(addr);
- CompiledMethod* nm = (cb != nullptr) ? cb->as_compiled_method_or_null() : nullptr;
- if (nm != nullptr) {
- // Clean inline caches pointing to bad nmethods
- if (clean_all || !nm->is_in_use() || nm->is_unloading() || (nm->method()->code() != nm)) {
- if (!ic->set_to_clean(!from->is_unloading())) {
- return false;
- }
- assert(ic->is_clean(), "nmethod " PTR_FORMAT "not clean %s", p2i(from), from->method()->name_and_sig_as_C_string());
- }
+ CodeBlob* cb = CodeCache::find_blob(callsite->destination());
+ if (!cb->is_compiled()) {
+ return;
+ }
+ CompiledMethod* cm = cb->as_compiled_method();
+ if (clean_all || !cm->is_in_use() || cm->is_unloading() || cm->method()->code() != cm) {
+ callsite->set_to_clean();
}
- return true;
-}
-
-static bool clean_if_nmethod_is_unloaded(CompiledIC *ic, CompiledMethod* from,
- bool clean_all) {
- return clean_if_nmethod_is_unloaded(ic, ic->ic_destination(), from, clean_all);
-}
-
-static bool clean_if_nmethod_is_unloaded(CompiledStaticCall *csc, CompiledMethod* from,
- bool clean_all) {
- return clean_if_nmethod_is_unloaded(csc, csc->destination(), from, clean_all);
}
// Cleans caches in nmethods that point to either classes that are unloaded
@@ -539,7 +453,7 @@ static bool clean_if_nmethod_is_unloaded(CompiledStaticCall *csc, CompiledMethod
// nmethods are unloaded. Return postponed=true in the parallel case for
// inline caches found that point to nmethods that are not yet visited during
// the do_unloading walk.
-bool CompiledMethod::unload_nmethod_caches(bool unloading_occurred) {
+void CompiledMethod::unload_nmethod_caches(bool unloading_occurred) {
ResourceMark rm;
// Exception cache only needs to be called if unloading occurred
@@ -547,16 +461,13 @@ bool CompiledMethod::unload_nmethod_caches(bool unloading_occurred) {
clean_exception_cache();
}
- if (!cleanup_inline_caches_impl(unloading_occurred, false)) {
- return false;
- }
+ cleanup_inline_caches_impl(unloading_occurred, false);
#ifdef ASSERT
// Check that the metadata embedded in the nmethod is alive
CheckClass check_class;
metadata_do(&check_class);
#endif
- return true;
}
void CompiledMethod::run_nmethod_entry_barrier() {
@@ -578,8 +489,7 @@ void CompiledMethod::run_nmethod_entry_barrier() {
void CompiledMethod::cleanup_inline_caches_whitebox() {
assert_locked_or_safepoint(CodeCache_lock);
CompiledICLocker ic_locker(this);
- guarantee(cleanup_inline_caches_impl(false /* unloading_occurred */, true /* clean_all */),
- "Inline cache cleaning in a safepoint can't fail");
+ cleanup_inline_caches_impl(false /* unloading_occurred */, true /* clean_all */);
}
address* CompiledMethod::orig_pc_addr(const frame* fr) {
@@ -587,7 +497,7 @@ address* CompiledMethod::orig_pc_addr(const frame* fr) {
}
// Called to clean up after class unloading for live nmethods
-bool CompiledMethod::cleanup_inline_caches_impl(bool unloading_occurred, bool clean_all) {
+void CompiledMethod::cleanup_inline_caches_impl(bool unloading_occurred, bool clean_all) {
assert(CompiledICLocker::is_safe(this), "mt unsafe call");
ResourceMark rm;
@@ -602,26 +512,15 @@ bool CompiledMethod::cleanup_inline_caches_impl(bool unloading_occurred, bool cl
if (unloading_occurred) {
// If class unloading occurred we first clear ICs where the cached metadata
// is referring to an unloaded klass or method.
- if (!clean_ic_if_metadata_is_dead(CompiledIC_at(&iter))) {
- return false;
- }
+ clean_ic_if_metadata_is_dead(CompiledIC_at(&iter));
}
- if (!clean_if_nmethod_is_unloaded(CompiledIC_at(&iter), this, clean_all)) {
- return false;
- }
+ clean_if_nmethod_is_unloaded(CompiledIC_at(&iter), this, clean_all);
break;
case relocInfo::opt_virtual_call_type:
- if (!clean_if_nmethod_is_unloaded(CompiledIC_at(&iter), this, clean_all)) {
- return false;
- }
- break;
-
case relocInfo::static_call_type:
- if (!clean_if_nmethod_is_unloaded(compiledStaticCall_at(iter.reloc()), this, clean_all)) {
- return false;
- }
+ clean_if_nmethod_is_unloaded(CompiledDirectCall::at(iter.reloc()), this, clean_all);
break;
case relocInfo::static_stub_type: {
@@ -672,8 +571,6 @@ bool CompiledMethod::cleanup_inline_caches_impl(bool unloading_occurred, bool cl
break;
}
}
-
- return true;
}
address CompiledMethod::continuation_for_implicit_exception(address pc, bool for_div0_check) {
@@ -687,12 +584,15 @@ address CompiledMethod::continuation_for_implicit_exception(address pc, bool for
ResourceMark rm(thread);
CodeBlob* cb = CodeCache::find_blob(pc);
assert(cb != nullptr && cb == this, "");
- ttyLocker ttyl;
- tty->print_cr("implicit exception happened at " INTPTR_FORMAT, p2i(pc));
- print();
- method()->print_codes();
- print_code();
- print_pcs();
+
+ // Keep tty output consistent. To avoid ttyLocker, we buffer in stream, and print all at once.
+ stringStream ss;
+ ss.print_cr("implicit exception happened at " INTPTR_FORMAT, p2i(pc));
+ print_on(&ss);
+ method()->print_codes_on(&ss);
+ print_code_on(&ss);
+ print_pcs_on(&ss);
+ tty->print("%s", ss.as_string()); // print all at once
}
#endif
if (cont_offset == 0) {
diff --git a/src/hotspot/share/code/compiledMethod.hpp b/src/hotspot/share/code/compiledMethod.hpp
index 6973911c063..42d68bda554 100644
--- a/src/hotspot/share/code/compiledMethod.hpp
+++ b/src/hotspot/share/code/compiledMethod.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -35,7 +35,7 @@ class ExceptionHandlerTable;
class ImplicitExceptionTable;
class AbstractCompiler;
class xmlStream;
-class CompiledStaticCall;
+class CompiledDirectCall;
class NativeCallWrapper;
class ScopeDesc;
class CompiledIC;
@@ -227,7 +227,7 @@ class CompiledMethod : public CodeBlob {
virtual bool is_osr_method() const = 0;
virtual int osr_entry_bci() const = 0;
Method* method() const { return _method; }
- virtual void print_pcs() = 0;
+ virtual void print_pcs_on(outputStream* st) = 0;
bool is_native_method() const { return _method != nullptr && _method->is_native(); }
bool is_java_method() const { return _method != nullptr && !_method->is_native(); }
@@ -364,7 +364,7 @@ class CompiledMethod : public CodeBlob {
// Inline cache support for class unloading and nmethod unloading
private:
- bool cleanup_inline_caches_impl(bool unloading_occurred, bool clean_all);
+ void cleanup_inline_caches_impl(bool unloading_occurred, bool clean_all);
address continuation_for_implicit_exception(address pc, bool for_div0_check);
@@ -373,13 +373,10 @@ class CompiledMethod : public CodeBlob {
void cleanup_inline_caches_whitebox();
virtual void clear_inline_caches();
- void clear_ic_callsites();
// Execute nmethod barrier code, as if entering through nmethod call.
void run_nmethod_entry_barrier();
- // Verify and count cached icholder relocations.
- int verify_icholder_relocations();
void verify_oop_relocations();
bool has_evol_metadata();
@@ -389,14 +386,8 @@ class CompiledMethod : public CodeBlob {
// corresponds to the given method as well.
virtual bool is_dependent_on_method(Method* dependee) = 0;
- virtual NativeCallWrapper* call_wrapper_at(address call) const = 0;
- virtual NativeCallWrapper* call_wrapper_before(address return_pc) const = 0;
virtual address call_instruction_address(address pc) const = 0;
- virtual CompiledStaticCall* compiledStaticCall_at(Relocation* call_site) const = 0;
- virtual CompiledStaticCall* compiledStaticCall_at(address addr) const = 0;
- virtual CompiledStaticCall* compiledStaticCall_before(address addr) const = 0;
-
Method* attached_method(address call_pc);
Method* attached_method_before_pc(address pc);
@@ -406,16 +397,13 @@ class CompiledMethod : public CodeBlob {
protected:
address oops_reloc_begin() const;
- private:
- bool static clean_ic_if_metadata_is_dead(CompiledIC *ic);
-
public:
// GC unloading support
// Cleans unloaded klasses and unloaded nmethods in inline caches
virtual bool is_unloading() = 0;
- bool unload_nmethod_caches(bool class_unloading_occurred);
+ void unload_nmethod_caches(bool class_unloading_occurred);
virtual void do_unloading(bool unloading_occurred) = 0;
private:
diff --git a/src/hotspot/share/code/icBuffer.cpp b/src/hotspot/share/code/icBuffer.cpp
deleted file mode 100644
index ec489eff9c8..00000000000
--- a/src/hotspot/share/code/icBuffer.cpp
+++ /dev/null
@@ -1,293 +0,0 @@
-/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "code/codeCache.hpp"
-#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
-#include "code/nmethod.hpp"
-#include "code/scopeDesc.hpp"
-#include "gc/shared/collectedHeap.inline.hpp"
-#include "interpreter/interpreter.hpp"
-#include "interpreter/linkResolver.hpp"
-#include "memory/resourceArea.hpp"
-#include "oops/method.hpp"
-#include "oops/oop.inline.hpp"
-#include "runtime/atomic.hpp"
-#include "runtime/handles.inline.hpp"
-#include "runtime/javaThread.hpp"
-#include "runtime/mutexLocker.hpp"
-#include "runtime/stubRoutines.hpp"
-#include "runtime/vmOperations.hpp"
-
-DEF_STUB_INTERFACE(ICStub);
-
-StubQueue* InlineCacheBuffer::_buffer = nullptr;
-
-CompiledICHolder* volatile InlineCacheBuffer::_pending_released = nullptr;
-volatile int InlineCacheBuffer::_pending_count = 0;
-
-#ifdef ASSERT
-ICRefillVerifier::ICRefillVerifier()
- : _refill_requested(false),
- _refill_remembered(false)
-{
- Thread* thread = Thread::current();
- assert(thread->missed_ic_stub_refill_verifier() == nullptr, "nesting not supported");
- thread->set_missed_ic_stub_refill_verifier(this);
-}
-
-ICRefillVerifier::~ICRefillVerifier() {
- assert(!_refill_requested || _refill_remembered,
- "Forgot to refill IC stubs after failed IC transition");
- Thread::current()->set_missed_ic_stub_refill_verifier(nullptr);
-}
-
-ICRefillVerifierMark::ICRefillVerifierMark(ICRefillVerifier* verifier) {
- Thread* thread = Thread::current();
- assert(thread->missed_ic_stub_refill_verifier() == nullptr, "nesting not supported");
- thread->set_missed_ic_stub_refill_verifier(verifier);
-}
-
-ICRefillVerifierMark::~ICRefillVerifierMark() {
- Thread::current()->set_missed_ic_stub_refill_verifier(nullptr);
-}
-
-static ICRefillVerifier* current_ic_refill_verifier() {
- Thread* current = Thread::current();
- ICRefillVerifier* verifier = current->missed_ic_stub_refill_verifier();
- assert(verifier != nullptr, "need a verifier for safety");
- return verifier;
-}
-#endif
-
-void ICStub::finalize() {
- if (!is_empty()) {
- ResourceMark rm;
- CompiledIC *ic = CompiledIC_at(CodeCache::find_compiled(ic_site()), ic_site());
- assert(CodeCache::find_compiled(ic->instruction_address()) != nullptr, "inline cache in non-compiled?");
-
- assert(this == ICStub::from_destination_address(ic->stub_address()), "wrong owner of ic buffer");
- ic->set_ic_destination_and_value(destination(), cached_value());
- }
-}
-
-
-address ICStub::destination() const {
- return InlineCacheBuffer::ic_buffer_entry_point(code_begin());
-}
-
-void* ICStub::cached_value() const {
- return InlineCacheBuffer::ic_buffer_cached_value(code_begin());
-}
-
-
-void ICStub::set_stub(CompiledIC *ic, void* cached_val, address dest_addr) {
- // We cannot store a pointer to the 'ic' object, since it is resource allocated. Instead we
- // store the location of the inline cache. Then we have enough information recreate the CompiledIC
- // object when we need to remove the stub.
- _ic_site = ic->instruction_address();
-
- // Assemble new stub
- InlineCacheBuffer::assemble_ic_buffer_code(code_begin(), cached_val, dest_addr);
- assert(destination() == dest_addr, "can recover destination");
- assert(cached_value() == cached_val, "can recover destination");
-}
-
-
-void ICStub::clear() {
- if (CompiledIC::is_icholder_entry(destination())) {
- InlineCacheBuffer::queue_for_release((CompiledICHolder*)cached_value());
- }
- _ic_site = nullptr;
-}
-
-
-#ifndef PRODUCT
-// anybody calling to this stub will trap
-
-void ICStub::verify() {
-}
-
-void ICStub::print() {
- tty->print_cr("ICStub: site: " INTPTR_FORMAT, p2i(_ic_site));
-}
-#endif
-
-//-----------------------------------------------------------------------------------------------
-// Implementation of InlineCacheBuffer
-
-
-void InlineCacheBuffer::initialize() {
- if (_buffer != nullptr) return; // already initialized
- _buffer = new StubQueue(new ICStubInterface, checked_cast(InlineCacheBufferSize), InlineCacheBuffer_lock, "InlineCacheBuffer");
- assert (_buffer != nullptr, "cannot allocate InlineCacheBuffer");
-}
-
-
-void InlineCacheBuffer::refill_ic_stubs() {
-#ifdef ASSERT
- ICRefillVerifier* verifier = current_ic_refill_verifier();
- verifier->request_remembered();
-#endif
- // we ran out of inline cache buffer space; must enter safepoint.
- // We do this by forcing a safepoint
- VM_ICBufferFull ibf;
- VMThread::execute(&ibf);
-}
-
-bool InlineCacheBuffer::needs_update_inline_caches() {
- // Stub removal
- if (buffer()->number_of_stubs() > 0) {
- return true;
- }
-
- // Release pending CompiledICHolder
- if (pending_icholder_count() > 0) {
- return true;
- }
-
- return false;
-}
-
-void InlineCacheBuffer::update_inline_caches() {
- if (buffer()->number_of_stubs() > 0) {
- if (TraceICBuffer) {
- tty->print_cr("[updating inline caches with %d stubs]", buffer()->number_of_stubs());
- }
- buffer()->remove_all();
- }
- release_pending_icholders();
-}
-
-
-bool InlineCacheBuffer::contains(address instruction_address) {
- return buffer()->contains(instruction_address);
-}
-
-
-bool InlineCacheBuffer::is_empty() {
- return buffer()->number_of_stubs() == 0;
-}
-
-
-void InlineCacheBuffer_init() {
- InlineCacheBuffer::initialize();
-}
-
-bool InlineCacheBuffer::create_transition_stub(CompiledIC *ic, void* cached_value, address entry) {
- assert(!SafepointSynchronize::is_at_safepoint(), "should not be called during a safepoint");
- assert(CompiledICLocker::is_safe(ic->instruction_address()), "mt unsafe call");
- if (TraceICBuffer) {
- tty->print_cr(" create transition stub for " INTPTR_FORMAT " destination " INTPTR_FORMAT " cached value " INTPTR_FORMAT,
- p2i(ic->instruction_address()), p2i(entry), p2i(cached_value));
- }
-
- // allocate and initialize new "out-of-line" inline-cache
- ICStub* ic_stub = (ICStub*) buffer()->request_committed(ic_stub_code_size());
- if (ic_stub == nullptr) {
-#ifdef ASSERT
- ICRefillVerifier* verifier = current_ic_refill_verifier();
- verifier->request_refill();
-#endif
- return false;
- }
-
-#ifdef ASSERT
- {
- ICStub* rev_stub = ICStub::from_destination_address(ic_stub->code_begin());
- assert(ic_stub == rev_stub,
- "ICStub mapping is reversible: stub=" PTR_FORMAT ", code=" PTR_FORMAT ", rev_stub=" PTR_FORMAT,
- p2i(ic_stub), p2i(ic_stub->code_begin()), p2i(rev_stub));
- }
-#endif
-
- // If an transition stub is already associate with the inline cache, then we remove the association.
- if (ic->is_in_transition_state()) {
- ICStub* old_stub = ICStub::from_destination_address(ic->stub_address());
- old_stub->clear();
- }
-
- ic_stub->set_stub(ic, cached_value, entry);
-
- // Update inline cache in nmethod to point to new "out-of-line" allocated inline cache
- ic->set_ic_destination(ic_stub);
- return true;
-}
-
-
-address InlineCacheBuffer::ic_destination_for(CompiledIC *ic) {
- ICStub* stub = ICStub::from_destination_address(ic->stub_address());
- return stub->destination();
-}
-
-
-void* InlineCacheBuffer::cached_value_for(CompiledIC *ic) {
- ICStub* stub = ICStub::from_destination_address(ic->stub_address());
- return stub->cached_value();
-}
-
-
-// Free CompiledICHolder*s that are no longer in use
-void InlineCacheBuffer::release_pending_icholders() {
- assert(SafepointSynchronize::is_at_safepoint(), "should only be called during a safepoint");
- CompiledICHolder* holder = Atomic::load(&_pending_released);
- _pending_released = nullptr;
- int count = 0;
- while (holder != nullptr) {
- CompiledICHolder* next = holder->next();
- delete holder;
- holder = next;
- count++;
- }
- assert(pending_icholder_count() == count, "wrong count");
- Atomic::store(&_pending_count, 0);
-}
-
-// Enqueue this icholder for release during the next safepoint. It's
-// not safe to free them until then since they might be visible to
-// another thread.
-void InlineCacheBuffer::queue_for_release(CompiledICHolder* icholder) {
- assert(icholder->next() == nullptr, "multiple enqueue?");
-
- CompiledICHolder* old = Atomic::load(&_pending_released);
- for (;;) {
- icholder->set_next(old);
- // The only reader runs at a safepoint serially so there is no need for a more strict atomic.
- CompiledICHolder* cur = Atomic::cmpxchg(&_pending_released, old, icholder, memory_order_relaxed);
- if (cur == old) {
- break;
- }
- old = cur;
- }
- Atomic::inc(&_pending_count, memory_order_relaxed);
-
- if (TraceICBuffer) {
- tty->print_cr("enqueueing icholder " INTPTR_FORMAT " to be freed", p2i(icholder));
- }
-}
-
-int InlineCacheBuffer::pending_icholder_count() {
- return Atomic::load(&_pending_count);
-}
diff --git a/src/hotspot/share/code/icBuffer.hpp b/src/hotspot/share/code/icBuffer.hpp
deleted file mode 100644
index f67080e6b58..00000000000
--- a/src/hotspot/share/code/icBuffer.hpp
+++ /dev/null
@@ -1,193 +0,0 @@
-/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef SHARE_CODE_ICBUFFER_HPP
-#define SHARE_CODE_ICBUFFER_HPP
-
-#include "asm/codeBuffer.hpp"
-#include "code/stubs.hpp"
-#include "interpreter/bytecodes.hpp"
-#include "memory/allocation.hpp"
-#include "runtime/safepointVerifiers.hpp"
-#include "utilities/align.hpp"
-#include "utilities/debug.hpp"
-#include "utilities/macros.hpp"
-
-class CompiledIC;
-class CompiledICHolder;
-
-//
-// For CompiledIC's:
-//
-// In cases where we do not have MT-safe state transformation,
-// we go to a transition state, using ICStubs. At a safepoint,
-// the inline caches are transferred from the transitional code:
-//
-// instruction_address --> 01 set xxx_oop, Ginline_cache_klass
-// 23 jump_to Gtemp, yyyy
-// 4 nop
-
-class ICStub: public Stub {
- private:
- int _size; // total size of the stub incl. code
- address _ic_site; // points at call instruction of owning ic-buffer
- /* stub code follows here */
- protected:
- friend class ICStubInterface;
- // This will be called only by ICStubInterface
- void initialize(int size) { _size = size; _ic_site = nullptr; }
- void finalize(); // called when a method is removed
-
- // General info
- int size() const { return _size; }
-
- // To be cautious, we want to make sure that each ICStub is in a separate instruction
- // cache line. This would allow for piggybacking on instruction cache coherency on
- // some architectures to order the updates to ICStub and setting the destination to
- // the ICStub. Note that cache line size might be larger than CodeEntryAlignment
- // that is normal alignment for CodeBlobs.
- static int alignment() { return DEFAULT_CACHE_LINE_SIZE; }
-
- // Aligning the code section is normally done for performance reasons, which is not
- // required for ICStubs, as these stubs are transitional. Setting code alignment
- // to CodeEntryAlignment would waste a lot of memory in ICBuffer. Aligning to
- // word size should be enough. This also offsets the costs of aligning the entire
- // ICStub to cache line (see above), as smaller code alignment would allow ICStub
- // to fit a _single_ cache line.
- static int code_alignment() { return HeapWordSize; }
-
- public:
- // Creation
- void set_stub(CompiledIC *ic, void* cached_value, address dest_addr);
-
- // Code info
- address code_begin() const { return align_up((address)this + sizeof(ICStub), code_alignment()); }
- address code_end() const { return (address)this + size(); }
-
- // Call site info
- address ic_site() const { return _ic_site; }
- void clear();
- bool is_empty() const { return _ic_site == nullptr; }
-
- // stub info
- address destination() const; // destination of jump instruction
- void* cached_value() const; // cached_value for stub
-
- // Debugging
- void verify() PRODUCT_RETURN;
- void print() PRODUCT_RETURN;
-
- // Creation
- static inline ICStub* from_destination_address(address destination_address) {
- ICStub* stub = (ICStub*) align_down(destination_address - sizeof(ICStub), alignment());
-#ifdef ASSERT
- stub->verify();
-#endif
- return stub;
- }
-};
-
-#ifdef ASSERT
-// The ICRefillVerifier class is a stack allocated RAII object used to
-// detect if a failed IC transition that required IC stub refilling has
-// been accidentally missed. It is up to the caller to in that case
-// refill IC stubs.
-class ICRefillVerifier: StackObj {
- bool _refill_requested;
- bool _refill_remembered;
-
- public:
- ICRefillVerifier();
- ~ICRefillVerifier();
-
- void request_refill() { _refill_requested = true; }
- void request_remembered() { _refill_remembered = true; }
-};
-
-// The ICRefillVerifierMark is used to set the thread's current
-// ICRefillVerifier to a provided one. This is useful in particular
-// when transitioning IC stubs in parallel and refilling from the
-// master thread invoking the IC stub transitioning code.
-class ICRefillVerifierMark: StackObj {
- public:
- ICRefillVerifierMark(ICRefillVerifier* verifier);
- ~ICRefillVerifierMark();
-};
-#else
-class ICRefillVerifier: StackObj {
- public:
- ICRefillVerifier() {}
-};
-class ICRefillVerifierMark: StackObj {
- public:
- ICRefillVerifierMark(ICRefillVerifier* verifier) {}
-};
-#endif
-
-class InlineCacheBuffer: public AllStatic {
- private:
- // friends
- friend class ICStub;
-
- static int ic_stub_code_size();
-
- static StubQueue* _buffer;
-
- static CompiledICHolder* volatile _pending_released;
- static volatile int _pending_count;
-
- static StubQueue* buffer() { return _buffer; }
-
- // Machine-dependent implementation of ICBuffer
- static void assemble_ic_buffer_code(address code_begin, void* cached_value, address entry_point);
- static address ic_buffer_entry_point (address code_begin);
- static void* ic_buffer_cached_value (address code_begin);
-
- public:
-
- // Initialization; must be called before first usage
- static void initialize();
-
- // Access
- static bool contains(address instruction_address);
-
- // removes the ICStubs after backpatching
- static bool needs_update_inline_caches();
- static void update_inline_caches();
- static void refill_ic_stubs();
-
- // for debugging
- static bool is_empty();
-
- static void release_pending_icholders();
- static void queue_for_release(CompiledICHolder* icholder);
- static int pending_icholder_count();
-
- // New interface
- static bool create_transition_stub(CompiledIC *ic, void* cached_value, address entry);
- static address ic_destination_for(CompiledIC *ic);
- static void* cached_value_for(CompiledIC *ic);
-};
-
-#endif // SHARE_CODE_ICBUFFER_HPP
diff --git a/src/hotspot/share/code/nmethod.cpp b/src/hotspot/share/code/nmethod.cpp
index b18077fddfd..2755df32513 100644
--- a/src/hotspot/share/code/nmethod.cpp
+++ b/src/hotspot/share/code/nmethod.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -640,6 +640,7 @@ nmethod::nmethod(
ByteSize basic_lock_sp_offset,
OopMapSet* oop_maps )
: CompiledMethod(method, "native nmethod", type, nmethod_size, sizeof(nmethod), code_buffer, offsets->value(CodeOffsets::Frame_Complete), frame_size, oop_maps, false, true),
+ _compiled_ic_data(nullptr),
_is_unlinked(false),
_native_receiver_sp_offset(basic_lock_owner_sp_offset),
_native_basic_lock_sp_offset(basic_lock_sp_offset),
@@ -697,12 +698,12 @@ nmethod::nmethod(
clear_unloading_state();
+ finalize_relocations();
+
Universe::heap()->register_nmethod(this);
debug_only(Universe::heap()->verify_nmethod(this));
CodeCache::commit(this);
-
- finalize_relocations();
}
if (PrintNativeNMethods || PrintDebugInfo || PrintRelocations || PrintDependencies) {
@@ -784,6 +785,7 @@ nmethod::nmethod(
#endif
)
: CompiledMethod(method, "nmethod", type, nmethod_size, sizeof(nmethod), code_buffer, offsets->value(CodeOffsets::Frame_Complete), frame_size, oop_maps, false, true),
+ _compiled_ic_data(nullptr),
_is_unlinked(false),
_native_receiver_sp_offset(in_ByteSize(-1)),
_native_basic_lock_sp_offset(in_ByteSize(-1)),
@@ -887,13 +889,13 @@ nmethod::nmethod(
}
#endif
+ finalize_relocations();
+
Universe::heap()->register_nmethod(this);
debug_only(Universe::heap()->verify_nmethod(this));
CodeCache::commit(this);
- finalize_relocations();
-
// Copy contents of ExceptionHandlerTable to nmethod
handler_table->copy_to(this);
nul_chk_table->copy_to(this);
@@ -1021,7 +1023,7 @@ void nmethod::print_nmethod(bool printmethod) {
tty->print_cr("- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ");
print_metadata(tty);
tty->print_cr("- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ");
- print_pcs();
+ print_pcs_on(tty);
tty->print_cr("- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ");
if (oop_maps() != nullptr) {
tty->print("oop maps:"); // oop_maps()->print_on(tty) outputs a cr() at the beginning
@@ -1145,16 +1147,33 @@ static void install_post_call_nop_displacement(nmethod* nm, address pc) {
void nmethod::finalize_relocations() {
NoSafepointVerifier nsv;
+ GrowableArray virtual_call_data;
+
// Make sure that post call nops fill in nmethod offsets eagerly so
// we don't have to race with deoptimization
RelocIterator iter(this);
while (iter.next()) {
- if (iter.type() == relocInfo::post_call_nop_type) {
+ if (iter.type() == relocInfo::virtual_call_type) {
+ virtual_call_Relocation* r = iter.virtual_call_reloc();
+ NativeMovConstReg* value = nativeMovConstReg_at(r->cached_value());
+ virtual_call_data.append(value);
+ } else if (iter.type() == relocInfo::post_call_nop_type) {
post_call_nop_Relocation* const reloc = iter.post_call_nop_reloc();
address pc = reloc->addr();
install_post_call_nop_displacement(this, pc);
}
}
+
+ if (virtual_call_data.length() > 0) {
+ // We allocate a block of CompiledICData per nmethod so the GC can purge this faster.
+ _compiled_ic_data = new CompiledICData[virtual_call_data.length()];
+ CompiledICData* next_data = _compiled_ic_data;
+
+ for (NativeMovConstReg* value : virtual_call_data) {
+ value->set_data((intptr_t)next_data);
+ next_data++;
+ }
+ }
}
void nmethod::make_deoptimized() {
@@ -1180,8 +1199,7 @@ void nmethod::make_deoptimized() {
while (iter.next()) {
switch (iter.type()) {
- case relocInfo::virtual_call_type:
- case relocInfo::opt_virtual_call_type: {
+ case relocInfo::virtual_call_type: {
CompiledIC *ic = CompiledIC_at(&iter);
address pc = ic->end_of_call();
NativePostCallNop* nop = nativePostCallNop_at(pc);
@@ -1191,8 +1209,9 @@ void nmethod::make_deoptimized() {
assert(NativeDeoptInstruction::is_deopt_at(pc), "check");
break;
}
- case relocInfo::static_call_type: {
- CompiledStaticCall *csc = compiledStaticCall_at(iter.reloc());
+ case relocInfo::static_call_type:
+ case relocInfo::opt_virtual_call_type: {
+ CompiledDirectCall *csc = CompiledDirectCall::at(iter.reloc());
address pc = csc->end_of_call();
NativePostCallNop* nop = nativePostCallNop_at(pc);
//tty->print_cr(" - static pc %p", pc);
@@ -1219,29 +1238,29 @@ void nmethod::verify_clean_inline_caches() {
RelocIterator iter(this, oops_reloc_begin());
while(iter.next()) {
switch(iter.type()) {
- case relocInfo::virtual_call_type:
- case relocInfo::opt_virtual_call_type: {
+ case relocInfo::virtual_call_type: {
CompiledIC *ic = CompiledIC_at(&iter);
- CodeBlob *cb = CodeCache::find_blob(ic->ic_destination());
+ CodeBlob *cb = CodeCache::find_blob(ic->destination());
assert(cb != nullptr, "destination not in CodeBlob?");
nmethod* nm = cb->as_nmethod_or_null();
- if( nm != nullptr ) {
+ if (nm != nullptr) {
// Verify that inline caches pointing to bad nmethods are clean
- if (!nm->is_in_use() || (nm->method()->code() != nm)) {
+ if (!nm->is_in_use() || nm->is_unloading()) {
assert(ic->is_clean(), "IC should be clean");
}
}
break;
}
- case relocInfo::static_call_type: {
- CompiledStaticCall *csc = compiledStaticCall_at(iter.reloc());
- CodeBlob *cb = CodeCache::find_blob(csc->destination());
+ case relocInfo::static_call_type:
+ case relocInfo::opt_virtual_call_type: {
+ CompiledDirectCall *cdc = CompiledDirectCall::at(iter.reloc());
+ CodeBlob *cb = CodeCache::find_blob(cdc->destination());
assert(cb != nullptr, "destination not in CodeBlob?");
nmethod* nm = cb->as_nmethod_or_null();
- if( nm != nullptr ) {
+ if (nm != nullptr) {
// Verify that inline caches pointing to bad nmethods are clean
- if (!nm->is_in_use() || (nm->method()->code() != nm)) {
- assert(csc->is_clean(), "IC should be clean");
+ if (!nm->is_in_use() || nm->is_unloading() || nm->method()->code() != nm) {
+ assert(cdc->is_clean(), "IC should be clean");
}
}
break;
@@ -1405,9 +1424,7 @@ bool nmethod::make_not_entrant() {
// For concurrent GCs, there must be a handshake between unlink and flush
void nmethod::unlink() {
if (_is_unlinked) {
- // Already unlinked. It can be invoked twice because concurrent code cache
- // unloading might need to restart when inline cache cleaning fails due to
- // running out of ICStubs, which can only be refilled at safepoints
+ // Already unlinked.
return;
}
@@ -1418,7 +1435,6 @@ void nmethod::unlink() {
// the Method, because it is only concurrently unlinked by
// the entry barrier, which acquires the per nmethod lock.
unlink_from_method();
- clear_ic_callsites();
if (is_osr_method()) {
invalidate_osr_method();
@@ -1463,10 +1479,11 @@ void nmethod::purge(bool free_code_cache_data, bool unregister_nmethod) {
ec = next;
}
+ delete[] _compiled_ic_data;
+
if (unregister_nmethod) {
Universe::heap()->unregister_nmethod(this);
}
-
CodeCache::unregister_old_nmethod(this);
CodeBlob::purge(free_code_cache_data, unregister_nmethod);
@@ -1604,16 +1621,7 @@ void nmethod::metadata_do(MetadataClosure* f) {
// Check compiledIC holders associated with this nmethod
ResourceMark rm;
CompiledIC *ic = CompiledIC_at(&iter);
- if (ic->is_icholder_call()) {
- CompiledICHolder* cichk = ic->cached_icholder();
- f->do_metadata(cichk->holder_metadata());
- f->do_metadata(cichk->holder_klass());
- } else {
- Metadata* ic_oop = ic->cached_metadata();
- if (ic_oop != nullptr) {
- f->do_metadata(ic_oop);
- }
- }
+ ic->metadata_do(f);
}
}
}
@@ -1750,8 +1758,7 @@ void nmethod::do_unloading(bool unloading_occurred) {
if (is_unloading()) {
unlink();
} else {
- guarantee(unload_nmethod_caches(unloading_occurred),
- "Should not need transition stubs");
+ unload_nmethod_caches(unloading_occurred);
BarrierSetNMethod* bs_nm = BarrierSet::barrier_set()->barrier_set_nmethod();
if (bs_nm != nullptr) {
bs_nm->disarm(this);
@@ -2284,15 +2291,23 @@ void nmethod::verify() {
}
-void nmethod::verify_interrupt_point(address call_site) {
+void nmethod::verify_interrupt_point(address call_site, bool is_inline_cache) {
// Verify IC only when nmethod installation is finished.
if (!is_not_installed()) {
if (CompiledICLocker::is_safe(this)) {
- CompiledIC_at(this, call_site);
+ if (is_inline_cache) {
+ CompiledIC_at(this, call_site);
+ } else {
+ CompiledDirectCall::at(call_site);
+ }
} else {
CompiledICLocker ml_verify(this);
- CompiledIC_at(this, call_site);
+ if (is_inline_cache) {
+ CompiledIC_at(this, call_site);
+ } else {
+ CompiledDirectCall::at(call_site);
+ }
}
}
@@ -2316,15 +2331,15 @@ void nmethod::verify_scopes() {
address stub = nullptr;
switch (iter.type()) {
case relocInfo::virtual_call_type:
- verify_interrupt_point(iter.addr());
+ verify_interrupt_point(iter.addr(), true /* is_inline_cache */);
break;
case relocInfo::opt_virtual_call_type:
stub = iter.opt_virtual_call_reloc()->static_stub();
- verify_interrupt_point(iter.addr());
+ verify_interrupt_point(iter.addr(), false /* is_inline_cache */);
break;
case relocInfo::static_call_type:
stub = iter.static_call_reloc()->static_stub();
- //verify_interrupt_point(iter.addr());
+ verify_interrupt_point(iter.addr(), false /* is_inline_cache */);
break;
case relocInfo::runtime_call_type:
case relocInfo::runtime_call_w_cp_type: {
@@ -3239,75 +3254,6 @@ void nmethod::print_code_comment_on(outputStream* st, int column, address begin,
#endif
-class DirectNativeCallWrapper: public NativeCallWrapper {
-private:
- NativeCall* _call;
-
-public:
- DirectNativeCallWrapper(NativeCall* call) : _call(call) {}
-
- virtual address destination() const { return _call->destination(); }
- virtual address instruction_address() const { return _call->instruction_address(); }
- virtual address next_instruction_address() const { return _call->next_instruction_address(); }
- virtual address return_address() const { return _call->return_address(); }
-
- virtual address get_resolve_call_stub(bool is_optimized) const {
- if (is_optimized) {
- return SharedRuntime::get_resolve_opt_virtual_call_stub();
- }
- return SharedRuntime::get_resolve_virtual_call_stub();
- }
-
- virtual void set_destination_mt_safe(address dest) {
- _call->set_destination_mt_safe(dest);
- }
-
- virtual void set_to_interpreted(const methodHandle& method, CompiledICInfo& info) {
- CompiledDirectStaticCall* csc = CompiledDirectStaticCall::at(instruction_address());
- {
- csc->set_to_interpreted(method, info.entry());
- }
- }
-
- virtual void verify() const {
- // make sure code pattern is actually a call imm32 instruction
- _call->verify();
- _call->verify_alignment();
- }
-
- virtual void verify_resolve_call(address dest) const {
- CodeBlob* db = CodeCache::find_blob(dest);
- assert(db != nullptr && !db->is_adapter_blob(), "must use stub!");
- }
-
- virtual bool is_call_to_interpreted(address dest) const {
- CodeBlob* cb = CodeCache::find_blob(_call->instruction_address());
- return cb->contains(dest);
- }
-
- virtual bool is_safe_for_patching() const { return false; }
-
- virtual NativeInstruction* get_load_instruction(virtual_call_Relocation* r) const {
- return nativeMovConstReg_at(r->cached_value());
- }
-
- virtual void *get_data(NativeInstruction* instruction) const {
- return (void*)((NativeMovConstReg*) instruction)->data();
- }
-
- virtual void set_data(NativeInstruction* instruction, intptr_t data) {
- ((NativeMovConstReg*) instruction)->set_data(data);
- }
-};
-
-NativeCallWrapper* nmethod::call_wrapper_at(address call) const {
- return new DirectNativeCallWrapper((NativeCall*) call);
-}
-
-NativeCallWrapper* nmethod::call_wrapper_before(address return_pc) const {
- return new DirectNativeCallWrapper(nativeCall_before(return_pc));
-}
-
address nmethod::call_instruction_address(address pc) const {
if (NativeCall::is_call_before(pc)) {
NativeCall *ncall = nativeCall_before(pc);
@@ -3316,18 +3262,6 @@ address nmethod::call_instruction_address(address pc) const {
return nullptr;
}
-CompiledStaticCall* nmethod::compiledStaticCall_at(Relocation* call_site) const {
- return CompiledDirectStaticCall::at(call_site);
-}
-
-CompiledStaticCall* nmethod::compiledStaticCall_at(address call_site) const {
- return CompiledDirectStaticCall::at(call_site);
-}
-
-CompiledStaticCall* nmethod::compiledStaticCall_before(address return_addr) const {
- return CompiledDirectStaticCall::before(return_addr);
-}
-
#if defined(SUPPORT_DATA_STRUCTS)
void nmethod::print_value_on(outputStream* st) const {
st->print("nmethod");
@@ -3341,15 +3275,15 @@ void nmethod::print_calls(outputStream* st) {
RelocIterator iter(this);
while (iter.next()) {
switch (iter.type()) {
- case relocInfo::virtual_call_type:
- case relocInfo::opt_virtual_call_type: {
+ case relocInfo::virtual_call_type: {
CompiledICLocker ml_verify(this);
CompiledIC_at(&iter)->print();
break;
}
case relocInfo::static_call_type:
- st->print_cr("Static call at " INTPTR_FORMAT, p2i(iter.reloc()->addr()));
- CompiledDirectStaticCall::at(iter.reloc())->print();
+ case relocInfo::opt_virtual_call_type:
+ st->print_cr("Direct call at " INTPTR_FORMAT, p2i(iter.reloc()->addr()));
+ CompiledDirectCall::at(iter.reloc())->print();
break;
default:
break;
diff --git a/src/hotspot/share/code/nmethod.hpp b/src/hotspot/share/code/nmethod.hpp
index 656dae4171b..2993db21305 100644
--- a/src/hotspot/share/code/nmethod.hpp
+++ b/src/hotspot/share/code/nmethod.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -27,6 +27,7 @@
#include "code/compiledMethod.hpp"
+class CompiledICData;
class CompileTask;
class DepChange;
class DirectiveSet;
@@ -196,6 +197,7 @@ class nmethod : public CompiledMethod {
address _verified_entry_point; // entry point without class check
address _osr_entry_point; // entry point for on stack replacement
+ CompiledICData* _compiled_ic_data;
bool _is_unlinked;
// Shared fields for all nmethod's
@@ -604,7 +606,7 @@ class nmethod : public CompiledMethod {
// verify operations
void verify();
void verify_scopes();
- void verify_interrupt_point(address interrupt_point);
+ void verify_interrupt_point(address interrupt_point, bool is_inline_cache);
// Disassemble this nmethod with additional debug information, e.g. information about blocks.
void decode2(outputStream* st) const;
@@ -621,7 +623,6 @@ class nmethod : public CompiledMethod {
#if defined(SUPPORT_DATA_STRUCTS)
// print output in opt build for disassembler library
void print_relocations() PRODUCT_RETURN;
- void print_pcs() { print_pcs_on(tty); }
void print_pcs_on(outputStream* st);
void print_scopes() { print_scopes_on(tty); }
void print_scopes_on(outputStream* st) PRODUCT_RETURN;
@@ -635,8 +636,7 @@ class nmethod : public CompiledMethod {
void print_oops(outputStream* st); // oops from the underlying CodeBlob.
void print_metadata(outputStream* st); // metadata in metadata pool.
#else
- // void print_pcs() PRODUCT_RETURN;
- void print_pcs() { return; }
+ void print_pcs_on(outputStream* st) { return; }
#endif
void print_calls(outputStream* st) PRODUCT_RETURN;
@@ -701,14 +701,8 @@ class nmethod : public CompiledMethod {
virtual void metadata_do(MetadataClosure* f);
- NativeCallWrapper* call_wrapper_at(address call) const;
- NativeCallWrapper* call_wrapper_before(address return_pc) const;
address call_instruction_address(address pc) const;
- virtual CompiledStaticCall* compiledStaticCall_at(Relocation* call_site) const;
- virtual CompiledStaticCall* compiledStaticCall_at(address addr) const;
- virtual CompiledStaticCall* compiledStaticCall_before(address addr) const;
-
virtual void make_deoptimized();
void finalize_relocations();
};
diff --git a/src/hotspot/share/code/relocInfo.cpp b/src/hotspot/share/code/relocInfo.cpp
index bfb3db72d72..ef908757675 100644
--- a/src/hotspot/share/code/relocInfo.cpp
+++ b/src/hotspot/share/code/relocInfo.cpp
@@ -641,12 +641,10 @@ Method* virtual_call_Relocation::method_value() {
return (Method*)m;
}
-bool virtual_call_Relocation::clear_inline_cache() {
- // No stubs for ICs
- // Clean IC
+void virtual_call_Relocation::clear_inline_cache() {
ResourceMark rm;
CompiledIC* icache = CompiledIC_at(this);
- return icache->set_to_clean();
+ icache->set_to_clean();
}
@@ -669,18 +667,10 @@ Method* opt_virtual_call_Relocation::method_value() {
return (Method*)m;
}
-template
-static bool set_to_clean_no_ic_refill(CompiledICorStaticCall* ic) {
- guarantee(ic->set_to_clean(), "Should not need transition stubs");
- return true;
-}
-
-bool opt_virtual_call_Relocation::clear_inline_cache() {
- // No stubs for ICs
- // Clean IC
+void opt_virtual_call_Relocation::clear_inline_cache() {
ResourceMark rm;
- CompiledIC* icache = CompiledIC_at(this);
- return set_to_clean_no_ic_refill(icache);
+ CompiledDirectCall* callsite = CompiledDirectCall::at(this);
+ callsite->set_to_clean();
}
address opt_virtual_call_Relocation::static_stub() {
@@ -717,10 +707,10 @@ void static_call_Relocation::unpack_data() {
_method_index = unpack_1_int();
}
-bool static_call_Relocation::clear_inline_cache() {
- // Safe call site info
- CompiledStaticCall* handler = this->code()->compiledStaticCall_at(this);
- return set_to_clean_no_ic_refill(handler);
+void static_call_Relocation::clear_inline_cache() {
+ ResourceMark rm;
+ CompiledDirectCall* callsite = CompiledDirectCall::at(this);
+ callsite->set_to_clean();
}
@@ -759,11 +749,10 @@ address trampoline_stub_Relocation::get_trampoline_for(address call, nmethod* co
return nullptr;
}
-bool static_stub_Relocation::clear_inline_cache() {
+void static_stub_Relocation::clear_inline_cache() {
// Call stub is only used when calling the interpreted code.
// It does not really need to be cleared, except that we want to clean out the methodoop.
- CompiledDirectStaticCall::set_stub_to_clean(this);
- return true;
+ CompiledDirectCall::set_stub_to_clean(this);
}
diff --git a/src/hotspot/share/code/relocInfo.hpp b/src/hotspot/share/code/relocInfo.hpp
index 77e24708bb1..5f67f94bdad 100644
--- a/src/hotspot/share/code/relocInfo.hpp
+++ b/src/hotspot/share/code/relocInfo.hpp
@@ -862,7 +862,7 @@ class Relocation {
// all relocations are able to reassert their values
virtual void set_value(address x);
- virtual bool clear_inline_cache() { return true; }
+ virtual void clear_inline_cache() {}
// This method assumes that all virtual/static (inline) caches are cleared (since for static_call_type and
// ic_call_type is not always position dependent (depending on the state of the cache)). However, this is
@@ -1141,7 +1141,7 @@ class virtual_call_Relocation : public CallRelocation {
void pack_data_to(CodeSection* dest) override;
void unpack_data() override;
- bool clear_inline_cache() override;
+ void clear_inline_cache() override;
};
@@ -1170,7 +1170,7 @@ class opt_virtual_call_Relocation : public CallRelocation {
void pack_data_to(CodeSection* dest) override;
void unpack_data() override;
- bool clear_inline_cache() override;
+ void clear_inline_cache() override;
// find the matching static_stub
address static_stub();
@@ -1202,7 +1202,7 @@ class static_call_Relocation : public CallRelocation {
void pack_data_to(CodeSection* dest) override;
void unpack_data() override;
- bool clear_inline_cache() override;
+ void clear_inline_cache() override;
// find the matching static_stub
address static_stub();
@@ -1227,7 +1227,7 @@ class static_stub_Relocation : public Relocation {
static_stub_Relocation() : Relocation(relocInfo::static_stub_type) { }
public:
- bool clear_inline_cache() override;
+ void clear_inline_cache() override;
address static_call() { return _static_call; }
diff --git a/src/hotspot/share/code/vtableStubs.cpp b/src/hotspot/share/code/vtableStubs.cpp
index eed3dc8e787..5a54426d6a4 100644
--- a/src/hotspot/share/code/vtableStubs.cpp
+++ b/src/hotspot/share/code/vtableStubs.cpp
@@ -283,13 +283,6 @@ VtableStub* VtableStubs::entry_point(address pc) {
return (s == stub) ? s : nullptr;
}
-bool VtableStubs::is_icholder_entry(address pc) {
- assert(contains(pc), "must contain all vtable blobs");
- VtableStub* stub = (VtableStub*)(pc - VtableStub::entry_offset());
- // itable stubs use CompiledICHolder.
- return stub->is_itable_stub();
-}
-
bool VtableStubs::contains(address pc) {
// simple solution for now - we may want to use
// a faster way if this function is called often
diff --git a/src/hotspot/share/code/vtableStubs.hpp b/src/hotspot/share/code/vtableStubs.hpp
index 7076e50f3e3..3993e1e72d5 100644
--- a/src/hotspot/share/code/vtableStubs.hpp
+++ b/src/hotspot/share/code/vtableStubs.hpp
@@ -107,7 +107,6 @@ class VtableStubs : AllStatic {
static address find_itable_stub(int itable_index) { return find_stub(false, itable_index); }
static VtableStub* entry_point(address pc); // vtable stub entry point for a pc
- static bool is_icholder_entry(address pc); // is the blob containing pc (which must be a vtable blob) an icholder?
static bool contains(address pc); // is pc within any stub?
static VtableStub* stub_containing(address pc); // stub containing pc or nullptr
static void initialize();
diff --git a/src/hotspot/share/compiler/compileBroker.cpp b/src/hotspot/share/compiler/compileBroker.cpp
index ff5fa44050c..3a251128914 100644
--- a/src/hotspot/share/compiler/compileBroker.cpp
+++ b/src/hotspot/share/compiler/compileBroker.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -582,7 +582,7 @@ CompilerCounters::CompilerCounters() {
// c2 uses explicit CompilerPhaseType idToPhase mapping in opto/phasetype.hpp,
// so if c2 is used, it should be always registered first.
// This function is called during vm initialization.
-void register_jfr_phasetype_serializer(CompilerType compiler_type) {
+static void register_jfr_phasetype_serializer(CompilerType compiler_type) {
ResourceMark rm;
static bool first_registration = true;
if (compiler_type == compiler_jvmci) {
@@ -841,8 +841,15 @@ void DeoptimizeObjectsALotThread::deoptimize_objects_alot_loop_all() {
JavaThread* CompileBroker::make_thread(ThreadType type, jobject thread_handle, CompileQueue* queue, AbstractCompiler* comp, JavaThread* THREAD) {
- JavaThread* new_thread = nullptr;
+ Handle thread_oop(THREAD, JNIHandles::resolve_non_null(thread_handle));
+ if (java_lang_Thread::thread(thread_oop()) != nullptr) {
+ assert(type == compiler_t, "should only happen with reused compiler threads");
+ // The compiler thread hasn't actually exited yet so don't try to reuse it
+ return nullptr;
+ }
+
+ JavaThread* new_thread = nullptr;
switch (type) {
case compiler_t:
assert(comp != nullptr, "Compiler instance missing.");
@@ -871,7 +878,6 @@ JavaThread* CompileBroker::make_thread(ThreadType type, jobject thread_handle, C
// JavaThread due to lack of resources. We will handle that failure below.
// Also check new_thread so that static analysis is happy.
if (new_thread != nullptr && new_thread->osthread() != nullptr) {
- Handle thread_oop(THREAD, JNIHandles::resolve_non_null(thread_handle));
if (type == compiler_t) {
CompilerThread::cast(new_thread)->set_compiler(comp);
diff --git a/src/hotspot/share/compiler/compilerDirectives.cpp b/src/hotspot/share/compiler/compilerDirectives.cpp
index d7d4c5b60ed..a16e5f23ed9 100644
--- a/src/hotspot/share/compiler/compilerDirectives.cpp
+++ b/src/hotspot/share/compiler/compilerDirectives.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -32,6 +32,7 @@
#include "memory/allocation.inline.hpp"
#include "memory/resourceArea.hpp"
#include "opto/phasetype.hpp"
+#include "opto/traceAutoVectorizationTag.hpp"
#include "runtime/globals_extension.hpp"
CompilerDirectives::CompilerDirectives() : _next(nullptr), _match(nullptr), _ref_count(0) {
@@ -300,7 +301,8 @@ void DirectiveSet::init_control_intrinsic() {
DirectiveSet::DirectiveSet(CompilerDirectives* d) :
_inlinematchers(nullptr),
_directive(d),
- _ideal_phase_name_set(PHASE_NUM_TYPES, mtCompiler)
+ _ideal_phase_name_set(PHASE_NUM_TYPES, mtCompiler),
+ _trace_auto_vectorization_tags(TRACE_AUTO_VECTORIZATION_TAG_NUM, mtCompiler)
{
#define init_defaults_definition(name, type, dvalue, compiler) this->name##Option = dvalue;
compilerdirectives_common_flags(init_defaults_definition)
@@ -433,6 +435,16 @@ DirectiveSet* DirectiveSet::compilecommand_compatibility_init(const methodHandle
// Parse PrintIdealPhaseName and create a lookup set
#ifndef PRODUCT
#ifdef COMPILER2
+ if (!_modified[TraceAutoVectorizationIndex]) {
+ // Parse ccstr and create mask
+ ccstrlist option;
+ if (CompilerOracle::has_option_value(method, CompileCommand::TraceAutoVectorization, option)) {
+ TraceAutoVectorizationTagValidator validator(option, false);
+ if (validator.is_valid()) {
+ set.cloned()->set_trace_auto_vectorization_tags(validator.tags());
+ }
+ }
+ }
if (!_modified[PrintIdealPhaseIndex]) {
// Parse ccstr and create set
ccstrlist option;
diff --git a/src/hotspot/share/compiler/compilerDirectives.hpp b/src/hotspot/share/compiler/compilerDirectives.hpp
index 4c9b51724f9..747c8781817 100644
--- a/src/hotspot/share/compiler/compilerDirectives.hpp
+++ b/src/hotspot/share/compiler/compilerDirectives.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -87,10 +87,10 @@ NOT_PRODUCT(cflags(PrintIdeal, bool, PrintIdeal, PrintIdeal)) \
cflags(Vectorize, bool, false, Vectorize) \
cflags(CloneMapDebug, bool, false, CloneMapDebug) \
NOT_PRODUCT(cflags(IGVPrintLevel, intx, PrintIdealGraphLevel, IGVPrintLevel)) \
- cflags(VectorizeDebug, uintx, 0, VectorizeDebug) \
cflags(IncrementalInlineForceCleanup, bool, IncrementalInlineForceCleanup, IncrementalInlineForceCleanup) \
cflags(MaxNodeLimit, intx, MaxNodeLimit, MaxNodeLimit)
#define compilerdirectives_c2_string_flags(cflags) \
+NOT_PRODUCT(cflags(TraceAutoVectorization, ccstrlist, "", TraceAutoVectorization)) \
NOT_PRODUCT(cflags(PrintIdealPhase, ccstrlist, "", PrintIdealPhase))
#else
#define compilerdirectives_c2_other_flags(cflags)
@@ -131,6 +131,7 @@ class DirectiveSet : public CHeapObj {
CompilerDirectives* _directive;
TriBoolArray<(size_t)vmIntrinsics::number_of_intrinsics(), int> _intrinsic_control_words;
CHeapBitMap _ideal_phase_name_set;
+ CHeapBitMap _trace_auto_vectorization_tags;
public:
DirectiveSet(CompilerDirectives* directive);
@@ -205,6 +206,12 @@ void set_##name(void* value) { \
bool should_print_phase(const CompilerPhaseType cpt) const {
return _ideal_phase_name_set.at(cpt);
};
+ void set_trace_auto_vectorization_tags(const CHeapBitMap& tags) {
+ _trace_auto_vectorization_tags.set_from(tags);
+ };
+ const CHeapBitMap& trace_auto_vectorization_tags() {
+ return _trace_auto_vectorization_tags;
+ };
void print_intx(outputStream* st, ccstr n, intx v, bool mod) { if (mod) { st->print("%s:" INTX_FORMAT " ", n, v); } }
void print_uintx(outputStream* st, ccstr n, intx v, bool mod) { if (mod) { st->print("%s:" UINTX_FORMAT " ", n, v); } }
diff --git a/src/hotspot/share/compiler/compilerOracle.cpp b/src/hotspot/share/compiler/compilerOracle.cpp
index 66eea29fcc1..a8eee10fac5 100644
--- a/src/hotspot/share/compiler/compilerOracle.cpp
+++ b/src/hotspot/share/compiler/compilerOracle.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -35,6 +35,7 @@
#include "oops/method.inline.hpp"
#include "oops/symbol.hpp"
#include "opto/phasetype.hpp"
+#include "opto/traceAutoVectorizationTag.hpp"
#include "runtime/globals_extension.hpp"
#include "runtime/handles.inline.hpp"
#include "runtime/jniHandles.hpp"
@@ -47,7 +48,7 @@ static const char* optiontype_names[] = {
#undef enum_of_types
};
-const char* optiontype2name(enum OptionType type) {
+static const char* optiontype2name(enum OptionType type) {
return optiontype_names[static_cast(type)];
}
@@ -57,7 +58,7 @@ static enum OptionType option_types[] = {
#undef enum_of_options
};
-enum OptionType option2type(enum CompileCommand option) {
+static enum OptionType option2type(enum CompileCommand option) {
return option_types[static_cast(option)];
}
@@ -67,7 +68,7 @@ static const char* option_names[] = {
#undef enum_of_options
};
-const char* option2name(enum CompileCommand option) {
+static const char* option2name(enum CompileCommand option) {
return option_names[static_cast(option)];
}
@@ -107,7 +108,7 @@ static bool print_final_memstat_report = false;
// A filter for quick lookup if an option is set
static bool option_filter[static_cast(CompileCommand::Unknown) + 1] = { 0 };
-void command_set_in_filter(enum CompileCommand option) {
+static void command_set_in_filter(enum CompileCommand option) {
assert(option != CompileCommand::Unknown, "sanity");
assert(option2type(option) != OptionType::Unknown, "sanity");
@@ -119,7 +120,7 @@ void command_set_in_filter(enum CompileCommand option) {
option_filter[static_cast(option)] = true;
}
-bool has_command(enum CompileCommand option) {
+static bool has_command(enum CompileCommand option) {
return option_filter[static_cast(option)];
}
@@ -546,7 +547,7 @@ enum OptionType CompilerOracle::parse_option_type(const char* type_str) {
return OptionType::Unknown;
}
-void print_tip() { // CMH Update info
+static void print_tip() { // CMH Update info
tty->cr();
tty->print_cr("Usage: '-XX:CompileCommand=,' - to set boolean option to true");
tty->print_cr("Usage: '-XX:CompileCommand=,,'");
@@ -554,13 +555,13 @@ void print_tip() { // CMH Update info
tty->cr();
}
-void print_option(enum CompileCommand option, const char* name, enum OptionType type) {
+static void print_option(enum CompileCommand option, const char* name, enum OptionType type) {
if (type != OptionType::Unknown) {
tty->print_cr(" %s (%s)", name, optiontype2name(type));
}
}
-void print_commands() {
+static void print_commands() {
tty->cr();
tty->print_cr("All available options:");
#define enum_of_options(option, name, ctype) print_option(CompileCommand::option, name, OptionType::ctype);
@@ -620,7 +621,7 @@ static void usage() {
tty->cr();
};
-int skip_whitespace(char* &line) {
+static int skip_whitespace(char* &line) {
// Skip any leading spaces
int whitespace_read = 0;
sscanf(line, "%*[ \t]%n", &whitespace_read);
@@ -628,7 +629,7 @@ int skip_whitespace(char* &line) {
return whitespace_read;
}
-void skip_comma(char* &line) {
+static void skip_comma(char* &line) {
// Skip any leading spaces
if (*line == ',') {
line++;
@@ -775,8 +776,14 @@ static void scan_value(enum OptionType type, char* line, int& total_bytes_read,
jio_snprintf(errorbuf, buf_size, "Unrecognized intrinsic detected in %s: %s", option2name(option), validator.what());
}
}
-#ifndef PRODUCT
- else if (option == CompileCommand::PrintIdealPhase) {
+#if !defined(PRODUCT) && defined(COMPILER2)
+ else if (option == CompileCommand::TraceAutoVectorization) {
+ TraceAutoVectorizationTagValidator validator(value, true);
+
+ if (!validator.is_valid()) {
+ jio_snprintf(errorbuf, buf_size, "Unrecognized tag name in %s: %s", option2name(option), validator.what());
+ }
+ } else if (option == CompileCommand::PrintIdealPhase) {
PhaseNameValidator validator(value);
if (!validator.is_valid()) {
diff --git a/src/hotspot/share/compiler/compilerOracle.hpp b/src/hotspot/share/compiler/compilerOracle.hpp
index be1a270f3c9..192e292d35a 100644
--- a/src/hotspot/share/compiler/compilerOracle.hpp
+++ b/src/hotspot/share/compiler/compilerOracle.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -86,8 +86,8 @@ NOT_PRODUCT(option(TraceEscapeAnalysis, "TraceEscapeAnalysis", Bool)) \
NOT_PRODUCT(option(PrintIdeal, "PrintIdeal", Bool)) \
NOT_PRODUCT(option(PrintIdealPhase, "PrintIdealPhase", Ccstrlist)) \
NOT_PRODUCT(option(IGVPrintLevel, "IGVPrintLevel", Intx)) \
+NOT_PRODUCT(option(TraceAutoVectorization, "TraceAutoVectorization", Ccstrlist)) \
option(Vectorize, "Vectorize", Bool) \
- option(VectorizeDebug, "VectorizeDebug", Uintx) \
option(CloneMapDebug, "CloneMapDebug", Bool) \
option(IncrementalInlineForceCleanup, "IncrementalInlineForceCleanup", Bool) \
option(MaxNodeLimit, "MaxNodeLimit", Intx) \
diff --git a/src/hotspot/share/compiler/directivesParser.cpp b/src/hotspot/share/compiler/directivesParser.cpp
index 60955615133..55014900283 100644
--- a/src/hotspot/share/compiler/directivesParser.cpp
+++ b/src/hotspot/share/compiler/directivesParser.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -28,6 +28,7 @@
#include "memory/allocation.inline.hpp"
#include "memory/resourceArea.hpp"
#include "opto/phasetype.hpp"
+#include "opto/traceAutoVectorizationTag.hpp"
#include "runtime/os.hpp"
#include
@@ -335,6 +336,17 @@ bool DirectivesParser::set_option_flag(JSON_TYPE t, JSON_VAL* v, const key* opti
if (!valid) {
error(VALUE_ERROR, "Unrecognized intrinsic detected in DisableIntrinsic: %s", validator.what());
}
+ }
+#if !defined(PRODUCT) && defined(COMPILER2)
+ else if (strncmp(option_key->name, "TraceAutoVectorization", 22) == 0) {
+ TraceAutoVectorizationTagValidator validator(s, false);
+
+ valid = validator.is_valid();
+ if (valid) {
+ set->set_trace_auto_vectorization_tags(validator.tags());
+ } else {
+ error(VALUE_ERROR, "Unrecognized tag name detected in TraceAutoVectorization: %s", validator.what());
+ }
} else if (strncmp(option_key->name, "PrintIdealPhase", 15) == 0) {
PhaseNameValidator validator(s);
@@ -345,6 +357,7 @@ bool DirectivesParser::set_option_flag(JSON_TYPE t, JSON_VAL* v, const key* opti
error(VALUE_ERROR, "Unrecognized phase name detected in PrintIdealPhase: %s", validator.what());
}
}
+#endif
if (!valid) {
FREE_C_HEAP_ARRAY(char, s);
diff --git a/src/hotspot/share/compiler/methodMatcher.cpp b/src/hotspot/share/compiler/methodMatcher.cpp
index 3e2ee288c80..de2e4e82bba 100644
--- a/src/hotspot/share/compiler/methodMatcher.cpp
+++ b/src/hotspot/share/compiler/methodMatcher.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -246,7 +246,7 @@ static MethodMatcher::Mode check_mode(char name[], const char*& error_msg) {
}
// Skip any leading spaces
-void skip_leading_spaces(char*& line, int* total_bytes_read ) {
+static void skip_leading_spaces(char*& line, int* total_bytes_read ) {
int bytes_read = 0;
sscanf(line, "%*[ \t]%n", &bytes_read);
if (bytes_read > 0) {
diff --git a/src/hotspot/share/gc/epsilon/epsilonHeap.hpp b/src/hotspot/share/gc/epsilon/epsilonHeap.hpp
index 741f1d65603..f22d1c22ac8 100644
--- a/src/hotspot/share/gc/epsilon/epsilonHeap.hpp
+++ b/src/hotspot/share/gc/epsilon/epsilonHeap.hpp
@@ -37,7 +37,6 @@
class EpsilonHeap : public CollectedHeap {
friend class VMStructs;
private:
- SoftRefPolicy _soft_ref_policy;
EpsilonMonitoringSupport* _monitoring_support;
MemoryPool* _pool;
GCMemoryManager _memory_manager;
@@ -65,10 +64,6 @@ class EpsilonHeap : public CollectedHeap {
return "Epsilon";
}
- SoftRefPolicy* soft_ref_policy() override {
- return &_soft_ref_policy;
- }
-
jint initialize() override;
void initialize_serviceability() override;
diff --git a/src/hotspot/share/gc/g1/g1Arguments.cpp b/src/hotspot/share/gc/g1/g1Arguments.cpp
index 79b50cd0ea9..5f022bfdc3c 100644
--- a/src/hotspot/share/gc/g1/g1Arguments.cpp
+++ b/src/hotspot/share/gc/g1/g1Arguments.cpp
@@ -143,11 +143,6 @@ void G1Arguments::initialize_card_set_configuration() {
G1RemSetArrayOfCardsEntriesBase << region_size_log_mb));
}
- // Round to next 8 byte boundary for array to maximize space usage.
- size_t const cur_size = G1CardSetArray::size_in_bytes(G1RemSetArrayOfCardsEntries);
- FLAG_SET_ERGO(G1RemSetArrayOfCardsEntries,
- G1RemSetArrayOfCardsEntries + (uint)(align_up(cur_size, G1CardSetAllocOptions::SlotAlignment) - cur_size) / sizeof(G1CardSetArray::EntryDataType));
-
// Howl card set container globals.
if (FLAG_IS_DEFAULT(G1RemSetHowlNumBuckets)) {
FLAG_SET_ERGO(G1RemSetHowlNumBuckets, G1CardSetHowl::num_buckets(HeapRegion::CardsPerRegion,
diff --git a/src/hotspot/share/gc/g1/g1BarrierSet.cpp b/src/hotspot/share/gc/g1/g1BarrierSet.cpp
index 3f2b27c2d06..c2bc7bf5425 100644
--- a/src/hotspot/share/gc/g1/g1BarrierSet.cpp
+++ b/src/hotspot/share/gc/g1/g1BarrierSet.cpp
@@ -27,6 +27,7 @@
#include "gc/g1/g1BarrierSetAssembler.hpp"
#include "gc/g1/g1CardTable.inline.hpp"
#include "gc/g1/g1CollectedHeap.inline.hpp"
+#include "gc/g1/g1RegionPinCache.inline.hpp"
#include "gc/g1/g1SATBMarkQueueSet.hpp"
#include "gc/g1/g1ThreadLocalData.hpp"
#include "gc/g1/heapRegion.hpp"
@@ -171,4 +172,8 @@ void G1BarrierSet::on_thread_detach(Thread* thread) {
qset.flush_queue(queue);
qset.record_detached_refinement_stats(queue.refinement_stats());
}
+ {
+ G1RegionPinCache& cache = G1ThreadLocalData::pin_count_cache(thread);
+ cache.flush();
+ }
}
diff --git a/src/hotspot/share/gc/g1/g1CardSet.cpp b/src/hotspot/share/gc/g1/g1CardSet.cpp
index 58ae441f51f..e6b2d5029e5 100644
--- a/src/hotspot/share/gc/g1/g1CardSet.cpp
+++ b/src/hotspot/share/gc/g1/g1CardSet.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2021, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2021, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -286,7 +286,10 @@ class G1CardSetHashTable : public CHeapObj {
size_t initial_log_table_size = InitialLogTableSize) :
_inserted_card(false),
_mm(mm),
- _table(mm, initial_log_table_size, false),
+ _table(Mutex::service-1,
+ mm,
+ initial_log_table_size,
+ false /* enable_statistics */),
_table_scanner(&_table, BucketClaimSize) {
}
diff --git a/src/hotspot/share/gc/g1/g1CodeRootSet.cpp b/src/hotspot/share/gc/g1/g1CodeRootSet.cpp
index 3b933dfa7f9..a1eb7b8543a 100644
--- a/src/hotspot/share/gc/g1/g1CodeRootSet.cpp
+++ b/src/hotspot/share/gc/g1/g1CodeRootSet.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -84,8 +84,10 @@ class G1CodeRootSetHashTable : public CHeapObj {
public:
G1CodeRootSetHashTable() :
- _table(Log2DefaultNumBuckets,
- HashTable::DEFAULT_MAX_SIZE_LOG2),
+ _table(Mutex::service-1,
+ nullptr,
+ Log2DefaultNumBuckets,
+ false /* enable_statistics */),
_table_scanner(&_table, BucketClaimSize), _num_entries(0) {
clear();
}
diff --git a/src/hotspot/share/gc/g1/g1CollectedHeap.cpp b/src/hotspot/share/gc/g1/g1CollectedHeap.cpp
index 7a630c27f79..f4453a7ba0b 100644
--- a/src/hotspot/share/gc/g1/g1CollectedHeap.cpp
+++ b/src/hotspot/share/gc/g1/g1CollectedHeap.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -27,7 +27,6 @@
#include "classfile/metadataOnStackMark.hpp"
#include "classfile/systemDictionary.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "compiler/oopMap.hpp"
#include "gc/g1/g1Allocator.inline.hpp"
#include "gc/g1/g1Arguments.hpp"
@@ -59,6 +58,7 @@
#include "gc/g1/g1PeriodicGCTask.hpp"
#include "gc/g1/g1Policy.hpp"
#include "gc/g1/g1RedirtyCardsQueue.hpp"
+#include "gc/g1/g1RegionPinCache.inline.hpp"
#include "gc/g1/g1RegionToSpaceMapper.hpp"
#include "gc/g1/g1RemSet.hpp"
#include "gc/g1/g1RootClosures.hpp"
@@ -1136,7 +1136,6 @@ G1CollectedHeap::G1CollectedHeap() :
_workers(nullptr),
_card_table(nullptr),
_collection_pause_end(Ticks::now()),
- _soft_ref_policy(),
_old_set("Old Region Set", new OldRegionSetChecker()),
_humongous_set("Humongous Region Set", new HumongousRegionSetChecker()),
_bot(nullptr),
@@ -1525,10 +1524,6 @@ void G1CollectedHeap::ref_processing_init() {
&_is_alive_closure_stw); // is alive closure
}
-SoftRefPolicy* G1CollectedHeap::soft_ref_policy() {
- return &_soft_ref_policy;
-}
-
size_t G1CollectedHeap::capacity() const {
return _hrm.length() * HeapRegion::GrainBytes;
}
@@ -2202,8 +2197,6 @@ void G1CollectedHeap::trace_heap(GCWhen::Type when, const GCTracer* gc_tracer) {
}
void G1CollectedHeap::gc_prologue(bool full) {
- assert(InlineCacheBuffer::is_empty(), "should have cleaned up ICBuffer");
-
// Update common counters.
increment_total_collections(full /* full gc */);
if (full || collector_state()->in_concurrent_start_gc()) {
@@ -2471,6 +2464,12 @@ void G1CollectedHeap::retire_tlabs() {
ensure_parsability(true);
}
+void G1CollectedHeap::flush_region_pin_cache() {
+ for (JavaThreadIteratorWithHandle jtiwh; JavaThread *thread = jtiwh.next(); ) {
+ G1ThreadLocalData::pin_count_cache(thread).flush();
+ }
+}
+
void G1CollectedHeap::do_collection_pause_at_safepoint_helper() {
ResourceMark rm;
diff --git a/src/hotspot/share/gc/g1/g1CollectedHeap.hpp b/src/hotspot/share/gc/g1/g1CollectedHeap.hpp
index 58ac36a734f..189a09fe042 100644
--- a/src/hotspot/share/gc/g1/g1CollectedHeap.hpp
+++ b/src/hotspot/share/gc/g1/g1CollectedHeap.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -178,8 +178,6 @@ class G1CollectedHeap : public CollectedHeap {
Ticks _collection_pause_end;
- SoftRefPolicy _soft_ref_policy;
-
static size_t _humongous_object_threshold_in_words;
// These sets keep track of old and humongous regions respectively.
@@ -775,6 +773,10 @@ class G1CollectedHeap : public CollectedHeap {
void retire_tlabs();
+ // Update all region's pin counts from the per-thread caches and resets them.
+ // Must be called before any decision based on pin counts.
+ void flush_region_pin_cache();
+
void expand_heap_after_young_collection();
// Update object copying statistics.
void record_obj_copy_mem_stats();
@@ -923,8 +925,6 @@ class G1CollectedHeap : public CollectedHeap {
inline bool is_collection_set_candidate(const HeapRegion* r) const;
- SoftRefPolicy* soft_ref_policy() override;
-
void initialize_serviceability() override;
MemoryUsage memory_usage() override;
GrowableArray memory_managers() override;
diff --git a/src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp b/src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp
index 7b81a93abe1..537b6863fb9 100644
--- a/src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp
+++ b/src/hotspot/share/gc/g1/g1CollectedHeap.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -32,6 +32,7 @@
#include "gc/g1/g1ConcurrentMark.inline.hpp"
#include "gc/g1/g1EvacFailureRegions.hpp"
#include "gc/g1/g1Policy.hpp"
+#include "gc/g1/g1RegionPinCache.inline.hpp"
#include "gc/g1/g1RemSet.hpp"
#include "gc/g1/heapRegion.inline.hpp"
#include "gc/g1/heapRegionManager.inline.hpp"
@@ -266,15 +267,17 @@ inline void G1CollectedHeap::pin_object(JavaThread* thread, oop obj) {
assert(obj != nullptr, "obj must not be null");
assert(!is_gc_active(), "must not pin objects during a GC");
assert(obj->is_typeArray(), "must be typeArray");
- HeapRegion *r = heap_region_containing(obj);
- r->increment_pinned_object_count();
+
+ uint obj_region_idx = heap_region_containing(obj)->hrm_index();
+ G1ThreadLocalData::pin_count_cache(thread).inc_count(obj_region_idx);
}
inline void G1CollectedHeap::unpin_object(JavaThread* thread, oop obj) {
assert(obj != nullptr, "obj must not be null");
assert(!is_gc_active(), "must not unpin objects during a GC");
- HeapRegion *r = heap_region_containing(obj);
- r->decrement_pinned_object_count();
+
+ uint obj_region_idx = heap_region_containing(obj)->hrm_index();
+ G1ThreadLocalData::pin_count_cache(thread).dec_count(obj_region_idx);
}
inline bool G1CollectedHeap::is_obj_dead(const oop obj) const {
diff --git a/src/hotspot/share/gc/g1/g1ConcurrentMark.cpp b/src/hotspot/share/gc/g1/g1ConcurrentMark.cpp
index 533817f9724..cb7f90b0847 100644
--- a/src/hotspot/share/gc/g1/g1ConcurrentMark.cpp
+++ b/src/hotspot/share/gc/g1/g1ConcurrentMark.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
diff --git a/src/hotspot/share/gc/g1/g1FullCollector.cpp b/src/hotspot/share/gc/g1/g1FullCollector.cpp
index 6ae43c770cc..430d6f32788 100644
--- a/src/hotspot/share/gc/g1/g1FullCollector.cpp
+++ b/src/hotspot/share/gc/g1/g1FullCollector.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -120,8 +120,8 @@ G1FullCollector::G1FullCollector(G1CollectedHeap* heap,
_oop_queue_set(_num_workers),
_array_queue_set(_num_workers),
_preserved_marks_set(true),
- _serial_compaction_point(this),
- _humongous_compaction_point(this),
+ _serial_compaction_point(this, nullptr),
+ _humongous_compaction_point(this, nullptr),
_is_alive(this, heap->concurrent_mark()->mark_bitmap()),
_is_alive_mutator(heap->ref_processor_stw(), &_is_alive),
_humongous_compaction_regions(8),
@@ -142,11 +142,13 @@ G1FullCollector::G1FullCollector(G1CollectedHeap* heap,
}
for (uint i = 0; i < _num_workers; i++) {
- _markers[i] = new G1FullGCMarker(this, i, _preserved_marks_set.get(i), _live_stats);
- _compaction_points[i] = new G1FullGCCompactionPoint(this);
+ _markers[i] = new G1FullGCMarker(this, i, _live_stats);
+ _compaction_points[i] = new G1FullGCCompactionPoint(this, _preserved_marks_set.get(i));
_oop_queue_set.register_queue(i, marker(i)->oop_stack());
_array_queue_set.register_queue(i, marker(i)->objarray_stack());
}
+ _serial_compaction_point.set_preserved_stack(_preserved_marks_set.get(0));
+ _humongous_compaction_point.set_preserved_stack(_preserved_marks_set.get(0));
_region_attr_table.initialize(heap->reserved(), HeapRegion::GrainBytes);
}
@@ -189,6 +191,7 @@ void G1FullCollector::prepare_collection() {
_heap->gc_prologue(true);
_heap->retire_tlabs();
+ _heap->flush_region_pin_cache();
_heap->prepare_heap_for_full_collection();
PrepareRegionsClosure cl(this);
diff --git a/src/hotspot/share/gc/g1/g1FullGCAdjustTask.cpp b/src/hotspot/share/gc/g1/g1FullGCAdjustTask.cpp
index a45c3eb17de..76aebeb7a83 100644
--- a/src/hotspot/share/gc/g1/g1FullGCAdjustTask.cpp
+++ b/src/hotspot/share/gc/g1/g1FullGCAdjustTask.cpp
@@ -91,8 +91,8 @@ void G1FullGCAdjustTask::work(uint worker_id) {
ResourceMark rm;
// Adjust preserved marks first since they are not balanced.
- G1FullGCMarker* marker = collector()->marker(worker_id);
- marker->preserved_stack()->adjust_during_full_gc();
+ G1FullGCCompactionPoint* cp = collector()->compaction_point(worker_id);
+ cp->preserved_stack()->adjust_during_full_gc();
{
// Adjust the weak roots.
diff --git a/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.cpp b/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.cpp
index 7669771eb6c..a1db2aa87ba 100644
--- a/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.cpp
+++ b/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.cpp
@@ -30,10 +30,11 @@
#include "oops/oop.inline.hpp"
#include "utilities/debug.hpp"
-G1FullGCCompactionPoint::G1FullGCCompactionPoint(G1FullCollector* collector) :
+G1FullGCCompactionPoint::G1FullGCCompactionPoint(G1FullCollector* collector, PreservedMarks* preserved_stack) :
_collector(collector),
_current_region(nullptr),
- _compaction_top(nullptr) {
+ _compaction_top(nullptr),
+ _preserved_stack(preserved_stack) {
_compaction_regions = new (mtGC) GrowableArray(32, mtGC);
_compaction_region_iterator = _compaction_regions->begin();
}
@@ -102,6 +103,9 @@ void G1FullGCCompactionPoint::forward(oop object, size_t size) {
// Store a forwarding pointer if the object should be moved.
if (cast_from_oop(object) != _compaction_top) {
+ if (!object->is_forwarded()) {
+ preserved_stack()->push_if_necessary(object, object->mark());
+ }
object->forward_to(cast_to_oop(_compaction_top));
assert(object->is_forwarded(), "must be forwarded");
} else {
@@ -165,7 +169,7 @@ void G1FullGCCompactionPoint::forward_humongous(HeapRegion* hr) {
}
// Preserve the mark for the humongous object as the region was initially not compacting.
- _collector->marker(0)->preserved_stack()->push_if_necessary(obj, obj->mark());
+ preserved_stack()->push_if_necessary(obj, obj->mark());
HeapRegion* dest_hr = _compaction_regions->at(range_begin);
obj->forward_to(cast_to_oop(dest_hr->bottom()));
diff --git a/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.hpp b/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.hpp
index ca76f7e6b94..6d87e44b213 100644
--- a/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.hpp
+++ b/src/hotspot/share/gc/g1/g1FullGCCompactionPoint.hpp
@@ -32,11 +32,13 @@
class G1FullCollector;
class HeapRegion;
+class PreservedMarks;
class G1FullGCCompactionPoint : public CHeapObj {
G1FullCollector* _collector;
HeapRegion* _current_region;
HeapWord* _compaction_top;
+ PreservedMarks* _preserved_stack;
GrowableArray* _compaction_regions;
GrowableArrayIterator _compaction_region_iterator;
@@ -47,7 +49,7 @@ class G1FullGCCompactionPoint : public CHeapObj {
uint find_contiguous_before(HeapRegion* hr, uint num_regions);
public:
- G1FullGCCompactionPoint(G1FullCollector* collector);
+ G1FullGCCompactionPoint(G1FullCollector* collector, PreservedMarks* preserved_stack);
~G1FullGCCompactionPoint();
bool has_regions();
@@ -63,6 +65,16 @@ class G1FullGCCompactionPoint : public CHeapObj {
HeapRegion* current_region();
GrowableArray* regions();
+
+ PreservedMarks* preserved_stack() const {
+ assert(_preserved_stack != nullptr, "must be initialized");
+ return _preserved_stack;
+ }
+
+ void set_preserved_stack(PreservedMarks* preserved_stack) {
+ assert(_preserved_stack == nullptr, "only initialize once");
+ _preserved_stack = preserved_stack;
+ }
};
#endif // SHARE_GC_G1_G1FULLGCCOMPACTIONPOINT_HPP
diff --git a/src/hotspot/share/gc/g1/g1FullGCMarker.cpp b/src/hotspot/share/gc/g1/g1FullGCMarker.cpp
index ac71b82d04a..b98bb41240c 100644
--- a/src/hotspot/share/gc/g1/g1FullGCMarker.cpp
+++ b/src/hotspot/share/gc/g1/g1FullGCMarker.cpp
@@ -33,14 +33,12 @@
G1FullGCMarker::G1FullGCMarker(G1FullCollector* collector,
uint worker_id,
- PreservedMarks* preserved_stack,
G1RegionMarkStats* mark_stats) :
_collector(collector),
_worker_id(worker_id),
_bitmap(collector->mark_bitmap()),
_oop_stack(),
_objarray_stack(),
- _preserved_stack(preserved_stack),
_mark_closure(worker_id, this, ClassLoaderData::_claim_stw_fullgc_mark, G1CollectedHeap::heap()->ref_processor_stw()),
_stack_closure(this),
_cld_closure(mark_closure(), ClassLoaderData::_claim_stw_fullgc_mark),
diff --git a/src/hotspot/share/gc/g1/g1FullGCMarker.hpp b/src/hotspot/share/gc/g1/g1FullGCMarker.hpp
index cfe9d125e20..3dcc746a26e 100644
--- a/src/hotspot/share/gc/g1/g1FullGCMarker.hpp
+++ b/src/hotspot/share/gc/g1/g1FullGCMarker.hpp
@@ -28,7 +28,6 @@
#include "gc/g1/g1FullGCOopClosures.hpp"
#include "gc/g1/g1OopClosures.hpp"
#include "gc/g1/g1RegionMarkStatsCache.hpp"
-#include "gc/shared/preservedMarks.hpp"
#include "gc/shared/stringdedup/stringDedup.hpp"
#include "gc/shared/taskqueue.hpp"
#include "memory/iterator.hpp"
@@ -59,7 +58,6 @@ class G1FullGCMarker : public CHeapObj {
// Mark stack
OopQueue _oop_stack;
ObjArrayTaskQueue _objarray_stack;
- PreservedMarks* _preserved_stack;
// Marking closures
G1MarkAndPushClosure _mark_closure;
@@ -89,14 +87,12 @@ class G1FullGCMarker : public CHeapObj {
public:
G1FullGCMarker(G1FullCollector* collector,
uint worker_id,
- PreservedMarks* preserved_stack,
G1RegionMarkStats* mark_stats);
~G1FullGCMarker();
// Stack getters
OopQueue* oop_stack() { return &_oop_stack; }
ObjArrayTaskQueue* objarray_stack() { return &_objarray_stack; }
- PreservedMarks* preserved_stack() { return _preserved_stack; }
// Marking entry points
template inline void mark_and_push(T* p);
diff --git a/src/hotspot/share/gc/g1/g1FullGCMarker.inline.hpp b/src/hotspot/share/gc/g1/g1FullGCMarker.inline.hpp
index 8080a1d113e..646cf9e3df5 100644
--- a/src/hotspot/share/gc/g1/g1FullGCMarker.inline.hpp
+++ b/src/hotspot/share/gc/g1/g1FullGCMarker.inline.hpp
@@ -51,13 +51,6 @@ inline bool G1FullGCMarker::mark_object(oop obj) {
return false;
}
- // Marked by us, preserve if needed.
- if (_collector->is_compacting(obj)) {
- // It is not necessary to preserve marks for objects in regions we do not
- // compact because we do not change their headers (i.e. forward them).
- preserved_stack()->push_if_necessary(obj, obj->mark());
- }
-
// Check if deduplicatable string.
if (StringDedup::is_enabled() &&
java_lang_String::is_instance(obj) &&
diff --git a/src/hotspot/share/gc/g1/g1MonitoringSupport.hpp b/src/hotspot/share/gc/g1/g1MonitoringSupport.hpp
index 769a6fe831c..a1b9d6c985b 100644
--- a/src/hotspot/share/gc/g1/g1MonitoringSupport.hpp
+++ b/src/hotspot/share/gc/g1/g1MonitoringSupport.hpp
@@ -180,8 +180,6 @@ class G1MonitoringSupport : public CHeapObj {
// Recalculate all the sizes.
void recalculate_sizes();
- void recalculate_eden_size();
-
public:
G1MonitoringSupport(G1CollectedHeap* g1h);
~G1MonitoringSupport();
diff --git a/src/hotspot/share/gc/g1/g1RegionMarkStatsCache.hpp b/src/hotspot/share/gc/g1/g1RegionMarkStatsCache.hpp
index d3236f58fae..f74453b81ce 100644
--- a/src/hotspot/share/gc/g1/g1RegionMarkStatsCache.hpp
+++ b/src/hotspot/share/gc/g1/g1RegionMarkStatsCache.hpp
@@ -48,8 +48,6 @@ struct G1RegionMarkStats {
// are updated by the atomic mark. We do not remark objects after overflow.
void clear_during_overflow() {
}
-
- bool is_clear() const { return _live_words == 0; }
};
// Per-marking thread cache for the region mark statistics.
diff --git a/src/hotspot/share/gc/g1/g1RegionPinCache.hpp b/src/hotspot/share/gc/g1/g1RegionPinCache.hpp
new file mode 100644
index 00000000000..6df1c1793c2
--- /dev/null
+++ b/src/hotspot/share/gc/g1/g1RegionPinCache.hpp
@@ -0,0 +1,57 @@
+/*
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ *
+ */
+
+#ifndef SHARE_GC_G1_G1REGIONPINCACHE_HPP
+#define SHARE_GC_G1_G1REGIONPINCACHE_HPP
+
+#include "gc/g1/heapRegion.hpp"
+#include "memory/allocation.hpp"
+#include "utilities/globalDefinitions.hpp"
+
+// Holds (caches) the pending pinned object count adjustment for the region
+// _region_idx on a per thread basis.
+// Keeping such a cache avoids the expensive atomic operations when updating the
+// pin count for the very common case that the application pins and unpins the
+// same object without any interleaving by a garbage collection or pinning/unpinning
+// to an object in another region.
+class G1RegionPinCache : public StackObj {
+ uint _region_idx;
+ size_t _count;
+
+ void flush_and_set(uint new_region_idx, size_t new_count);
+
+public:
+ G1RegionPinCache() : _region_idx(G1_NO_HRM_INDEX), _count(0) { }
+
+#ifdef ASSERT
+ size_t count() const { return _count; }
+#endif
+
+ void inc_count(uint region_idx);
+ void dec_count(uint region_idx);
+
+ void flush();
+};
+
+#endif /* SHARE_GC_G1_G1REGIONPINCACHE_HPP */
diff --git a/src/hotspot/share/gc/g1/g1RegionPinCache.inline.hpp b/src/hotspot/share/gc/g1/g1RegionPinCache.inline.hpp
new file mode 100644
index 00000000000..7df81d208f1
--- /dev/null
+++ b/src/hotspot/share/gc/g1/g1RegionPinCache.inline.hpp
@@ -0,0 +1,60 @@
+/*
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ *
+ */
+
+#ifndef SHARE_GC_G1_G1REGIONPINCACHE_INLINE_HPP
+#define SHARE_GC_G1_G1REGIONPINCACHE_INLINE_HPP
+
+#include "gc/g1/g1RegionPinCache.hpp"
+
+#include "gc/g1/g1CollectedHeap.inline.hpp"
+
+inline void G1RegionPinCache::inc_count(uint region_idx) {
+ if (region_idx == _region_idx) {
+ ++_count;
+ } else {
+ flush_and_set(region_idx, (size_t)1);
+ }
+}
+
+inline void G1RegionPinCache::dec_count(uint region_idx) {
+ if (region_idx == _region_idx) {
+ --_count;
+ } else {
+ flush_and_set(region_idx, ~(size_t)0);
+ }
+}
+
+inline void G1RegionPinCache::flush_and_set(uint new_region_idx, size_t new_count) {
+ if (_count != 0) {
+ G1CollectedHeap::heap()->region_at(_region_idx)->add_pinned_object_count(_count);
+ }
+ _region_idx = new_region_idx;
+ _count = new_count;
+}
+
+inline void G1RegionPinCache::flush() {
+ flush_and_set(G1_NO_HRM_INDEX, 0);
+}
+
+#endif /* SHARE_GC_G1_G1REGIONPINCACHE_INLINE_HPP */
diff --git a/src/hotspot/share/gc/g1/g1RemSet.cpp b/src/hotspot/share/gc/g1/g1RemSet.cpp
index ac445850085..d26a77837d7 100644
--- a/src/hotspot/share/gc/g1/g1RemSet.cpp
+++ b/src/hotspot/share/gc/g1/g1RemSet.cpp
@@ -1174,24 +1174,27 @@ class G1MergeHeapRootsTask : public WorkerTask {
}
HeapRegion* r = g1h->region_at(region_index);
- if (r->rem_set()->is_empty()) {
- return false;
- }
+
+ assert(r->rem_set()->is_complete(), "humongous candidates must have complete remset");
guarantee(r->rem_set()->occupancy_less_or_equal_than(G1EagerReclaimRemSetThreshold),
"Found a not-small remembered set here. This is inconsistent with previous assumptions.");
- _cl.merge_card_set_for_region(r);
+ if (!r->rem_set()->is_empty()) {
+ _cl.merge_card_set_for_region(r);
- // We should only clear the card based remembered set here as we will not
- // implicitly rebuild anything else during eager reclaim. Note that at the moment
- // (and probably never) we do not enter this path if there are other kind of
- // remembered sets for this region.
- // We want to continue collecting remembered set entries for humongous regions
- // that were not reclaimed.
- r->rem_set()->clear(true /* only_cardset */, true /* keep_tracked */);
+ // We should only clear the card based remembered set here as we will not
+ // implicitly rebuild anything else during eager reclaim. Note that at the moment
+ // (and probably never) we do not enter this path if there are other kind of
+ // remembered sets for this region.
+ // We want to continue collecting remembered set entries for humongous regions
+ // that were not reclaimed.
+ r->rem_set()->clear(true /* only_cardset */, true /* keep_tracked */);
+ }
- assert(r->rem_set()->is_empty() && r->rem_set()->is_complete(), "must be for eager reclaim candidates");
+ // Postcondition
+ assert(r->rem_set()->is_empty(), "must be empty after flushing");
+ assert(r->rem_set()->is_complete(), "should still be after flushing");
return false;
}
diff --git a/src/hotspot/share/gc/g1/g1ThreadLocalData.hpp b/src/hotspot/share/gc/g1/g1ThreadLocalData.hpp
index e17a8c7960a..d0dcb59d7f0 100644
--- a/src/hotspot/share/gc/g1/g1ThreadLocalData.hpp
+++ b/src/hotspot/share/gc/g1/g1ThreadLocalData.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2018, 2021, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -26,6 +26,7 @@
#include "gc/g1/g1BarrierSet.hpp"
#include "gc/g1/g1DirtyCardQueue.hpp"
+#include "gc/g1/g1RegionPinCache.hpp"
#include "gc/shared/gc_globals.hpp"
#include "gc/shared/satbMarkQueue.hpp"
#include "runtime/javaThread.hpp"
@@ -37,9 +38,15 @@ class G1ThreadLocalData {
SATBMarkQueue _satb_mark_queue;
G1DirtyCardQueue _dirty_card_queue;
+ // Per-thread cache of pinned object count to reduce atomic operation traffic
+ // due to region pinning. Holds the last region where the mutator pinned an
+ // object and the number of pin operations since the last change of the region.
+ G1RegionPinCache _pin_cache;
+
G1ThreadLocalData() :
_satb_mark_queue(&G1BarrierSet::satb_mark_queue_set()),
- _dirty_card_queue(&G1BarrierSet::dirty_card_queue_set()) {}
+ _dirty_card_queue(&G1BarrierSet::dirty_card_queue_set()),
+ _pin_cache() {}
static G1ThreadLocalData* data(Thread* thread) {
assert(UseG1GC, "Sanity");
@@ -90,6 +97,10 @@ class G1ThreadLocalData {
static ByteSize dirty_card_queue_buffer_offset() {
return dirty_card_queue_offset() + G1DirtyCardQueue::byte_offset_of_buf();
}
+
+ static G1RegionPinCache& pin_count_cache(Thread* thread) {
+ return data(thread)->_pin_cache;
+ }
};
#endif // SHARE_GC_G1_G1THREADLOCALDATA_HPP
diff --git a/src/hotspot/share/gc/g1/g1YoungCollector.cpp b/src/hotspot/share/gc/g1/g1YoungCollector.cpp
index b52bfdfbb08..4fa3b928763 100644
--- a/src/hotspot/share/gc/g1/g1YoungCollector.cpp
+++ b/src/hotspot/share/gc/g1/g1YoungCollector.cpp
@@ -41,6 +41,7 @@
#include "gc/g1/g1ParScanThreadState.inline.hpp"
#include "gc/g1/g1Policy.hpp"
#include "gc/g1/g1RedirtyCardsQueue.hpp"
+#include "gc/g1/g1RegionPinCache.inline.hpp"
#include "gc/g1/g1RemSet.hpp"
#include "gc/g1/g1RootProcessor.hpp"
#include "gc/g1/g1Trace.hpp"
diff --git a/src/hotspot/share/gc/g1/g1YoungGCPreEvacuateTasks.cpp b/src/hotspot/share/gc/g1/g1YoungGCPreEvacuateTasks.cpp
index d3a6436e432..d7460623669 100644
--- a/src/hotspot/share/gc/g1/g1YoungGCPreEvacuateTasks.cpp
+++ b/src/hotspot/share/gc/g1/g1YoungGCPreEvacuateTasks.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2023, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -28,6 +28,8 @@
#include "gc/g1/g1ConcurrentRefineStats.hpp"
#include "gc/g1/g1DirtyCardQueue.hpp"
#include "gc/g1/g1YoungGCPreEvacuateTasks.hpp"
+#include "gc/g1/g1RegionPinCache.inline.hpp"
+#include "gc/g1/g1ThreadLocalData.hpp"
#include "gc/shared/barrierSet.inline.hpp"
#include "gc/shared/threadLocalAllocBuffer.inline.hpp"
#include "memory/allocation.inline.hpp"
@@ -57,12 +59,15 @@ class G1PreEvacuateCollectionSetBatchTask::JavaThreadRetireTLABAndFlushLogs : pu
assert(thread->is_Java_thread(), "must be");
// Flushes deferred card marks, so must precede concatenating logs.
BarrierSet::barrier_set()->make_parsable((JavaThread*)thread);
+ // Retire TLABs.
if (UseTLAB) {
thread->tlab().retire(&_tlab_stats);
}
-
+ // Concatenate logs.
G1DirtyCardQueueSet& qset = G1BarrierSet::dirty_card_queue_set();
_refinement_stats += qset.concatenate_log_and_stats(thread);
+ // Flush region pin count cache.
+ G1ThreadLocalData::pin_count_cache(thread).flush();
}
};
@@ -132,6 +137,8 @@ class G1PreEvacuateCollectionSetBatchTask::NonJavaThreadFlushLogs : public G1Abs
void do_thread(Thread* thread) override {
G1DirtyCardQueueSet& qset = G1BarrierSet::dirty_card_queue_set();
_refinement_stats += qset.concatenate_log_and_stats(thread);
+
+ assert(G1ThreadLocalData::pin_count_cache(thread).count() == 0, "NonJava thread has pinned Java objects");
}
} _tc;
diff --git a/src/hotspot/share/gc/g1/heapRegion.hpp b/src/hotspot/share/gc/g1/heapRegion.hpp
index 725433215c4..85f4a9e5f9f 100644
--- a/src/hotspot/share/gc/g1/heapRegion.hpp
+++ b/src/hotspot/share/gc/g1/heapRegion.hpp
@@ -302,8 +302,9 @@ class HeapRegion : public CHeapObj {
static uint LogOfHRGrainBytes;
static uint LogCardsPerRegion;
- inline void increment_pinned_object_count();
- inline void decrement_pinned_object_count();
+ // Atomically adjust the pinned object count by the given value. Value must not
+ // be zero.
+ inline void add_pinned_object_count(size_t value);
static size_t GrainBytes;
static size_t GrainWords;
diff --git a/src/hotspot/share/gc/g1/heapRegion.inline.hpp b/src/hotspot/share/gc/g1/heapRegion.inline.hpp
index 4a42b922182..608cc2025a2 100644
--- a/src/hotspot/share/gc/g1/heapRegion.inline.hpp
+++ b/src/hotspot/share/gc/g1/heapRegion.inline.hpp
@@ -553,12 +553,10 @@ inline void HeapRegion::record_surv_words_in_group(size_t words_survived) {
_surv_rate_group->record_surviving_words(age, words_survived);
}
-inline void HeapRegion::increment_pinned_object_count() {
- Atomic::add(&_pinned_object_count, (size_t)1, memory_order_relaxed);
-}
-
-inline void HeapRegion::decrement_pinned_object_count() {
- Atomic::sub(&_pinned_object_count, (size_t)1, memory_order_relaxed);
+inline void HeapRegion::add_pinned_object_count(size_t value) {
+ assert(value != 0, "wasted effort");
+ assert(!is_free(), "trying to pin free region %u, adding %zu", hrm_index(), value);
+ Atomic::add(&_pinned_object_count, value, memory_order_relaxed);
}
#endif // SHARE_GC_G1_HEAPREGION_INLINE_HPP
diff --git a/src/hotspot/share/gc/parallel/mutableSpace.cpp b/src/hotspot/share/gc/parallel/mutableSpace.cpp
index 73329aa6d33..c19ca962162 100644
--- a/src/hotspot/share/gc/parallel/mutableSpace.cpp
+++ b/src/hotspot/share/gc/parallel/mutableSpace.cpp
@@ -120,11 +120,12 @@ void MutableSpace::initialize(MemRegion mr,
}
if (AlwaysPreTouch) {
+ size_t pretouch_page_size = UseLargePages ? page_size : os::vm_page_size();
PretouchTask::pretouch("ParallelGC PreTouch head", (char*)head.start(), (char*)head.end(),
- page_size, pretouch_workers);
+ pretouch_page_size, pretouch_workers);
PretouchTask::pretouch("ParallelGC PreTouch tail", (char*)tail.start(), (char*)tail.end(),
- page_size, pretouch_workers);
+ pretouch_page_size, pretouch_workers);
}
// Remember where we stopped so that we can continue later.
diff --git a/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp b/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
index 2531ecb2d5c..64d3a8d4a10 100644
--- a/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
+++ b/src/hotspot/share/gc/parallel/parallelScavengeHeap.cpp
@@ -102,7 +102,6 @@ jint ParallelScavengeHeap::initialize() {
assert(old_gen()->max_gen_size() == old_rs.size(), "Consistency check");
double max_gc_pause_sec = ((double) MaxGCPauseMillis)/1000.0;
- double max_gc_minor_pause_sec = ((double) MaxGCMinorPauseMillis)/1000.0;
const size_t eden_capacity = _young_gen->eden_space()->capacity_in_bytes();
const size_t old_capacity = _old_gen->capacity_in_bytes();
@@ -113,7 +112,6 @@ jint ParallelScavengeHeap::initialize() {
young_gen()->to_space()->capacity_in_bytes(),
GenAlignment,
max_gc_pause_sec,
- max_gc_minor_pause_sec,
GCTimeRatio
);
diff --git a/src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp b/src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp
index eec32a2c1c2..0b8ff00e2cb 100644
--- a/src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp
+++ b/src/hotspot/share/gc/parallel/parallelScavengeHeap.hpp
@@ -80,8 +80,6 @@ class ParallelScavengeHeap : public CollectedHeap {
static PSAdaptiveSizePolicy* _size_policy;
static PSGCAdaptivePolicyCounters* _gc_policy_counters;
- SoftRefPolicy _soft_ref_policy;
-
unsigned int _death_march_count;
GCMemoryManager* _young_manager;
@@ -135,8 +133,6 @@ class ParallelScavengeHeap : public CollectedHeap {
return "Parallel";
}
- SoftRefPolicy* soft_ref_policy() override { return &_soft_ref_policy; }
-
GrowableArray memory_managers() override;
GrowableArray memory_pools() override;
diff --git a/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.cpp b/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.cpp
index 15287437cff..c8a15c2cb92 100644
--- a/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.cpp
+++ b/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.cpp
@@ -41,7 +41,6 @@ PSAdaptiveSizePolicy::PSAdaptiveSizePolicy(size_t init_eden_size,
size_t init_survivor_size,
size_t space_alignment,
double gc_pause_goal_sec,
- double gc_minor_pause_goal_sec,
uint gc_cost_ratio) :
AdaptiveSizePolicy(init_eden_size,
init_promo_size,
@@ -55,7 +54,6 @@ PSAdaptiveSizePolicy::PSAdaptiveSizePolicy(size_t init_eden_size,
_major_pause_young_estimator(new LinearLeastSquareFit(AdaptiveSizePolicyWeight)),
_latest_major_mutator_interval_seconds(0),
_space_alignment(space_alignment),
- _gc_minor_pause_goal_sec(gc_minor_pause_goal_sec),
_live_at_last_full_gc(init_promo_size),
_change_old_gen_for_min_pauses(0),
_change_young_gen_for_maj_pauses(0),
@@ -282,7 +280,7 @@ void PSAdaptiveSizePolicy::compute_eden_space_size(
// at a time.
adjust_eden_for_pause_time(&desired_eden_size);
- } else if (_avg_minor_pause->padded_average() > gc_minor_pause_goal_sec()) {
+ } else if (_avg_minor_pause->padded_average() > gc_pause_goal_sec()) {
// Adjust only for the minor pause time goal
adjust_eden_for_minor_pause_time(&desired_eden_size);
diff --git a/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp b/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp
index f6d4be029fb..997a4784667 100644
--- a/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp
+++ b/src/hotspot/share/gc/parallel/psAdaptiveSizePolicy.hpp
@@ -91,8 +91,6 @@ class PSAdaptiveSizePolicy : public AdaptiveSizePolicy {
const size_t _space_alignment; // alignment for eden, survivors
- const double _gc_minor_pause_goal_sec; // goal for maximum minor gc pause
-
// The amount of live data in the heap at the last full GC, used
// as a baseline to help us determine when we need to perform the
// next full GC.
@@ -113,9 +111,6 @@ class PSAdaptiveSizePolicy : public AdaptiveSizePolicy {
private:
- // Accessors
- double gc_minor_pause_goal_sec() const { return _gc_minor_pause_goal_sec; }
-
void adjust_eden_for_minor_pause_time(size_t* desired_eden_size_ptr);
// Change the generation sizes to achieve a GC pause time goal
// Returned sizes are not necessarily aligned.
@@ -187,7 +182,6 @@ class PSAdaptiveSizePolicy : public AdaptiveSizePolicy {
size_t init_survivor_size,
size_t space_alignment,
double gc_pause_goal_sec,
- double gc_minor_pause_goal_sec,
uint gc_time_ratio);
// Methods indicating events of interest to the adaptive size policy,
diff --git a/src/hotspot/share/gc/parallel/psCardTable.cpp b/src/hotspot/share/gc/parallel/psCardTable.cpp
index 91fcf026069..345d81ed147 100644
--- a/src/hotspot/share/gc/parallel/psCardTable.cpp
+++ b/src/hotspot/share/gc/parallel/psCardTable.cpp
@@ -38,13 +38,11 @@
// Checks an individual oop for missing precise marks. Mark
// may be either dirty or newgen.
-class CheckForUnmarkedOops : public BasicOopIterateClosure {
- private:
+class PSCheckForUnmarkedOops : public BasicOopIterateClosure {
PSYoungGen* _young_gen;
PSCardTable* _card_table;
HeapWord* _unmarked_addr;
- protected:
template void do_oop_work(T* p) {
oop obj = RawAccess<>::oop_load(p);
if (_young_gen->is_in_reserved(obj) &&
@@ -57,11 +55,11 @@ class CheckForUnmarkedOops : public BasicOopIterateClosure {
}
public:
- CheckForUnmarkedOops(PSYoungGen* young_gen, PSCardTable* card_table) :
+ PSCheckForUnmarkedOops(PSYoungGen* young_gen, PSCardTable* card_table) :
_young_gen(young_gen), _card_table(card_table), _unmarked_addr(nullptr) { }
- virtual void do_oop(oop* p) { CheckForUnmarkedOops::do_oop_work(p); }
- virtual void do_oop(narrowOop* p) { CheckForUnmarkedOops::do_oop_work(p); }
+ void do_oop(oop* p) override { do_oop_work(p); }
+ void do_oop(narrowOop* p) override { do_oop_work(p); }
bool has_unmarked_oop() {
return _unmarked_addr != nullptr;
@@ -70,13 +68,13 @@ class CheckForUnmarkedOops : public BasicOopIterateClosure {
// Checks all objects for the existence of some type of mark,
// precise or imprecise, dirty or newgen.
-class CheckForUnmarkedObjects : public ObjectClosure {
+class PSCheckForUnmarkedObjects : public ObjectClosure {
private:
PSYoungGen* _young_gen;
PSCardTable* _card_table;
public:
- CheckForUnmarkedObjects() {
+ PSCheckForUnmarkedObjects() {
ParallelScavengeHeap* heap = ParallelScavengeHeap::heap();
_young_gen = heap->young_gen();
_card_table = heap->card_table();
@@ -87,7 +85,7 @@ class CheckForUnmarkedObjects : public ObjectClosure {
// we test for missing precise marks first. If any are found, we don't
// fail unless the object head is also unmarked.
virtual void do_object(oop obj) {
- CheckForUnmarkedOops object_check(_young_gen, _card_table);
+ PSCheckForUnmarkedOops object_check(_young_gen, _card_table);
obj->oop_iterate(&object_check);
if (object_check.has_unmarked_oop()) {
guarantee(_card_table->is_dirty_for_addr(obj), "Found unmarked young_gen object");
@@ -379,7 +377,7 @@ void PSCardTable::scavenge_contents_parallel(ObjectStartArray* start_array,
// This should be called before a scavenge.
void PSCardTable::verify_all_young_refs_imprecise() {
- CheckForUnmarkedObjects check;
+ PSCheckForUnmarkedObjects check;
ParallelScavengeHeap* heap = ParallelScavengeHeap::heap();
PSOldGen* old_gen = heap->old_gen();
diff --git a/src/hotspot/share/gc/parallel/psParallelCompact.cpp b/src/hotspot/share/gc/parallel/psParallelCompact.cpp
index e45b6085112..d7bfe5242a9 100644
--- a/src/hotspot/share/gc/parallel/psParallelCompact.cpp
+++ b/src/hotspot/share/gc/parallel/psParallelCompact.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -246,7 +246,7 @@ void PSParallelCompact::print_region_ranges() {
}
}
-void
+static void
print_generic_summary_region(size_t i, const ParallelCompactData::RegionData* c)
{
#define REGION_IDX_FORMAT SIZE_FORMAT_W(7)
@@ -312,7 +312,7 @@ print_generic_summary_data(ParallelCompactData& summary_data,
}
}
-void
+static void
print_initial_summary_data(ParallelCompactData& summary_data,
const MutableSpace* space) {
if (space->top() == space->bottom()) {
@@ -393,7 +393,7 @@ print_initial_summary_data(ParallelCompactData& summary_data,
max_reclaimed_ratio_region, max_dead_to_right, max_live_to_right, max_reclaimed_ratio);
}
-void
+static void
print_initial_summary_data(ParallelCompactData& summary_data,
SpaceInfo* space_info) {
if (!log_develop_is_enabled(Trace, gc, compaction)) {
diff --git a/src/hotspot/share/gc/serial/cardTableRS.cpp b/src/hotspot/share/gc/serial/cardTableRS.cpp
index c4af2c8edb1..87c0eba8192 100644
--- a/src/hotspot/share/gc/serial/cardTableRS.cpp
+++ b/src/hotspot/share/gc/serial/cardTableRS.cpp
@@ -26,37 +26,29 @@
#include "classfile/classLoaderDataGraph.hpp"
#include "gc/serial/cardTableRS.hpp"
#include "gc/serial/generation.hpp"
-#include "gc/serial/serialHeap.hpp"
+#include "gc/serial/serialHeap.inline.hpp"
#include "gc/shared/space.inline.hpp"
#include "memory/iterator.inline.hpp"
#include "utilities/align.hpp"
-void CardTableRS::younger_refs_in_space_iterate(TenuredSpace* sp,
- OopIterateClosure* cl) {
- verify_used_region_at_save_marks(sp);
-
+void CardTableRS::scan_old_to_young_refs(TenuredSpace* sp) {
+ const MemRegion ur = sp->used_region();
const MemRegion urasm = sp->used_region_at_save_marks();
- if (!urasm.is_empty()) {
- non_clean_card_iterate(sp, urasm, cl, this);
- }
-}
-
-#ifdef ASSERT
-void CardTableRS::verify_used_region_at_save_marks(Space* sp) const {
- MemRegion ur = sp->used_region();
- MemRegion urasm = sp->used_region_at_save_marks();
assert(ur.contains(urasm),
"Did you forget to call save_marks()? "
"[" PTR_FORMAT ", " PTR_FORMAT ") is not contained in "
"[" PTR_FORMAT ", " PTR_FORMAT ")",
p2i(urasm.start()), p2i(urasm.end()), p2i(ur.start()), p2i(ur.end()));
-}
-#endif
-void CardTableRS::maintain_old_to_young_invariant(Generation* old_gen, bool is_young_gen_empty) {
- assert(SerialHeap::heap()->is_old_gen(old_gen), "precondition");
+ if (!urasm.is_empty()) {
+ OldGenScanClosure cl(SerialHeap::heap()->young_gen());
+ non_clean_card_iterate(sp, urasm, &cl);
+ }
+}
+void CardTableRS::maintain_old_to_young_invariant(TenuredGeneration* old_gen,
+ bool is_young_gen_empty) {
if (is_young_gen_empty) {
clear_MemRegion(old_gen->prev_used_region());
} else {
@@ -72,274 +64,59 @@ void CardTableRS::maintain_old_to_young_invariant(Generation* old_gen, bool is_y
}
}
-class VerifyCleanCardClosure: public BasicOopIterateClosure {
-private:
- HeapWord* _boundary;
- HeapWord* _begin;
- HeapWord* _end;
-protected:
+class SerialCheckForUnmarkedOops : public BasicOopIterateClosure {
+ DefNewGeneration* _young_gen;
+ CardTableRS* _card_table;
+ HeapWord* _unmarked_addr;
+
template void do_oop_work(T* p) {
- HeapWord* jp = (HeapWord*)p;
- assert(jp >= _begin && jp < _end,
- "Error: jp " PTR_FORMAT " should be within "
- "[_begin, _end) = [" PTR_FORMAT "," PTR_FORMAT ")",
- p2i(jp), p2i(_begin), p2i(_end));
oop obj = RawAccess<>::oop_load(p);
- guarantee(obj == nullptr || cast_from_oop(obj) >= _boundary,
- "pointer " PTR_FORMAT " at " PTR_FORMAT " on "
- "clean card crosses boundary" PTR_FORMAT,
- p2i(obj), p2i(jp), p2i(_boundary));
+ if (_young_gen->is_in_reserved(obj) &&
+ !_card_table->is_dirty_for_addr(p)) {
+ // Don't overwrite the first missing card mark
+ if (_unmarked_addr == nullptr) {
+ _unmarked_addr = (HeapWord*)p;
+ }
+ }
}
-public:
- VerifyCleanCardClosure(HeapWord* b, HeapWord* begin, HeapWord* end) :
- _boundary(b), _begin(begin), _end(end) {
- assert(b <= begin,
- "Error: boundary " PTR_FORMAT " should be at or below begin " PTR_FORMAT,
- p2i(b), p2i(begin));
- assert(begin <= end,
- "Error: begin " PTR_FORMAT " should be strictly below end " PTR_FORMAT,
- p2i(begin), p2i(end));
- }
+ public:
+ SerialCheckForUnmarkedOops(DefNewGeneration* young_gen, CardTableRS* card_table) :
+ _young_gen(young_gen),
+ _card_table(card_table),
+ _unmarked_addr(nullptr) {}
- virtual void do_oop(oop* p) { VerifyCleanCardClosure::do_oop_work(p); }
- virtual void do_oop(narrowOop* p) { VerifyCleanCardClosure::do_oop_work(p); }
-};
+ void do_oop(oop* p) override { do_oop_work(p); }
+ void do_oop(narrowOop* p) override { do_oop_work(p); }
-class VerifyCTSpaceClosure: public SpaceClosure {
-private:
- CardTableRS* _ct;
- HeapWord* _boundary;
-public:
- VerifyCTSpaceClosure(CardTableRS* ct, HeapWord* boundary) :
- _ct(ct), _boundary(boundary) {}
- virtual void do_space(Space* s) { _ct->verify_space(s, _boundary); }
+ bool has_unmarked_oop() {
+ return _unmarked_addr != nullptr;
+ }
};
-class VerifyCTGenClosure: public SerialHeap::GenClosure {
- CardTableRS* _ct;
-public:
- VerifyCTGenClosure(CardTableRS* ct) : _ct(ct) {}
- void do_generation(Generation* gen) {
- // Skip the youngest generation.
- if (SerialHeap::heap()->is_young_gen(gen)) {
- return;
+void CardTableRS::verify() {
+ class CheckForUnmarkedObjects : public ObjectClosure {
+ DefNewGeneration* _young_gen;
+ CardTableRS* _card_table;
+
+ public:
+ CheckForUnmarkedObjects() {
+ SerialHeap* heap = SerialHeap::heap();
+ _young_gen = heap->young_gen();
+ _card_table = heap->rem_set();
}
- // Normally, we're interested in pointers to younger generations.
- VerifyCTSpaceClosure blk(_ct, gen->reserved().start());
- gen->space_iterate(&blk, true);
- }
-};
-void CardTableRS::verify_space(Space* s, HeapWord* gen_boundary) {
- // We don't need to do young-gen spaces.
- if (s->end() <= gen_boundary) return;
- MemRegion used = s->used_region();
-
- CardValue* cur_entry = byte_for(used.start());
- CardValue* limit = byte_after(used.last());
- while (cur_entry < limit) {
- if (*cur_entry == clean_card_val()) {
- CardValue* first_dirty = cur_entry+1;
- while (first_dirty < limit &&
- *first_dirty == clean_card_val()) {
- first_dirty++;
+ void do_object(oop obj) override {
+ SerialCheckForUnmarkedOops object_check(_young_gen, _card_table);
+ obj->oop_iterate(&object_check);
+ // If this obj is imprecisely-marked, the card for obj-start must be dirty.
+ if (object_check.has_unmarked_oop()) {
+ guarantee(_card_table->is_dirty_for_addr(obj), "Found unmarked old-to-young pointer");
}
- // If the first object is a regular object, and it has a
- // young-to-old field, that would mark the previous card.
- HeapWord* boundary = addr_for(cur_entry);
- HeapWord* end = (first_dirty >= limit) ? used.end() : addr_for(first_dirty);
- HeapWord* boundary_block = s->block_start(boundary);
- HeapWord* begin = boundary; // Until proven otherwise.
- HeapWord* start_block = boundary_block; // Until proven otherwise.
- if (boundary_block < boundary) {
- if (s->block_is_obj(boundary_block) && s->obj_is_alive(boundary_block)) {
- oop boundary_obj = cast_to_oop(boundary_block);
- if (!boundary_obj->is_objArray() &&
- !boundary_obj->is_typeArray()) {
- guarantee(cur_entry > byte_for(used.start()),
- "else boundary would be boundary_block");
- if (*byte_for(boundary_block) != clean_card_val()) {
- begin = boundary_block + s->block_size(boundary_block);
- start_block = begin;
- }
- }
- }
- }
- // Now traverse objects until end.
- if (begin < end) {
- MemRegion mr(begin, end);
- VerifyCleanCardClosure verify_blk(gen_boundary, begin, end);
- for (HeapWord* cur = start_block; cur < end; cur += s->block_size(cur)) {
- if (s->block_is_obj(cur) && s->obj_is_alive(cur)) {
- cast_to_oop(cur)->oop_iterate(&verify_blk, mr);
- }
- }
- }
- cur_entry = first_dirty;
- } else {
- // We'd normally expect that cur_youngergen_and_prev_nonclean_card
- // is a transient value, that cannot be in the card table
- // except during GC, and thus assert that:
- // guarantee(*cur_entry != cur_youngergen_and_prev_nonclean_card,
- // "Illegal CT value");
- // That however, need not hold, as will become clear in the
- // following...
-
- // We'd normally expect that if we are in the parallel case,
- // we can't have left a prev value (which would be different
- // from the current value) in the card table, and so we'd like to
- // assert that:
- // guarantee(cur_youngergen_card_val() == youngergen_card
- // || !is_prev_youngergen_card_val(*cur_entry),
- // "Illegal CT value");
- // That, however, may not hold occasionally, because of
- // CMS or MSC in the old gen. To wit, consider the
- // following two simple illustrative scenarios:
- // (a) CMS: Consider the case where a large object L
- // spanning several cards is allocated in the old
- // gen, and has a young gen reference stored in it, dirtying
- // some interior cards. A young collection scans the card,
- // finds a young ref and installs a youngergenP_n value.
- // L then goes dead. Now a CMS collection starts,
- // finds L dead and sweeps it up. Assume that L is
- // abutting _unallocated_blk, so _unallocated_blk is
- // adjusted down to (below) L. Assume further that
- // no young collection intervenes during this CMS cycle.
- // The next young gen cycle will not get to look at this
- // youngergenP_n card since it lies in the unoccupied
- // part of the space.
- // Some young collections later the blocks on this
- // card can be re-allocated either due to direct allocation
- // or due to absorbing promotions. At this time, the
- // before-gc verification will fail the above assert.
- // (b) MSC: In this case, an object L with a young reference
- // is on a card that (therefore) holds a youngergen_n value.
- // Suppose also that L lies towards the end of the used
- // the used space before GC. An MSC collection
- // occurs that compacts to such an extent that this
- // card is no longer in the occupied part of the space.
- // Since current code in MSC does not always clear cards
- // in the unused part of old gen, this stale youngergen_n
- // value is left behind and can later be covered by
- // an object when promotion or direct allocation
- // re-allocates that part of the heap.
- //
- // Fortunately, the presence of such stale card values is
- // "only" a minor annoyance in that subsequent young collections
- // might needlessly scan such cards, but would still never corrupt
- // the heap as a result. However, it's likely not to be a significant
- // performance inhibitor in practice. For instance,
- // some recent measurements with unoccupied cards eagerly cleared
- // out to maintain this invariant, showed next to no
- // change in young collection times; of course one can construct
- // degenerate examples where the cost can be significant.)
- // Note, in particular, that if the "stale" card is modified
- // after re-allocation, it would be dirty, not "stale". Thus,
- // we can never have a younger ref in such a card and it is
- // safe not to scan that card in any collection. [As we see
- // below, we do some unnecessary scanning
- // in some cases in the current parallel scanning algorithm.]
- //
- // The main point below is that the parallel card scanning code
- // deals correctly with these stale card values. There are two main
- // cases to consider where we have a stale "young gen" value and a
- // "derivative" case to consider, where we have a stale
- // "cur_younger_gen_and_prev_non_clean" value, as will become
- // apparent in the case analysis below.
- // o Case 1. If the stale value corresponds to a younger_gen_n
- // value other than the cur_younger_gen value then the code
- // treats this as being tantamount to a prev_younger_gen
- // card. This means that the card may be unnecessarily scanned.
- // There are two sub-cases to consider:
- // o Case 1a. Let us say that the card is in the occupied part
- // of the generation at the time the collection begins. In
- // that case the card will be either cleared when it is scanned
- // for young pointers, or will be set to cur_younger_gen as a
- // result of promotion. (We have elided the normal case where
- // the scanning thread and the promoting thread interleave
- // possibly resulting in a transient
- // cur_younger_gen_and_prev_non_clean value before settling
- // to cur_younger_gen. [End Case 1a.]
- // o Case 1b. Consider now the case when the card is in the unoccupied
- // part of the space which becomes occupied because of promotions
- // into it during the current young GC. In this case the card
- // will never be scanned for young references. The current
- // code will set the card value to either
- // cur_younger_gen_and_prev_non_clean or leave
- // it with its stale value -- because the promotions didn't
- // result in any younger refs on that card. Of these two
- // cases, the latter will be covered in Case 1a during
- // a subsequent scan. To deal with the former case, we need
- // to further consider how we deal with a stale value of
- // cur_younger_gen_and_prev_non_clean in our case analysis
- // below. This we do in Case 3 below. [End Case 1b]
- // [End Case 1]
- // o Case 2. If the stale value corresponds to cur_younger_gen being
- // a value not necessarily written by a current promotion, the
- // card will not be scanned by the younger refs scanning code.
- // (This is OK since as we argued above such cards cannot contain
- // any younger refs.) The result is that this value will be
- // treated as a prev_younger_gen value in a subsequent collection,
- // which is addressed in Case 1 above. [End Case 2]
- // o Case 3. We here consider the "derivative" case from Case 1b. above
- // because of which we may find a stale
- // cur_younger_gen_and_prev_non_clean card value in the table.
- // Once again, as in Case 1, we consider two subcases, depending
- // on whether the card lies in the occupied or unoccupied part
- // of the space at the start of the young collection.
- // o Case 3a. Let us say the card is in the occupied part of
- // the old gen at the start of the young collection. In that
- // case, the card will be scanned by the younger refs scanning
- // code which will set it to cur_younger_gen. In a subsequent
- // scan, the card will be considered again and get its final
- // correct value. [End Case 3a]
- // o Case 3b. Now consider the case where the card is in the
- // unoccupied part of the old gen, and is occupied as a result
- // of promotions during thus young gc. In that case,
- // the card will not be scanned for younger refs. The presence
- // of newly promoted objects on the card will then result in
- // its keeping the value cur_younger_gen_and_prev_non_clean
- // value, which we have dealt with in Case 3 here. [End Case 3b]
- // [End Case 3]
- //
- // (Please refer to the code in the helper class
- // ClearNonCleanCardWrapper and in CardTable for details.)
- //
- // The informal arguments above can be tightened into a formal
- // correctness proof and it behooves us to write up such a proof,
- // or to use model checking to prove that there are no lingering
- // concerns.
- //
- // Clearly because of Case 3b one cannot bound the time for
- // which a card will retain what we have called a "stale" value.
- // However, one can obtain a Loose upper bound on the redundant
- // work as a result of such stale values. Note first that any
- // time a stale card lies in the occupied part of the space at
- // the start of the collection, it is scanned by younger refs
- // code and we can define a rank function on card values that
- // declines when this is so. Note also that when a card does not
- // lie in the occupied part of the space at the beginning of a
- // young collection, its rank can either decline or stay unchanged.
- // In this case, no extra work is done in terms of redundant
- // younger refs scanning of that card.
- // Then, the case analysis above reveals that, in the worst case,
- // any such stale card will be scanned unnecessarily at most twice.
- //
- // It is nonetheless advisable to try and get rid of some of this
- // redundant work in a subsequent (low priority) re-design of
- // the card-scanning code, if only to simplify the underlying
- // state machine analysis/proof. ysr 1/28/2002. XXX
- cur_entry++;
}
- }
-}
+ } check;
-void CardTableRS::verify() {
- // At present, we only know how to verify the card table RS for
- // generational heaps.
- VerifyCTGenClosure blk(this);
- SerialHeap::heap()->generation_iterate(&blk, false);
+ SerialHeap::heap()->old_gen()->object_iterate(&check);
}
CardTableRS::CardTableRS(MemRegion whole_heap) :
@@ -391,7 +168,6 @@ CardTable::CardValue* CardTableRS::find_first_dirty_card(CardValue* const start_
template
CardTable::CardValue* CardTableRS::find_first_clean_card(CardValue* const start_card,
CardValue* const end_card,
- CardTableRS* ct,
Func& object_start) {
for (CardValue* current_card = start_card; current_card < end_card; /* empty */) {
if (is_dirty(current_card)) {
@@ -400,7 +176,7 @@ CardTable::CardValue* CardTableRS::find_first_clean_card(CardValue* const start_
}
// A potential candidate.
- HeapWord* addr = ct->addr_for(current_card);
+ HeapWord* addr = addr_for(current_card);
HeapWord* obj_start_addr = object_start(addr);
if (obj_start_addr == addr) {
@@ -416,7 +192,7 @@ CardTable::CardValue* CardTableRS::find_first_clean_card(CardValue* const start_
}
// Final card occupied by obj.
- CardValue* obj_final_card = ct->byte_for(obj_start_addr + obj->size() - 1);
+ CardValue* obj_final_card = byte_for(obj_start_addr + obj->size() - 1);
if (is_clean(obj_final_card)) {
return obj_final_card;
}
@@ -440,7 +216,7 @@ static void prefetch_write(void *p) {
}
static void scan_obj_with_limit(oop obj,
- OopIterateClosure* cl,
+ OldGenScanClosure* cl,
HeapWord* start,
HeapWord* end) {
if (!obj->is_typeArray()) {
@@ -451,8 +227,7 @@ static void scan_obj_with_limit(oop obj,
void CardTableRS::non_clean_card_iterate(TenuredSpace* sp,
MemRegion mr,
- OopIterateClosure* cl,
- CardTableRS* ct) {
+ OldGenScanClosure* cl) {
struct {
HeapWord* start_addr;
HeapWord* end_addr;
@@ -471,8 +246,8 @@ void CardTableRS::non_clean_card_iterate(TenuredSpace* sp,
return result;
};
- CardValue* const start_card = ct->byte_for(mr.start());
- CardValue* const end_card = ct->byte_for(mr.last()) + 1;
+ CardValue* const start_card = byte_for(mr.start());
+ CardValue* const end_card = byte_for(mr.last()) + 1;
// if mr.end() is not card-aligned, that final card should not be cleared
// because it can be annotated dirty due to old-to-young pointers in
@@ -482,22 +257,20 @@ void CardTableRS::non_clean_card_iterate(TenuredSpace* sp,
for (CardValue* current_card = start_card; current_card < end_card; /* empty */) {
CardValue* const dirty_l = find_first_dirty_card(current_card, end_card);
-
if (dirty_l == end_card) {
// No dirty cards to iterate.
return;
}
- HeapWord* const addr_l = ct->addr_for(dirty_l);
+ HeapWord* const addr_l = addr_for(dirty_l);
HeapWord* obj_addr = object_start(addr_l);
CardValue* const dirty_r = find_first_clean_card(dirty_l + 1,
end_card,
- ct,
object_start);
assert(dirty_l < dirty_r, "inv");
HeapWord* const addr_r = dirty_r == end_card ? mr.end()
- : ct->addr_for(dirty_r);
+ : addr_for(dirty_r);
clear_cards(MIN2(dirty_l, clear_limit_card),
MIN2(dirty_r, clear_limit_card));
diff --git a/src/hotspot/share/gc/serial/cardTableRS.hpp b/src/hotspot/share/gc/serial/cardTableRS.hpp
index a4f333803a0..7ad6f8aef0b 100644
--- a/src/hotspot/share/gc/serial/cardTableRS.hpp
+++ b/src/hotspot/share/gc/serial/cardTableRS.hpp
@@ -29,8 +29,8 @@
#include "memory/memRegion.hpp"
#include "oops/oop.hpp"
-class Generation;
-class Space;
+class OldGenScanClosure;
+class TenuredGeneration;
class TenuredSpace;
// This RemSet uses a card table both as shared data structure
@@ -38,10 +38,6 @@ class TenuredSpace;
class CardTableRS : public CardTable {
friend class VMStructs;
- // Below are private classes used in impl.
- friend class VerifyCTSpaceClosure;
-
- void verify_space(Space* s, HeapWord* gen_start);
static bool is_dirty(const CardValue* const v) {
return !is_clean(v);
@@ -59,36 +55,37 @@ class CardTableRS : public CardTable {
template
CardValue* find_first_clean_card(CardValue* start_card,
CardValue* end_card,
- CardTableRS* ct,
Func& object_start);
public:
CardTableRS(MemRegion whole_heap);
- void younger_refs_in_space_iterate(TenuredSpace* sp, OopIterateClosure* cl);
-
- virtual void verify_used_region_at_save_marks(Space* sp) const NOT_DEBUG_RETURN;
+ void scan_old_to_young_refs(TenuredSpace* sp);
void inline_write_ref_field_gc(void* field) {
CardValue* byte = byte_for(field);
*byte = dirty_card_val();
}
+ bool is_dirty_for_addr(const void* p) const {
+ CardValue* card = byte_for(p);
+ return is_dirty(card);
+ }
+
void verify();
// Update old gen cards to maintain old-to-young-pointer invariant: Clear
// the old generation card table completely if the young generation had been
// completely evacuated, otherwise dirties the whole old generation to
// conservatively not loose any old-to-young pointer.
- void maintain_old_to_young_invariant(Generation* old_gen, bool is_young_gen_empty);
+ void maintain_old_to_young_invariant(TenuredGeneration* old_gen, bool is_young_gen_empty);
// Iterate over the portion of the card-table which covers the given
// region mr in the given space and apply cl to any dirty sub-regions
// of mr. Clears the dirty cards as they are processed.
void non_clean_card_iterate(TenuredSpace* sp,
MemRegion mr,
- OopIterateClosure* cl,
- CardTableRS* ct);
+ OldGenScanClosure* cl);
bool is_in_young(const void* p) const override;
};
diff --git a/src/hotspot/share/gc/serial/defNewGeneration.cpp b/src/hotspot/share/gc/serial/defNewGeneration.cpp
index 964c7661d31..3f1cc7c6e4f 100644
--- a/src/hotspot/share/gc/serial/defNewGeneration.cpp
+++ b/src/hotspot/share/gc/serial/defNewGeneration.cpp
@@ -61,91 +61,6 @@
#include "utilities/globalDefinitions.hpp"
#include "utilities/stack.inline.hpp"
-class ScavengeHelper {
- DefNewGeneration* _young_gen;
- HeapWord* _young_gen_end;
-public:
- ScavengeHelper(DefNewGeneration* young_gen) :
- _young_gen(young_gen),
- _young_gen_end(young_gen->reserved().end()) {}
-
- bool is_in_young_gen(void* p) const {
- return p < _young_gen_end;
- }
-
- template
- void try_scavenge(T* p, Func&& f) {
- T heap_oop = RawAccess<>::oop_load(p);
- // Should we copy the obj?
- if (!CompressedOops::is_null(heap_oop)) {
- oop obj = CompressedOops::decode_not_null(heap_oop);
- if (is_in_young_gen(obj)) {
- assert(!_young_gen->to()->is_in_reserved(obj), "Scanning field twice?");
- oop new_obj = obj->is_forwarded() ? obj->forwardee()
- : _young_gen->copy_to_survivor_space(obj);
- RawAccess::oop_store(p, new_obj);
-
- // callback
- f(new_obj);
- }
- }
- }
-};
-
-class InHeapScanClosure : public BasicOopIterateClosure {
- ScavengeHelper _helper;
-protected:
- bool is_in_young_gen(void* p) const {
- return _helper.is_in_young_gen(p);
- }
-
- template
- void try_scavenge(T* p, Func&& f) {
- _helper.try_scavenge(p, f);
- }
-
- InHeapScanClosure(DefNewGeneration* young_gen) :
- BasicOopIterateClosure(young_gen->ref_processor()),
- _helper(young_gen) {}
-};
-
-class OffHeapScanClosure : public OopClosure {
- ScavengeHelper _helper;
-protected:
- bool is_in_young_gen(void* p) const {
- return _helper.is_in_young_gen(p);
- }
-
- template
- void try_scavenge(T* p, Func&& f) {
- _helper.try_scavenge(p, f);
- }
-
- OffHeapScanClosure(DefNewGeneration* young_gen) : _helper(young_gen) {}
-};
-
-class OldGenScanClosure : public InHeapScanClosure {
- CardTableRS* _rs;
-
- template
- void do_oop_work(T* p) {
- assert(!is_in_young_gen(p), "precondition");
-
- try_scavenge(p, [&] (oop new_obj) {
- // If p points to a younger generation, mark the card.
- if (is_in_young_gen(new_obj)) {
- _rs->inline_write_ref_field_gc(p);
- }
- });
- }
-public:
- OldGenScanClosure(DefNewGeneration* g) : InHeapScanClosure(g),
- _rs(SerialHeap::heap()->rem_set()) {}
-
- void do_oop(oop* p) { do_oop_work(p); }
- void do_oop(narrowOop* p) { do_oop_work(p); }
-};
-
class PromoteFailureClosure : public InHeapScanClosure {
template
void do_oop_work(T* p) {
@@ -420,9 +335,9 @@ void DefNewGeneration::compute_space_boundaries(uintx minimum_eden_size,
char *to_end = to_start + survivor_size;
assert(to_end == _virtual_space.high(), "just checking");
- assert(Space::is_aligned(eden_start), "checking alignment");
- assert(Space::is_aligned(from_start), "checking alignment");
- assert(Space::is_aligned(to_start), "checking alignment");
+ assert(is_aligned(eden_start, SpaceAlignment), "checking alignment");
+ assert(is_aligned(from_start, SpaceAlignment), "checking alignment");
+ assert(is_aligned(to_start, SpaceAlignment), "checking alignment");
MemRegion edenMR((HeapWord*)eden_start, (HeapWord*)from_start);
MemRegion fromMR((HeapWord*)from_start, (HeapWord*)to_start);
@@ -649,6 +564,12 @@ size_t DefNewGeneration::max_capacity() const {
return reserved_bytes - compute_survivor_size(reserved_bytes, SpaceAlignment);
}
+bool DefNewGeneration::is_in(const void* p) const {
+ return eden()->is_in(p)
+ || from()->is_in(p)
+ || to() ->is_in(p);
+}
+
size_t DefNewGeneration::unsafe_max_alloc_nogc() const {
return eden()->free();
}
@@ -667,12 +588,15 @@ void DefNewGeneration::object_iterate(ObjectClosure* blk) {
from()->object_iterate(blk);
}
-
-void DefNewGeneration::space_iterate(SpaceClosure* blk,
- bool usedOnly) {
- blk->do_space(eden());
- blk->do_space(from());
- blk->do_space(to());
+HeapWord* DefNewGeneration::block_start(const void* p) const {
+ if (eden()->is_in_reserved(p)) {
+ return eden()->block_start_const(p);
+ }
+ if (from()->is_in_reserved(p)) {
+ return from()->block_start_const(p);
+ }
+ assert(to()->is_in_reserved(p), "inv");
+ return to()->block_start_const(p);
}
// The last collection bailed out, we are running out of heap space,
@@ -776,11 +700,19 @@ void DefNewGeneration::collect(bool full,
{
StrongRootsScope srs(0);
RootScanClosure root_cl{this};
- CLDScanClosure cld_scan_closure{this};
+ CLDScanClosure cld_cl{this};
+
+ MarkingCodeBlobClosure code_cl(&root_cl,
+ CodeBlobToOopClosure::FixRelocations,
+ false /* keepalive_nmethods */);
+
+ heap->process_roots(SerialHeap::SO_ScavengeCodeCache,
+ &root_cl,
+ &cld_cl,
+ &cld_cl,
+ &code_cl);
- heap->young_process_roots(&root_cl,
- &old_gen_cl,
- &cld_scan_closure);
+ _old_gen->scan_old_to_young_refs();
}
// "evacuate followers".
@@ -930,6 +862,9 @@ oop DefNewGeneration::copy_to_survivor_space(oop old) {
handle_promotion_failure(old);
return old;
}
+
+ ContinuationGCSupport::transform_stack_chunk(obj);
+
new_obj_is_tenured = true;
} else {
// Prefetch beyond obj
diff --git a/src/hotspot/share/gc/serial/defNewGeneration.hpp b/src/hotspot/share/gc/serial/defNewGeneration.hpp
index 2bacd94bb6a..5bfce3c2eef 100644
--- a/src/hotspot/share/gc/serial/defNewGeneration.hpp
+++ b/src/hotspot/share/gc/serial/defNewGeneration.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -159,8 +159,6 @@ class DefNewGeneration: public Generation {
size_t max_byte_size,
const char* policy="Serial young collection pauses");
- virtual Generation::Name kind() { return Generation::DefNew; }
-
// allocate and initialize ("weak") refs processing support
void ref_processor_init();
ReferenceProcessor* ref_processor() { return _ref_processor; }
@@ -176,7 +174,17 @@ class DefNewGeneration: public Generation {
size_t free() const;
size_t max_capacity() const;
size_t capacity_before_gc() const;
+
+ // Returns "TRUE" iff "p" points into the used areas in each space of young-gen.
+ bool is_in(const void* p) const;
+
+ // Return an estimate of the maximum allocation that could be performed
+ // in the generation without triggering any collection or expansion
+ // activity. It is "unsafe" because no locks are taken; the result
+ // should be treated as an approximation, not a guarantee, for use in
+ // heuristic resizing decisions.
size_t unsafe_max_alloc_nogc() const;
+
size_t contiguous_available() const;
size_t max_eden_size() const { return _max_eden_size; }
@@ -200,7 +208,7 @@ class DefNewGeneration: public Generation {
// Iteration
void object_iterate(ObjectClosure* blk);
- void space_iterate(SpaceClosure* blk, bool usedOnly = false);
+ HeapWord* block_start(const void* p) const;
// Allocation support
virtual bool should_allocate(size_t word_size, bool is_tlab) {
@@ -226,7 +234,7 @@ class DefNewGeneration: public Generation {
HeapWord* par_allocate(size_t word_size, bool is_tlab);
- virtual void gc_epilogue(bool full);
+ void gc_epilogue(bool full);
// Save the tops for eden, from, and to
virtual void record_spaces_top();
@@ -250,7 +258,7 @@ class DefNewGeneration: public Generation {
void reset_scratch();
// GC support
- virtual void compute_new_size();
+ void compute_new_size();
// Returns true if the collection is likely to be safely
// completed. Even if this method returns true, a collection
diff --git a/src/hotspot/share/gc/serial/defNewGeneration.inline.hpp b/src/hotspot/share/gc/serial/defNewGeneration.inline.hpp
index 1f0a5df78b1..808e04a3002 100644
--- a/src/hotspot/share/gc/serial/defNewGeneration.inline.hpp
+++ b/src/hotspot/share/gc/serial/defNewGeneration.inline.hpp
@@ -41,7 +41,6 @@ void DefNewGeneration::oop_since_save_marks_iterate(OopClosureType* cl) {
assert(from()->saved_mark_at_top(), "inv");
to()->oop_since_save_marks_iterate(cl);
- to()->set_saved_mark();
}
#endif // SHARE_GC_SERIAL_DEFNEWGENERATION_INLINE_HPP
diff --git a/src/hotspot/share/gc/serial/genMarkSweep.cpp b/src/hotspot/share/gc/serial/genMarkSweep.cpp
index 28ba4615f73..d0594a3a500 100644
--- a/src/hotspot/share/gc/serial/genMarkSweep.cpp
+++ b/src/hotspot/share/gc/serial/genMarkSweep.cpp
@@ -30,7 +30,6 @@
#include "classfile/systemDictionary.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "compiler/oopMap.hpp"
#include "gc/serial/cardTableRS.hpp"
#include "gc/serial/defNewGeneration.hpp"
@@ -441,10 +440,9 @@ void GenMarkSweep::invoke_at_safepoint(bool clear_all_softrefs) {
// Increment the invocation count
_total_invocations++;
- // Capture used regions for each generation that will be
- // subject to collection, so that card table adjustments can
- // be made intelligently (see clear / invalidate further below).
- gch->save_used_regions();
+ // Capture used regions for old-gen to reestablish old-to-young invariant
+ // after full-gc.
+ gch->old_gen()->save_used_region();
allocate_stacks();
diff --git a/src/hotspot/share/gc/serial/generation.cpp b/src/hotspot/share/gc/serial/generation.cpp
index 86fc3300e30..c1c195aaf38 100644
--- a/src/hotspot/share/gc/serial/generation.cpp
+++ b/src/hotspot/share/gc/serial/generation.cpp
@@ -27,7 +27,6 @@
#include "gc/serial/generation.hpp"
#include "gc/serial/serialHeap.hpp"
#include "gc/shared/collectedHeap.inline.hpp"
-#include "gc/shared/continuationGCSupport.inline.hpp"
#include "gc/shared/gcLocker.hpp"
#include "gc/shared/gcTimer.hpp"
#include "gc/shared/gcTrace.hpp"
@@ -83,26 +82,6 @@ void Generation::print_summary_info_on(outputStream* st) {
sr->invocations > 0 ? time / sr->invocations : 0.0);
}
-// Utility iterator classes
-
-class GenerationIsInClosure : public SpaceClosure {
- public:
- const void* _p;
- Space* sp;
- virtual void do_space(Space* s) {
- if (sp == nullptr) {
- if (s->is_in(_p)) sp = s;
- }
- }
- GenerationIsInClosure(const void* p) : _p(p), sp(nullptr) {}
-};
-
-bool Generation::is_in(const void* p) const {
- GenerationIsInClosure blk(p);
- ((Generation*)this)->space_iterate(&blk);
- return blk.sp != nullptr;
-}
-
size_t Generation::max_contiguous_available() const {
// The largest number of contiguous free words in this or any higher generation.
size_t avail = contiguous_available();
@@ -112,89 +91,3 @@ size_t Generation::max_contiguous_available() const {
}
return MAX2(avail, old_avail);
}
-
-// Ignores "ref" and calls allocate().
-oop Generation::promote(oop obj, size_t obj_size) {
- assert(obj_size == obj->size(), "bad obj_size passed in");
-
-#ifndef PRODUCT
- if (SerialHeap::heap()->promotion_should_fail()) {
- return nullptr;
- }
-#endif // #ifndef PRODUCT
-
- // Allocate new object.
- HeapWord* result = allocate(obj_size, false);
- if (result == nullptr) {
- // Promotion of obj into gen failed. Try to expand and allocate.
- result = expand_and_allocate(obj_size, false);
- if (result == nullptr) {
- return nullptr;
- }
- }
-
- // Copy to new location.
- Copy::aligned_disjoint_words(cast_from_oop(obj), result, obj_size);
- oop new_obj = cast_to_oop(result);
-
- // Transform object if it is a stack chunk.
- ContinuationGCSupport::transform_stack_chunk(new_obj);
-
- return new_obj;
-}
-
-// Some of these are mediocre general implementations. Should be
-// overridden to get better performance.
-
-class GenerationBlockStartClosure : public SpaceClosure {
- public:
- const void* _p;
- HeapWord* _start;
- virtual void do_space(Space* s) {
- if (_start == nullptr && s->is_in_reserved(_p)) {
- _start = s->block_start(_p);
- }
- }
- GenerationBlockStartClosure(const void* p) { _p = p; _start = nullptr; }
-};
-
-HeapWord* Generation::block_start(const void* p) const {
- GenerationBlockStartClosure blk(p);
- // Cast away const
- ((Generation*)this)->space_iterate(&blk);
- return blk._start;
-}
-
-class GenerationBlockIsObjClosure : public SpaceClosure {
- public:
- const HeapWord* _p;
- bool is_obj;
- virtual void do_space(Space* s) {
- if (!is_obj && s->is_in_reserved(_p)) {
- is_obj |= s->block_is_obj(_p);
- }
- }
- GenerationBlockIsObjClosure(const HeapWord* p) { _p = p; is_obj = false; }
-};
-
-bool Generation::block_is_obj(const HeapWord* p) const {
- GenerationBlockIsObjClosure blk(p);
- // Cast away const
- ((Generation*)this)->space_iterate(&blk);
- return blk.is_obj;
-}
-
-class GenerationObjIterateClosure : public SpaceClosure {
- private:
- ObjectClosure* _cl;
- public:
- virtual void do_space(Space* s) {
- s->object_iterate(_cl);
- }
- GenerationObjIterateClosure(ObjectClosure* cl) : _cl(cl) {}
-};
-
-void Generation::object_iterate(ObjectClosure* cl) {
- GenerationObjIterateClosure blk(cl);
- space_iterate(&blk);
-}
diff --git a/src/hotspot/share/gc/serial/generation.hpp b/src/hotspot/share/gc/serial/generation.hpp
index 70df097a574..2239046a4e9 100644
--- a/src/hotspot/share/gc/serial/generation.hpp
+++ b/src/hotspot/share/gc/serial/generation.hpp
@@ -58,9 +58,6 @@ class GCStats;
class Generation: public CHeapObj {
friend class VMStructs;
private:
- MemRegion _prev_used_region; // for collectors that want to "remember" a value for
- // used region at some specific point during collection.
-
GCMemoryManager* _gc_manager;
protected:
@@ -83,13 +80,6 @@ class Generation: public CHeapObj {
Generation(ReservedSpace rs, size_t initial_byte_size);
public:
- // The set of possible generation kinds.
- enum Name {
- DefNew,
- MarkSweepCompact,
- Other
- };
-
enum SomePublicConstants {
// Generations are GenGrain-aligned and have size that are multiples of
// GenGrain.
@@ -99,9 +89,6 @@ class Generation: public CHeapObj {
GenGrain = 1 << LogOfGenGrain
};
-
- virtual Generation::Name kind() { return Generation::Other; }
-
virtual size_t capacity() const = 0; // The maximum number of object bytes the
// generation can currently hold.
virtual size_t used() const = 0; // The number of used bytes in the gen.
@@ -112,23 +99,12 @@ class Generation: public CHeapObj {
// for the allocation of objects.
virtual size_t max_capacity() const;
- // If this is a young generation, the maximum number of bytes that can be
- // allocated in this generation before a GC is triggered.
- virtual size_t capacity_before_gc() const { return 0; }
-
// The largest number of contiguous free bytes in the generation,
// including expansion (Assumes called at a safepoint.)
virtual size_t contiguous_available() const = 0;
// The largest number of contiguous free bytes in this or any higher generation.
virtual size_t max_contiguous_available() const;
- // Return an estimate of the maximum allocation that could be performed
- // in the generation without triggering any collection or expansion
- // activity. It is "unsafe" because no locks are taken; the result
- // should be treated as an approximation, not a guarantee, for use in
- // heuristic resizing decisions.
- virtual size_t unsafe_max_alloc_nogc() const = 0;
-
// Returns true if this generation cannot be expanded further
// without a GC. Override as appropriate.
virtual bool is_maximal_no_gc() const {
@@ -137,28 +113,11 @@ class Generation: public CHeapObj {
MemRegion reserved() const { return _reserved; }
- // Returns a region guaranteed to contain all the objects in the
- // generation.
- virtual MemRegion used_region() const { return _reserved; }
-
- MemRegion prev_used_region() const { return _prev_used_region; }
- virtual void save_used_region() { _prev_used_region = used_region(); }
-
- // Returns "TRUE" iff "p" points into the committed areas in the generation.
- // For some kinds of generations, this may be an expensive operation.
- // To avoid performance problems stemming from its inadvertent use in
- // product jvm's, we restrict its use to assertion checking or
- // verification only.
- virtual bool is_in(const void* p) const;
-
/* Returns "TRUE" iff "p" points into the reserved area of the generation. */
bool is_in_reserved(const void* p) const {
return _reserved.contains(p);
}
- // Iteration - do not use for time critical operations
- virtual void space_iterate(SpaceClosure* blk, bool usedOnly = false) = 0;
-
// Returns "true" iff this generation should be used to allocate an
// object of the given size. Young generations might
// wish to exclude very large objects, for example, since, if allocated
@@ -183,15 +142,6 @@ class Generation: public CHeapObj {
// Thread-local allocation buffers
virtual bool supports_tlab_allocation() const { return false; }
- // "obj" is the address of an object in a younger generation. Allocate space
- // for "obj" in the current (or some higher) generation, and copy "obj" into
- // the newly allocated space, if possible, returning the result (or null if
- // the allocation failed).
- //
- // The "obj_size" argument is just obj->size(), passed along so the caller can
- // avoid repeating the virtual call to retrieve it.
- virtual oop promote(oop obj, size_t obj_size);
-
// Returns "true" iff collect() should subsequently be called on this
// this generation. See comment below.
// This is a generic implementation which can be overridden.
@@ -224,14 +174,6 @@ class Generation: public CHeapObj {
// still unsuccessful, return "null".
virtual HeapWord* expand_and_allocate(size_t word_size, bool is_tlab) = 0;
- // Some generations may require some cleanup or preparation actions before
- // allowing a collection. The default is to do nothing.
- virtual void gc_prologue(bool full) {}
-
- // Some generations may require some cleanup actions after a collection.
- // The default is to do nothing.
- virtual void gc_epilogue(bool full) {}
-
// Save the high water marks for the used space in a generation.
virtual void record_spaces_top() {}
@@ -244,45 +186,10 @@ class Generation: public CHeapObj {
GCStats* gc_stats() const { return _gc_stats; }
virtual void update_gc_stats(Generation* current_generation, bool full) {}
- // Accessing "marks".
-
- // This function gives a generation a chance to note a point between
- // collections. For example, a contiguous generation might note the
- // beginning allocation point post-collection, which might allow some later
- // operations to be optimized.
- virtual void save_marks() {}
-
- // This function is "true" iff any no allocations have occurred in the
- // generation since the last call to "save_marks".
- virtual bool no_allocs_since_save_marks() = 0;
-
- // When an older generation has been collected, and perhaps resized,
- // this method will be invoked on all younger generations (from older to
- // younger), allowing them to resize themselves as appropriate.
- virtual void compute_new_size() = 0;
-
// Printing
virtual const char* name() const = 0;
virtual const char* short_name() const = 0;
- // Iteration.
-
- // Iterate over all objects in the generation, calling "cl.do_object" on
- // each.
- virtual void object_iterate(ObjectClosure* cl);
-
- // Block abstraction.
-
- // Returns the address of the start of the "block" that contains the
- // address "addr". We say "blocks" instead of "object" since some heaps
- // may not pack objects densely; a chunk may either be an object or a
- // non-object.
- virtual HeapWord* block_start(const void* addr) const;
-
- // Requires "addr" to be the start of a block, and returns "TRUE" iff
- // the block is an object.
- virtual bool block_is_obj(const HeapWord* addr) const;
-
virtual void print() const;
virtual void print_on(outputStream* st) const;
diff --git a/src/hotspot/share/gc/serial/serialHeap.cpp b/src/hotspot/share/gc/serial/serialHeap.cpp
index b65bab44946..ee7684c9107 100644
--- a/src/hotspot/share/gc/serial/serialHeap.cpp
+++ b/src/hotspot/share/gc/serial/serialHeap.cpp
@@ -28,7 +28,6 @@
#include "classfile/symbolTable.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "compiler/oopMap.hpp"
#include "gc/serial/cardTableRS.hpp"
#include "gc/serial/defNewGeneration.inline.hpp"
@@ -94,7 +93,6 @@ SerialHeap::SerialHeap() :
_young_gen(nullptr),
_old_gen(nullptr),
_rem_set(nullptr),
- _soft_ref_policy(),
_gc_policy_counters(new GCPolicyCounters("Copy:MSC", 2, 2)),
_incremental_collection_failed(false),
_young_manager(nullptr),
@@ -120,7 +118,7 @@ void SerialHeap::initialize_serviceability() {
young->max_survivor_size(),
false /* support_usage_threshold */);
TenuredGeneration* old = old_gen();
- _old_pool = new GenerationPool(old, "Tenured Gen", true);
+ _old_pool = new TenuredGenerationPool(old, "Tenured Gen", true);
_young_manager->add_pool(_eden_pool);
_young_manager->add_pool(_survivor_pool);
@@ -147,17 +145,6 @@ GrowableArray SerialHeap::memory_pools() {
return memory_pools;
}
-void SerialHeap::young_process_roots(OopClosure* root_closure,
- OopIterateClosure* old_gen_closure,
- CLDClosure* cld_closure) {
- MarkingCodeBlobClosure mark_code_closure(root_closure, CodeBlobToOopClosure::FixRelocations, false /* keepalive nmethods */);
-
- process_roots(SO_ScavengeCodeCache, root_closure,
- cld_closure, cld_closure, &mark_code_closure);
-
- old_gen()->younger_refs_iterate(old_gen_closure);
-}
-
void SerialHeap::safepoint_synchronize_begin() {
if (UseStringDeduplication) {
SuspendibleThreadSet::synchronize();
@@ -294,11 +281,6 @@ size_t SerialHeap::used() const {
return _young_gen->used() + _old_gen->used();
}
-void SerialHeap::save_used_regions() {
- _old_gen->save_used_region();
- _young_gen->save_used_region();
-}
-
size_t SerialHeap::max_capacity() const {
return _young_gen->max_capacity() + _old_gen->max_capacity();
}
@@ -924,12 +906,15 @@ HeapWord* SerialHeap::block_start(const void* addr) const {
bool SerialHeap::block_is_obj(const HeapWord* addr) const {
assert(is_in_reserved(addr), "block_is_obj of address outside of heap");
assert(block_start(addr) == addr, "addr must be a block start");
+
if (_young_gen->is_in_reserved(addr)) {
- return _young_gen->block_is_obj(addr);
+ return _young_gen->eden()->is_in(addr)
+ || _young_gen->from()->is_in(addr)
+ || _young_gen->to() ->is_in(addr);
}
- assert(_old_gen->is_in_reserved(addr), "Some generation should contain the address");
- return _old_gen->block_is_obj(addr);
+ assert(_old_gen->is_in_reserved(addr), "must be in old-gen");
+ return addr < _old_gen->space()->top();
}
size_t SerialHeap::tlab_capacity(Thread* thr) const {
@@ -1052,35 +1037,11 @@ void SerialHeap::print_heap_change(const PreGenGCValues& pre_gc_values) const {
MetaspaceUtils::print_metaspace_change(pre_gc_values.metaspace_sizes());
}
-class GenGCPrologueClosure: public SerialHeap::GenClosure {
- private:
- bool _full;
- public:
- void do_generation(Generation* gen) {
- gen->gc_prologue(_full);
- }
- GenGCPrologueClosure(bool full) : _full(full) {};
-};
-
void SerialHeap::gc_prologue(bool full) {
- assert(InlineCacheBuffer::is_empty(), "should have cleaned up ICBuffer");
-
// Fill TLAB's and such
ensure_parsability(true); // retire TLABs
- // Walk generations
- GenGCPrologueClosure blk(full);
- generation_iterate(&blk, false); // not old-to-young.
-};
-
-class GenGCEpilogueClosure: public SerialHeap::GenClosure {
- private:
- bool _full;
- public:
- void do_generation(Generation* gen) {
- gen->gc_epilogue(_full);
- }
- GenGCEpilogueClosure(bool full) : _full(full) {};
+ _old_gen->gc_prologue();
};
void SerialHeap::gc_epilogue(bool full) {
@@ -1090,8 +1051,8 @@ void SerialHeap::gc_epilogue(bool full) {
resize_all_tlabs();
- GenGCEpilogueClosure blk(full);
- generation_iterate(&blk, false); // not old-to-young.
+ _young_gen->gc_epilogue(full);
+ _old_gen->gc_epilogue();
MetaspaceCounters::update_performance_counters();
};
diff --git a/src/hotspot/share/gc/serial/serialHeap.hpp b/src/hotspot/share/gc/serial/serialHeap.hpp
index 71880e7c771..2b04d04eaea 100644
--- a/src/hotspot/share/gc/serial/serialHeap.hpp
+++ b/src/hotspot/share/gc/serial/serialHeap.hpp
@@ -90,8 +90,6 @@ class SerialHeap : public CollectedHeap {
// The singleton CardTable Remembered Set.
CardTableRS* _rem_set;
- SoftRefPolicy _soft_ref_policy;
-
GCPolicyCounters* _gc_policy_counters;
// Indicates that the most recent previous incremental collection failed.
@@ -154,17 +152,12 @@ class SerialHeap : public CollectedHeap {
MemRegion reserved_region() const { return _reserved; }
bool is_in_reserved(const void* addr) const { return _reserved.contains(addr); }
- SoftRefPolicy* soft_ref_policy() override { return &_soft_ref_policy; }
-
// Performance Counter support
GCPolicyCounters* counters() { return _gc_policy_counters; }
size_t capacity() const override;
size_t used() const override;
- // Save the "used_region" for both generations.
- void save_used_regions();
-
size_t max_capacity() const override;
HeapWord* mem_allocate(size_t size, bool* gc_overhead_limit_was_exceeded) override;
@@ -364,13 +357,11 @@ class SerialHeap : public CollectedHeap {
GrowableArray memory_pools() override;
DefNewGeneration* young_gen() const {
- assert(_young_gen->kind() == Generation::DefNew, "Wrong generation type");
- return static_cast(_young_gen);
+ return _young_gen;
}
TenuredGeneration* old_gen() const {
- assert(_old_gen->kind() == Generation::MarkSweepCompact, "Wrong generation type");
- return static_cast(_old_gen);
+ return _old_gen;
}
// Apply "cur->do_oop" or "older->do_oop" to all the oops in objects
@@ -381,10 +372,6 @@ class SerialHeap : public CollectedHeap {
void oop_since_save_marks_iterate(OopClosureType1* cur,
OopClosureType2* older);
- void young_process_roots(OopClosure* root_closure,
- OopIterateClosure* old_gen_closure,
- CLDClosure* cld_closure);
-
void safepoint_synchronize_begin() override;
void safepoint_synchronize_end() override;
diff --git a/src/hotspot/share/gc/serial/serialHeap.inline.hpp b/src/hotspot/share/gc/serial/serialHeap.inline.hpp
index 6e06c7b6d57..27819f0d1c9 100644
--- a/src/hotspot/share/gc/serial/serialHeap.inline.hpp
+++ b/src/hotspot/share/gc/serial/serialHeap.inline.hpp
@@ -37,4 +37,89 @@ void SerialHeap::oop_since_save_marks_iterate(OopClosureType1* cur,
old_gen()->oop_since_save_marks_iterate(older);
}
+class ScavengeHelper {
+ DefNewGeneration* _young_gen;
+ HeapWord* _young_gen_end;
+public:
+ ScavengeHelper(DefNewGeneration* young_gen) :
+ _young_gen(young_gen),
+ _young_gen_end(young_gen->reserved().end()) {}
+
+ bool is_in_young_gen(void* p) const {
+ return p < _young_gen_end;
+ }
+
+ template
+ void try_scavenge(T* p, Func&& f) {
+ T heap_oop = RawAccess<>::oop_load(p);
+ // Should we copy the obj?
+ if (!CompressedOops::is_null(heap_oop)) {
+ oop obj = CompressedOops::decode_not_null(heap_oop);
+ if (is_in_young_gen(obj)) {
+ assert(!_young_gen->to()->is_in_reserved(obj), "Scanning field twice?");
+ oop new_obj = obj->is_forwarded() ? obj->forwardee()
+ : _young_gen->copy_to_survivor_space(obj);
+ RawAccess::oop_store(p, new_obj);
+
+ // callback
+ f(new_obj);
+ }
+ }
+ }
+};
+
+class InHeapScanClosure : public BasicOopIterateClosure {
+ ScavengeHelper _helper;
+protected:
+ bool is_in_young_gen(void* p) const {
+ return _helper.is_in_young_gen(p);
+ }
+
+ template
+ void try_scavenge(T* p, Func&& f) {
+ _helper.try_scavenge(p, f);
+ }
+
+ InHeapScanClosure(DefNewGeneration* young_gen) :
+ BasicOopIterateClosure(young_gen->ref_processor()),
+ _helper(young_gen) {}
+};
+
+class OffHeapScanClosure : public OopClosure {
+ ScavengeHelper _helper;
+protected:
+ bool is_in_young_gen(void* p) const {
+ return _helper.is_in_young_gen(p);
+ }
+
+ template
+ void try_scavenge(T* p, Func&& f) {
+ _helper.try_scavenge(p, f);
+ }
+
+ OffHeapScanClosure(DefNewGeneration* young_gen) : _helper(young_gen) {}
+};
+
+class OldGenScanClosure : public InHeapScanClosure {
+ CardTableRS* _rs;
+
+ template
+ void do_oop_work(T* p) {
+ assert(!is_in_young_gen(p), "precondition");
+
+ try_scavenge(p, [&] (oop new_obj) {
+ // If p points to a younger generation, mark the card.
+ if (is_in_young_gen(new_obj)) {
+ _rs->inline_write_ref_field_gc(p);
+ }
+ });
+ }
+public:
+ OldGenScanClosure(DefNewGeneration* g) : InHeapScanClosure(g),
+ _rs(SerialHeap::heap()->rem_set()) {}
+
+ void do_oop(oop* p) { do_oop_work(p); }
+ void do_oop(narrowOop* p) { do_oop_work(p); }
+};
+
#endif // SHARE_GC_SERIAL_SERIALHEAP_INLINE_HPP
diff --git a/src/hotspot/share/gc/serial/serialMemoryPools.cpp b/src/hotspot/share/gc/serial/serialMemoryPools.cpp
index f688771f609..b82252b5a09 100644
--- a/src/hotspot/share/gc/serial/serialMemoryPools.cpp
+++ b/src/hotspot/share/gc/serial/serialMemoryPools.cpp
@@ -24,8 +24,8 @@
#include "precompiled.hpp"
#include "gc/serial/defNewGeneration.hpp"
-#include "gc/serial/generation.hpp"
#include "gc/serial/serialMemoryPools.hpp"
+#include "gc/serial/tenuredGeneration.hpp"
#include "gc/shared/space.hpp"
ContiguousSpacePool::ContiguousSpacePool(ContiguousSpace* space,
@@ -72,18 +72,18 @@ MemoryUsage SurvivorContiguousSpacePool::get_memory_usage() {
return MemoryUsage(initial_size(), used, committed, maxSize);
}
-GenerationPool::GenerationPool(Generation* gen,
- const char* name,
- bool support_usage_threshold) :
+TenuredGenerationPool::TenuredGenerationPool(TenuredGeneration* gen,
+ const char* name,
+ bool support_usage_threshold) :
CollectedMemoryPool(name, gen->capacity(), gen->max_capacity(),
support_usage_threshold), _gen(gen) {
}
-size_t GenerationPool::used_in_bytes() {
+size_t TenuredGenerationPool::used_in_bytes() {
return _gen->used();
}
-MemoryUsage GenerationPool::get_memory_usage() {
+MemoryUsage TenuredGenerationPool::get_memory_usage() {
size_t used = used_in_bytes();
size_t committed = _gen->capacity();
size_t maxSize = (available_for_allocation() ? max_size() : 0);
diff --git a/src/hotspot/share/gc/serial/serialMemoryPools.hpp b/src/hotspot/share/gc/serial/serialMemoryPools.hpp
index aa89e4e4c38..36cd9789055 100644
--- a/src/hotspot/share/gc/serial/serialMemoryPools.hpp
+++ b/src/hotspot/share/gc/serial/serialMemoryPools.hpp
@@ -62,11 +62,11 @@ class SurvivorContiguousSpacePool : public CollectedMemoryPool {
size_t committed_in_bytes();
};
-class GenerationPool : public CollectedMemoryPool {
+class TenuredGenerationPool : public CollectedMemoryPool {
private:
- Generation* _gen;
+ TenuredGeneration* _gen;
public:
- GenerationPool(Generation* gen, const char* name, bool support_usage_threshold);
+ TenuredGenerationPool(TenuredGeneration* gen, const char* name, bool support_usage_threshold);
MemoryUsage get_memory_usage();
size_t used_in_bytes();
diff --git a/src/hotspot/share/gc/serial/tenuredGeneration.cpp b/src/hotspot/share/gc/serial/tenuredGeneration.cpp
index ad5fda3aeab..fe37d997549 100644
--- a/src/hotspot/share/gc/serial/tenuredGeneration.cpp
+++ b/src/hotspot/share/gc/serial/tenuredGeneration.cpp
@@ -263,19 +263,12 @@ void TenuredGeneration::compute_new_size_inner() {
}
}
-void TenuredGeneration::space_iterate(SpaceClosure* blk,
- bool usedOnly) {
- blk->do_space(space());
+HeapWord* TenuredGeneration::block_start(const void* p) const {
+ return space()->block_start_const(p);
}
-void TenuredGeneration::younger_refs_iterate(OopIterateClosure* blk) {
- // Apply "cl->do_oop" to (the address of) (exactly) all the ref fields in
- // "sp" that point into the young generation.
- // The iteration is only over objects allocated at the start of the
- // iterations; objects allocated as a result of applying the closure are
- // not included.
-
- _rs->younger_refs_in_space_iterate(space(), blk);
+void TenuredGeneration::scan_old_to_young_refs() {
+ _rs->scan_old_to_young_refs(space());
}
TenuredGeneration::TenuredGeneration(ReservedSpace rs,
@@ -331,7 +324,7 @@ TenuredGeneration::TenuredGeneration(ReservedSpace rs,
_the_space, _gen_counters);
}
-void TenuredGeneration::gc_prologue(bool full) {
+void TenuredGeneration::gc_prologue() {
_capacity_at_prologue = capacity();
_used_at_prologue = used();
}
@@ -420,6 +413,31 @@ bool TenuredGeneration::promotion_attempt_is_safe(size_t max_promotion_in_bytes)
return res;
}
+oop TenuredGeneration::promote(oop obj, size_t obj_size) {
+ assert(obj_size == obj->size(), "bad obj_size passed in");
+
+#ifndef PRODUCT
+ if (SerialHeap::heap()->promotion_should_fail()) {
+ return nullptr;
+ }
+#endif // #ifndef PRODUCT
+
+ // Allocate new object.
+ HeapWord* result = allocate(obj_size, false);
+ if (result == nullptr) {
+ // Promotion of obj into gen failed. Try to expand and allocate.
+ result = expand_and_allocate(obj_size, false);
+ if (result == nullptr) {
+ return nullptr;
+ }
+ }
+
+ // Copy to new location.
+ Copy::aligned_disjoint_words(cast_from_oop(obj), result, obj_size);
+ oop new_obj = cast_to_oop(result);
+ return new_obj;
+}
+
void TenuredGeneration::collect(bool full,
bool clear_all_soft_refs,
size_t size,
@@ -450,10 +468,6 @@ TenuredGeneration::expand_and_allocate(size_t word_size, bool is_tlab) {
return _the_space->allocate(word_size);
}
-size_t TenuredGeneration::unsafe_max_alloc_nogc() const {
- return _the_space->free();
-}
-
size_t TenuredGeneration::contiguous_available() const {
return _the_space->free() + _virtual_space.uncommitted_size();
}
@@ -485,7 +499,7 @@ bool TenuredGeneration::no_allocs_since_save_marks() {
return _the_space->saved_mark_at_top();
}
-void TenuredGeneration::gc_epilogue(bool full) {
+void TenuredGeneration::gc_epilogue() {
// update the generation and space performance counters
update_counters();
if (ZapUnusedHeapArea) {
diff --git a/src/hotspot/share/gc/serial/tenuredGeneration.hpp b/src/hotspot/share/gc/serial/tenuredGeneration.hpp
index 2a1ae18b541..a2461540a35 100644
--- a/src/hotspot/share/gc/serial/tenuredGeneration.hpp
+++ b/src/hotspot/share/gc/serial/tenuredGeneration.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -29,6 +29,7 @@
#include "gc/serial/generation.hpp"
#include "gc/shared/gcStats.hpp"
#include "gc/shared/generationCounters.hpp"
+#include "gc/shared/space.hpp"
#include "utilities/macros.hpp"
class SerialBlockOffsetSharedArray;
@@ -45,7 +46,7 @@ class TenuredGeneration: public Generation {
// Abstractly, this is a subtype that gets access to protected fields.
friend class VM_PopulateDumpSharedSpace;
- protected:
+ MemRegion _prev_used_region;
// This is shared with other generations.
CardTableRS* _rs;
@@ -70,7 +71,6 @@ class TenuredGeneration: public Generation {
GenerationCounters* _gen_counters;
CSpaceCounters* _space_counters;
-
// Attempt to expand the generation by "bytes". Expand by at a
// minimum "expand_bytes". Return true if some amount (not
// necessarily the full "bytes") was done.
@@ -81,7 +81,7 @@ class TenuredGeneration: public Generation {
void compute_new_size_inner();
public:
- virtual void compute_new_size();
+ void compute_new_size();
TenuredSpace* space() const { return _the_space; }
@@ -93,11 +93,14 @@ class TenuredGeneration: public Generation {
size_t capacity() const;
size_t used() const;
size_t free() const;
- MemRegion used_region() const;
- void space_iterate(SpaceClosure* blk, bool usedOnly = false);
+ MemRegion used_region() const { return space()->used_region(); }
+ MemRegion prev_used_region() const { return _prev_used_region; }
+ void save_used_region() { _prev_used_region = used_region(); }
+
+ HeapWord* block_start(const void* p) const;
- void younger_refs_iterate(OopIterateClosure* blk);
+ void scan_old_to_young_refs();
bool is_in(const void* p) const;
@@ -107,13 +110,10 @@ class TenuredGeneration: public Generation {
size_t max_byte_size,
CardTableRS* remset);
- Generation::Name kind() { return Generation::MarkSweepCompact; }
-
// Printing
const char* name() const { return "tenured generation"; }
const char* short_name() const { return "Tenured"; }
- size_t unsafe_max_alloc_nogc() const;
size_t contiguous_available() const;
// Iteration
@@ -131,8 +131,6 @@ class TenuredGeneration: public Generation {
bool no_allocs_since_save_marks();
- inline bool block_is_obj(const HeapWord* addr) const;
-
virtual void collect(bool full,
bool clear_all_soft_refs,
size_t size,
@@ -140,8 +138,8 @@ class TenuredGeneration: public Generation {
HeapWord* expand_and_allocate(size_t size, bool is_tlab);
- virtual void gc_prologue(bool full);
- virtual void gc_epilogue(bool full);
+ void gc_prologue();
+ void gc_epilogue();
bool should_collect(bool full,
size_t word_size,
@@ -162,6 +160,14 @@ class TenuredGeneration: public Generation {
// might be attempted in the worst case.
bool promotion_attempt_is_safe(size_t max_promoted_in_bytes) const;
+ // "obj" is the address of an object in young-gen. Allocate space for "obj"
+ // in the old-gen, and copy "obj" into the newly allocated space, if
+ // possible, returning the result (or null if the allocation failed).
+ //
+ // The "obj_size" argument is just obj->size(), passed along so the caller can
+ // avoid repeating the virtual call to retrieve it.
+ oop promote(oop obj, size_t obj_size);
+
virtual void verify();
virtual void print_on(outputStream* st) const;
};
diff --git a/src/hotspot/share/gc/serial/tenuredGeneration.inline.hpp b/src/hotspot/share/gc/serial/tenuredGeneration.inline.hpp
index 31d32745b0d..25d57a76282 100644
--- a/src/hotspot/share/gc/serial/tenuredGeneration.inline.hpp
+++ b/src/hotspot/share/gc/serial/tenuredGeneration.inline.hpp
@@ -41,10 +41,6 @@ inline size_t TenuredGeneration::free() const {
return space()->free();
}
-inline MemRegion TenuredGeneration::used_region() const {
- return space()->used_region();
-}
-
inline bool TenuredGeneration::is_in(const void* p) const {
return space()->is_in(p);
}
@@ -61,15 +57,9 @@ HeapWord* TenuredGeneration::par_allocate(size_t word_size,
return _the_space->par_allocate(word_size);
}
-bool TenuredGeneration::block_is_obj(const HeapWord* addr) const {
- return addr < _the_space ->top();
-}
-
template
void TenuredGeneration::oop_since_save_marks_iterate(OopClosureType* blk) {
_the_space->oop_since_save_marks_iterate(blk);
-
- save_marks();
}
#endif // SHARE_GC_SERIAL_TENUREDGENERATION_INLINE_HPP
diff --git a/src/hotspot/share/gc/shared/ageTable.cpp b/src/hotspot/share/gc/shared/ageTable.cpp
index 0c2bcda73ed..34e4bb91887 100644
--- a/src/hotspot/share/gc/shared/ageTable.cpp
+++ b/src/hotspot/share/gc/shared/ageTable.cpp
@@ -71,6 +71,15 @@ void AgeTable::clear() {
}
}
+#ifndef PRODUCT
+bool AgeTable::is_clear() const {
+ for (const size_t* p = sizes; p < sizes + table_size; ++p) {
+ if (*p != 0) return false;
+ }
+ return true;
+}
+#endif // !PRODUCT
+
void AgeTable::merge(const AgeTable* subTable) {
for (int i = 0; i < table_size; i++) {
sizes[i]+= subTable->sizes[i];
diff --git a/src/hotspot/share/gc/shared/ageTable.hpp b/src/hotspot/share/gc/shared/ageTable.hpp
index 9f0c10ec312..decade6ee85 100644
--- a/src/hotspot/share/gc/shared/ageTable.hpp
+++ b/src/hotspot/share/gc/shared/ageTable.hpp
@@ -25,6 +25,7 @@
#ifndef SHARE_GC_SHARED_AGETABLE_HPP
#define SHARE_GC_SHARED_AGETABLE_HPP
+#include "memory/allocation.hpp"
#include "oops/markWord.hpp"
#include "oops/oop.hpp"
#include "runtime/perfDataTypes.hpp"
@@ -36,7 +37,7 @@
//
// Note: all sizes are in oops
-class AgeTable {
+class AgeTable: public CHeapObj {
friend class VMStructs;
public:
@@ -53,16 +54,20 @@ class AgeTable {
// clear table
void clear();
+#ifndef PRODUCT
+ // check whether it's clear
+ bool is_clear() const;
+#endif // !PRODUCT
+
// add entry
inline void add(oop p, size_t oop_size);
void add(uint age, size_t oop_size) {
- assert(age > 0 && age < table_size, "invalid age of object");
+ assert(age < table_size, "invalid age of object");
sizes[age] += oop_size;
}
- // Merge another age table with the current one. Used
- // for parallel young generation gc.
+ // Merge another age table with the current one.
void merge(const AgeTable* subTable);
// Calculate new tenuring threshold based on age information.
diff --git a/src/hotspot/share/gc/shared/collectedHeap.cpp b/src/hotspot/share/gc/shared/collectedHeap.cpp
index 1b65ee90b5d..dbcef1ba575 100644
--- a/src/hotspot/share/gc/shared/collectedHeap.cpp
+++ b/src/hotspot/share/gc/shared/collectedHeap.cpp
@@ -242,6 +242,7 @@ bool CollectedHeap::is_oop(oop object) const {
CollectedHeap::CollectedHeap() :
_capacity_at_last_gc(0),
_used_at_last_gc(0),
+ _soft_ref_policy(),
_is_gc_active(false),
_last_whole_heap_examined_time_ns(os::javaTimeNanos()),
_total_collections(0),
diff --git a/src/hotspot/share/gc/shared/collectedHeap.hpp b/src/hotspot/share/gc/shared/collectedHeap.hpp
index 0ac7faa06f0..b503d38c505 100644
--- a/src/hotspot/share/gc/shared/collectedHeap.hpp
+++ b/src/hotspot/share/gc/shared/collectedHeap.hpp
@@ -27,6 +27,7 @@
#include "gc/shared/gcCause.hpp"
#include "gc/shared/gcWhen.hpp"
+#include "gc/shared/softRefPolicy.hpp"
#include "gc/shared/verifyOption.hpp"
#include "memory/allocation.hpp"
#include "memory/metaspace.hpp"
@@ -56,7 +57,6 @@ class GCMemoryManager;
class MemoryPool;
class MetaspaceSummary;
class ReservedHeapSpace;
-class SoftRefPolicy;
class Thread;
class ThreadClosure;
class VirtualSpaceSummary;
@@ -105,6 +105,8 @@ class CollectedHeap : public CHeapObj {
size_t _capacity_at_last_gc;
size_t _used_at_last_gc;
+ SoftRefPolicy _soft_ref_policy;
+
// First, set it to java_lang_Object.
// Then, set it to FillerObject after the FillerObject_klass loading is complete.
static Klass* _filler_object_klass;
@@ -403,7 +405,7 @@ class CollectedHeap : public CHeapObj {
void increment_total_full_collections() { _total_full_collections++; }
// Return the SoftRefPolicy for the heap;
- virtual SoftRefPolicy* soft_ref_policy() = 0;
+ SoftRefPolicy* soft_ref_policy() { return &_soft_ref_policy; }
virtual MemoryUsage memory_usage();
virtual GrowableArray memory_managers() = 0;
diff --git a/src/hotspot/share/gc/shared/gcId.cpp b/src/hotspot/share/gc/shared/gcId.cpp
index 646de833910..8d123386180 100644
--- a/src/hotspot/share/gc/shared/gcId.cpp
+++ b/src/hotspot/share/gc/shared/gcId.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -43,7 +43,7 @@ void GCId::set_printer(GCIdPrinter* printer) {
_printer = printer;
}
-NamedThread* currentNamedthread() {
+static NamedThread* currentNamedthread() {
assert(Thread::current()->is_Named_thread(), "This thread must be NamedThread");
return (NamedThread*)Thread::current();
}
diff --git a/src/hotspot/share/gc/shared/gc_globals.hpp b/src/hotspot/share/gc/shared/gc_globals.hpp
index ae8674c84a9..652153922d1 100644
--- a/src/hotspot/share/gc/shared/gc_globals.hpp
+++ b/src/hotspot/share/gc/shared/gc_globals.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -309,23 +309,6 @@
"MaxRAM * MaxRAMPercentage / 100") \
range(0, max_uintx) \
\
- product(uintx, MaxRAMFraction, 4, \
- "Maximum fraction (1/n) of real memory used for maximum heap " \
- "size. " \
- "Deprecated, use MaxRAMPercentage instead") \
- range(1, max_uintx) \
- \
- product(uintx, MinRAMFraction, 2, \
- "Minimum fraction (1/n) of real memory used for maximum heap " \
- "size on systems with small physical memory size. " \
- "Deprecated, use MinRAMPercentage instead") \
- range(1, max_uintx) \
- \
- product(uintx, InitialRAMFraction, 64, \
- "Fraction (1/n) of real memory used for initial heap size. " \
- "Deprecated, use InitialRAMPercentage instead") \
- range(1, max_uintx) \
- \
product(double, MaxRAMPercentage, 25.0, \
"Maximum percentage of real memory used for maximum heap size") \
range(0.0, 100.0) \
@@ -442,11 +425,6 @@
"Time slice for MMU specification") \
constraint(GCPauseIntervalMillisConstraintFunc,AfterErgo) \
\
- product(uintx, MaxGCMinorPauseMillis, max_uintx, \
- "Adaptive size policy maximum GC minor pause time goal " \
- "in millisecond") \
- range(0, max_uintx) \
- \
product(uint, GCTimeRatio, 99, \
"Adaptive size policy application time to GC time ratio") \
range(0, UINT_MAX) \
diff --git a/src/hotspot/share/gc/shared/parallelCleaning.cpp b/src/hotspot/share/gc/shared/parallelCleaning.cpp
index 396f7e6c31e..ef82c8b676e 100644
--- a/src/hotspot/share/gc/shared/parallelCleaning.cpp
+++ b/src/hotspot/share/gc/shared/parallelCleaning.cpp
@@ -47,7 +47,6 @@ CodeCacheUnloadingTask::CodeCacheUnloadingTask(uint num_workers, bool unloading_
CodeCacheUnloadingTask::~CodeCacheUnloadingTask() {
CodeCache::verify_clean_inline_caches();
- CodeCache::verify_icholder_relocations();
}
void CodeCacheUnloadingTask::claim_nmethods(CompiledMethod** claimed_nmethods, int *num_claimed_nmethods) {
diff --git a/src/hotspot/share/gc/shared/pretouchTask.cpp b/src/hotspot/share/gc/shared/pretouchTask.cpp
index 5e35baf03de..aba7a715fd3 100644
--- a/src/hotspot/share/gc/shared/pretouchTask.cpp
+++ b/src/hotspot/share/gc/shared/pretouchTask.cpp
@@ -68,12 +68,6 @@ void PretouchTask::pretouch(const char* task_name, char* start_address, char* en
// Page-align the chunk size, so if start_address is also page-aligned (as
// is common) then there won't be any pages shared by multiple chunks.
size_t chunk_size = align_down_bounded(PretouchTask::chunk_size(), page_size);
-#ifdef LINUX
- // When using THP we need to always pre-touch using small pages as the OS will
- // initially always use small pages.
- page_size = UseTransparentHugePages ? (size_t)os::vm_page_size() : page_size;
-#endif
-
PretouchTask task(task_name, start_address, end_address, page_size, chunk_size);
size_t total_bytes = pointer_delta(end_address, start_address, sizeof(char));
diff --git a/src/hotspot/share/gc/shared/space.cpp b/src/hotspot/share/gc/shared/space.cpp
index 1a9e726625a..82e32a17b43 100644
--- a/src/hotspot/share/gc/shared/space.cpp
+++ b/src/hotspot/share/gc/shared/space.cpp
@@ -140,20 +140,12 @@ void ContiguousSpace::verify() const {
guarantee(p == top(), "end of last object must match end of space");
}
-bool Space::obj_is_alive(const HeapWord* p) const {
- assert (block_is_obj(p), "The address should point to an object");
- return true;
-}
-
void ContiguousSpace::object_iterate(ObjectClosure* blk) {
- if (is_empty()) return;
- object_iterate_from(bottom(), blk);
-}
-
-void ContiguousSpace::object_iterate_from(HeapWord* mark, ObjectClosure* blk) {
- while (mark < top()) {
- blk->do_object(cast_to_oop(mark));
- mark += cast_to_oop(mark)->size();
+ HeapWord* addr = bottom();
+ while (addr < top()) {
+ oop obj = cast_to_oop(addr);
+ blk->do_object(obj);
+ addr += obj->size();
}
}
@@ -205,7 +197,7 @@ inline HeapWord* ContiguousSpace::allocate_impl(size_t size) {
if (pointer_delta(end(), obj) >= size) {
HeapWord* new_top = obj + size;
set_top(new_top);
- assert(is_aligned(obj) && is_aligned(new_top), "checking alignment");
+ assert(is_object_aligned(obj) && is_object_aligned(new_top), "checking alignment");
return obj;
} else {
return nullptr;
@@ -223,7 +215,7 @@ inline HeapWord* ContiguousSpace::par_allocate_impl(size_t size) {
// the old top value: the exchange succeeded
// otherwise: the new value of the top is returned.
if (result == obj) {
- assert(is_aligned(obj) && is_aligned(new_top), "checking alignment");
+ assert(is_object_aligned(obj) && is_object_aligned(new_top), "checking alignment");
return obj;
}
} else {
diff --git a/src/hotspot/share/gc/shared/space.hpp b/src/hotspot/share/gc/shared/space.hpp
index 85f870ba1e2..41c1d2a0d75 100644
--- a/src/hotspot/share/gc/shared/space.hpp
+++ b/src/hotspot/share/gc/shared/space.hpp
@@ -77,13 +77,11 @@ class Space: public CHeapObj {
// Accessors
HeapWord* bottom() const { return _bottom; }
HeapWord* end() const { return _end; }
- virtual void set_bottom(HeapWord* value) { _bottom = value; }
- virtual void set_end(HeapWord* value) { _end = value; }
+ void set_bottom(HeapWord* value) { _bottom = value; }
+ void set_end(HeapWord* value) { _end = value; }
HeapWord* saved_mark_word() const { return _saved_mark_word; }
- void set_saved_mark_word(HeapWord* p) { _saved_mark_word = p; }
-
// Returns a subregion of the space containing only the allocated objects in
// the space.
virtual MemRegion used_region() const = 0;
@@ -119,54 +117,27 @@ class Space: public CHeapObj {
bool is_in(const void* p) const {
return used_region().contains(p);
}
- bool is_in(oop obj) const {
- return is_in((void*)obj);
- }
// Returns true iff the given reserved memory of the space contains the
// given address.
bool is_in_reserved(const void* p) const { return _bottom <= p && p < _end; }
- // Test whether p is double-aligned
- static bool is_aligned(void* p) {
- return ::is_aligned(p, sizeof(double));
- }
-
// Size computations. Sizes are in bytes.
size_t capacity() const { return byte_size(bottom(), end()); }
virtual size_t used() const = 0;
virtual size_t free() const = 0;
- // Iterate over all objects in the space, calling "cl.do_object" on
- // each. Objects allocated by applications of the closure are not
- // included in the iteration.
- virtual void object_iterate(ObjectClosure* blk) = 0;
-
// If "p" is in the space, returns the address of the start of the
// "block" that contains "p". We say "block" instead of "object" since
// some heaps may not pack objects densely; a chunk may either be an
// object or a non-object. If "p" is not in the space, return null.
virtual HeapWord* block_start_const(const void* p) const = 0;
- // The non-const version may have benevolent side effects on the data
- // structure supporting these calls, possibly speeding up future calls.
- // The default implementation, however, is simply to call the const
- // version.
- HeapWord* block_start(const void* p);
-
// Requires "addr" to be the start of a chunk, and returns its size.
// "addr + size" is required to be the start of a new chunk, or the end
// of the active area of the heap.
virtual size_t block_size(const HeapWord* addr) const = 0;
- // Requires "addr" to be the start of a block, and returns "TRUE" iff
- // the block is an object.
- virtual bool block_is_obj(const HeapWord* addr) const = 0;
-
- // Requires "addr" to be the start of a block, and returns "TRUE" iff
- // the block is an object and the object is alive.
- bool obj_is_alive(const HeapWord* addr) const;
-
// Allocation (return null if full). Assumes the caller has established
// mutually exclusive access to the space.
virtual HeapWord* allocate(size_t word_size) = 0;
@@ -273,7 +244,7 @@ class ContiguousSpace: public Space {
HeapWord* par_allocate(size_t word_size) override;
// Iteration
- void object_iterate(ObjectClosure* blk) override;
+ void object_iterate(ObjectClosure* blk);
// Apply "blk->do_oop" to the addresses of all reference fields in objects
// starting with the _saved_mark_word, which was noted during a generation's
@@ -285,16 +256,9 @@ class ContiguousSpace: public Space {
template
void oop_since_save_marks_iterate(OopClosureType* blk);
- // Same as object_iterate, but starting from "mark", which is required
- // to denote the start of an object. Objects allocated by
- // applications of the closure *are* included in the iteration.
- virtual void object_iterate_from(HeapWord* mark, ObjectClosure* blk);
-
// Very inefficient implementation.
HeapWord* block_start_const(const void* p) const override;
size_t block_size(const HeapWord* p) const override;
- // If a block is in the allocated area, it is an object.
- bool block_is_obj(const HeapWord* p) const override { return p < top(); }
// Addresses for inlined allocation
HeapWord** top_addr() { return &_top; }
diff --git a/src/hotspot/share/gc/shared/space.inline.hpp b/src/hotspot/share/gc/shared/space.inline.hpp
index bfaf84e8fac..e34f833775a 100644
--- a/src/hotspot/share/gc/shared/space.inline.hpp
+++ b/src/hotspot/share/gc/shared/space.inline.hpp
@@ -34,10 +34,6 @@
#include "runtime/prefetch.inline.hpp"
#include "runtime/safepoint.hpp"
-inline HeapWord* Space::block_start(const void* p) {
- return block_start_const(p);
-}
-
#if INCLUDE_SERIALGC
inline HeapWord* TenuredSpace::allocate(size_t size) {
HeapWord* res = ContiguousSpace::allocate(size);
@@ -71,13 +67,12 @@ void ContiguousSpace::oop_since_save_marks_iterate(OopClosureType* blk) {
t = top();
while (p < t) {
Prefetch::write(p, interval);
- debug_only(HeapWord* prev = p);
oop m = cast_to_oop(p);
p += m->oop_iterate_size(blk);
}
} while (t < top());
- set_saved_mark_word(p);
+ set_saved_mark();
}
#endif // SHARE_GC_SHARED_SPACE_INLINE_HPP
diff --git a/src/hotspot/share/gc/shared/tlab_globals.hpp b/src/hotspot/share/gc/shared/tlab_globals.hpp
index 0b047b4b9e8..0c135d490ec 100644
--- a/src/hotspot/share/gc/shared/tlab_globals.hpp
+++ b/src/hotspot/share/gc/shared/tlab_globals.hpp
@@ -46,9 +46,6 @@
product(bool, ZeroTLAB, false, \
"Zero out the newly created TLAB") \
\
- product(bool, TLABStats, true, \
- "Provide more detailed and expensive TLAB statistics.") \
- \
product(size_t, MinTLABSize, 2*K, \
"Minimum allowed TLAB size (in bytes)") \
range(1, max_uintx/2) \
diff --git a/src/hotspot/share/gc/shared/vmStructs_gc.hpp b/src/hotspot/share/gc/shared/vmStructs_gc.hpp
index 472a02c012a..69bb8c848e8 100644
--- a/src/hotspot/share/gc/shared/vmStructs_gc.hpp
+++ b/src/hotspot/share/gc/shared/vmStructs_gc.hpp
@@ -213,12 +213,6 @@
declare_constant(CollectedHeap::Parallel) \
declare_constant(CollectedHeap::G1) \
\
- /* constants from Generation::Name enum */ \
- \
- declare_constant(Generation::DefNew) \
- declare_constant(Generation::MarkSweepCompact) \
- declare_constant(Generation::Other) \
- \
declare_constant(Generation::LogOfGenGrain) \
declare_constant(Generation::GenGrain) \
diff --git a/src/hotspot/share/gc/shared/weakProcessor.cpp b/src/hotspot/share/gc/shared/weakProcessor.cpp
index 02ef9bbe88d..8adf3fab7d5 100644
--- a/src/hotspot/share/gc/shared/weakProcessor.cpp
+++ b/src/hotspot/share/gc/shared/weakProcessor.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -41,7 +41,7 @@
#include "prims/jvmtiTagMap.hpp"
#endif // INCLUDE_JVMTI
-void notify_jvmti_tagmaps() {
+static void notify_jvmti_tagmaps() {
#if INCLUDE_JVMTI
// Notify JVMTI tagmaps that a STW weak reference processing might be
// clearing entries, so the tagmaps need cleaning. Doing this here allows
diff --git a/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp b/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp
index c2620a22811..2d5af892c80 100644
--- a/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp
+++ b/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.cpp
@@ -45,8 +45,6 @@ int ShenandoahHeuristics::compare_by_garbage(RegionData a, RegionData b) {
ShenandoahHeuristics::ShenandoahHeuristics(ShenandoahSpaceInfo* space_info) :
_space_info(space_info),
_region_data(nullptr),
- _degenerated_cycles_in_a_row(0),
- _successful_cycles_in_a_row(0),
_cycle_start(os::elapsedTime()),
_last_cycle_end(0),
_gc_times_learned(0),
@@ -199,7 +197,7 @@ bool ShenandoahHeuristics::should_start_gc() {
}
bool ShenandoahHeuristics::should_degenerate_cycle() {
- return _degenerated_cycles_in_a_row <= ShenandoahFullGCThreshold;
+ return ShenandoahHeap::heap()->shenandoah_policy()->consecutive_degenerated_gc_count() <= ShenandoahFullGCThreshold;
}
void ShenandoahHeuristics::adjust_penalty(intx step) {
@@ -220,9 +218,6 @@ void ShenandoahHeuristics::adjust_penalty(intx step) {
}
void ShenandoahHeuristics::record_success_concurrent() {
- _degenerated_cycles_in_a_row = 0;
- _successful_cycles_in_a_row++;
-
_gc_time_history->add(time_since_last_gc());
_gc_times_learned++;
@@ -230,16 +225,10 @@ void ShenandoahHeuristics::record_success_concurrent() {
}
void ShenandoahHeuristics::record_success_degenerated() {
- _degenerated_cycles_in_a_row++;
- _successful_cycles_in_a_row = 0;
-
adjust_penalty(Degenerated_Penalty);
}
void ShenandoahHeuristics::record_success_full() {
- _degenerated_cycles_in_a_row = 0;
- _successful_cycles_in_a_row++;
-
adjust_penalty(Full_Penalty);
}
@@ -254,8 +243,7 @@ void ShenandoahHeuristics::record_requested_gc() {
}
bool ShenandoahHeuristics::can_unload_classes() {
- if (!ClassUnloading) return false;
- return true;
+ return ClassUnloading;
}
bool ShenandoahHeuristics::should_unload_classes() {
diff --git a/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.hpp b/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.hpp
index bc6d2dc40c5..c4bfaed400d 100644
--- a/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.hpp
+++ b/src/hotspot/share/gc/shenandoah/heuristics/shenandoahHeuristics.hpp
@@ -80,9 +80,6 @@ class ShenandoahHeuristics : public CHeapObj {
RegionData* _region_data;
- uint _degenerated_cycles_in_a_row;
- uint _successful_cycles_in_a_row;
-
double _cycle_start;
double _last_cycle_end;
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.cpp b/src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.cpp
index c5b6d787b95..c6352b2749c 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahCodeRoots.cpp
@@ -24,7 +24,6 @@
#include "precompiled.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "gc/shared/classUnloadingContext.hpp"
#include "gc/shenandoah/shenandoahClosures.inline.hpp"
@@ -128,26 +127,17 @@ void ShenandoahCodeRoots::disarm_nmethods() {
class ShenandoahNMethodUnlinkClosure : public NMethodClosure {
private:
bool _unloading_occurred;
- volatile bool _failed;
ShenandoahHeap* const _heap;
BarrierSetNMethod* const _bs;
- void set_failed() {
- Atomic::store(&_failed, true);
- }
-
public:
ShenandoahNMethodUnlinkClosure(bool unloading_occurred) :
_unloading_occurred(unloading_occurred),
- _failed(false),
_heap(ShenandoahHeap::heap()),
_bs(ShenandoahBarrierSet::barrier_set()->barrier_set_nmethod()) {}
virtual void do_nmethod(nmethod* nm) {
assert(_heap->is_concurrent_weak_root_in_progress(), "Only this phase");
- if (failed()) {
- return;
- }
ShenandoahNMethod* nm_data = ShenandoahNMethod::gc_data(nm);
assert(!nm_data->is_unregistered(), "Should not see unregistered entry");
@@ -170,27 +160,19 @@ class ShenandoahNMethodUnlinkClosure : public NMethodClosure {
}
// Clear compiled ICs and exception caches
- if (!nm->unload_nmethod_caches(_unloading_occurred)) {
- set_failed();
- }
- }
-
- bool failed() const {
- return Atomic::load(&_failed);
+ nm->unload_nmethod_caches(_unloading_occurred);
}
};
class ShenandoahUnlinkTask : public WorkerTask {
private:
ShenandoahNMethodUnlinkClosure _cl;
- ICRefillVerifier* _verifier;
ShenandoahConcurrentNMethodIterator _iterator;
public:
- ShenandoahUnlinkTask(bool unloading_occurred, ICRefillVerifier* verifier) :
+ ShenandoahUnlinkTask(bool unloading_occurred) :
WorkerTask("Shenandoah Unlink NMethods"),
_cl(unloading_occurred),
- _verifier(verifier),
_iterator(ShenandoahCodeRoots::table()) {
MutexLocker mu(CodeCache_lock, Mutex::_no_safepoint_check_flag);
_iterator.nmethods_do_begin();
@@ -202,35 +184,15 @@ class ShenandoahUnlinkTask : public WorkerTask {
}
virtual void work(uint worker_id) {
- ICRefillVerifierMark mark(_verifier);
_iterator.nmethods_do(&_cl);
}
-
- bool success() const {
- return !_cl.failed();
- }
};
void ShenandoahCodeRoots::unlink(WorkerThreads* workers, bool unloading_occurred) {
assert(ShenandoahHeap::heap()->unload_classes(), "Only when running concurrent class unloading");
- for (;;) {
- ICRefillVerifier verifier;
-
- {
- ShenandoahUnlinkTask task(unloading_occurred, &verifier);
- workers->run_task(&task);
- if (task.success()) {
- return;
- }
- }
-
- // Cleaning failed because we ran out of transitional IC stubs,
- // so we have to refill and try again. Refilling requires taking
- // a safepoint, so we temporarily leave the suspendible thread set.
- SuspendibleThreadSetLeaver sts;
- InlineCacheBuffer::refill_ic_stubs();
- }
+ ShenandoahUnlinkTask task(unloading_occurred);
+ workers->run_task(&task);
}
void ShenandoahCodeRoots::purge() {
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp b/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp
index 7a034e70936..d14bcc39f45 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.cpp
@@ -31,37 +31,24 @@
ShenandoahCollectorPolicy::ShenandoahCollectorPolicy() :
_success_concurrent_gcs(0),
+ _abbreviated_concurrent_gcs(0),
_success_degenerated_gcs(0),
+ _abbreviated_degenerated_gcs(0),
_success_full_gcs(0),
+ _consecutive_degenerated_gcs(0),
_alloc_failure_degenerated(0),
_alloc_failure_degenerated_upgrade_to_full(0),
- _alloc_failure_full(0),
- _explicit_concurrent(0),
- _explicit_full(0),
- _implicit_concurrent(0),
- _implicit_full(0),
- _cycle_counter(0) {
+ _alloc_failure_full(0) {
- Copy::zero_to_bytes(_degen_points, sizeof(size_t) * ShenandoahGC::_DEGENERATED_LIMIT);
+ Copy::zero_to_bytes(_degen_point_counts, sizeof(size_t) * ShenandoahGC::_DEGENERATED_LIMIT);
+ Copy::zero_to_bytes(_collection_cause_counts, sizeof(size_t) * GCCause::_last_gc_cause);
_tracer = new ShenandoahTracer();
-
-}
-
-void ShenandoahCollectorPolicy::record_explicit_to_concurrent() {
- _explicit_concurrent++;
-}
-
-void ShenandoahCollectorPolicy::record_explicit_to_full() {
- _explicit_full++;
}
-void ShenandoahCollectorPolicy::record_implicit_to_concurrent() {
- _implicit_concurrent++;
-}
-
-void ShenandoahCollectorPolicy::record_implicit_to_full() {
- _implicit_full++;
+void ShenandoahCollectorPolicy::record_collection_cause(GCCause::Cause cause) {
+ assert(cause < GCCause::_last_gc_cause, "Invalid GCCause");
+ _collection_cause_counts[cause]++;
}
void ShenandoahCollectorPolicy::record_alloc_failure_to_full() {
@@ -71,33 +58,35 @@ void ShenandoahCollectorPolicy::record_alloc_failure_to_full() {
void ShenandoahCollectorPolicy::record_alloc_failure_to_degenerated(ShenandoahGC::ShenandoahDegenPoint point) {
assert(point < ShenandoahGC::_DEGENERATED_LIMIT, "sanity");
_alloc_failure_degenerated++;
- _degen_points[point]++;
+ _degen_point_counts[point]++;
}
void ShenandoahCollectorPolicy::record_degenerated_upgrade_to_full() {
+ _consecutive_degenerated_gcs = 0;
_alloc_failure_degenerated_upgrade_to_full++;
}
-void ShenandoahCollectorPolicy::record_success_concurrent() {
+void ShenandoahCollectorPolicy::record_success_concurrent(bool is_abbreviated) {
+ _consecutive_degenerated_gcs = 0;
_success_concurrent_gcs++;
+ if (is_abbreviated) {
+ _abbreviated_concurrent_gcs++;
+ }
}
-void ShenandoahCollectorPolicy::record_success_degenerated() {
+void ShenandoahCollectorPolicy::record_success_degenerated(bool is_abbreviated) {
_success_degenerated_gcs++;
+ _consecutive_degenerated_gcs++;
+ if (is_abbreviated) {
+ _abbreviated_degenerated_gcs++;
+ }
}
void ShenandoahCollectorPolicy::record_success_full() {
+ _consecutive_degenerated_gcs = 0;
_success_full_gcs++;
}
-size_t ShenandoahCollectorPolicy::cycle_counter() const {
- return _cycle_counter;
-}
-
-void ShenandoahCollectorPolicy::record_cycle_start() {
- _cycle_counter++;
-}
-
void ShenandoahCollectorPolicy::record_shutdown() {
_in_shutdown.set();
}
@@ -106,32 +95,102 @@ bool ShenandoahCollectorPolicy::is_at_shutdown() {
return _in_shutdown.is_set();
}
+bool is_explicit_gc(GCCause::Cause cause) {
+ return GCCause::is_user_requested_gc(cause)
+ || GCCause::is_serviceability_requested_gc(cause);
+}
+
+bool is_implicit_gc(GCCause::Cause cause) {
+ return cause != GCCause::_allocation_failure
+ && cause != GCCause::_shenandoah_concurrent_gc
+ && !is_explicit_gc(cause);
+}
+
+#ifdef ASSERT
+bool is_valid_request(GCCause::Cause cause) {
+ return is_explicit_gc(cause)
+ || cause == GCCause::_metadata_GC_clear_soft_refs
+ || cause == GCCause::_codecache_GC_aggressive
+ || cause == GCCause::_codecache_GC_threshold
+ || cause == GCCause::_full_gc_alot
+ || cause == GCCause::_wb_young_gc
+ || cause == GCCause::_wb_full_gc
+ || cause == GCCause::_wb_breakpoint
+ || cause == GCCause::_scavenge_alot;
+}
+#endif
+
+bool ShenandoahCollectorPolicy::should_run_full_gc(GCCause::Cause cause) {
+ return is_explicit_gc(cause) ? !ExplicitGCInvokesConcurrent : !ShenandoahImplicitGCInvokesConcurrent;
+}
+
+bool ShenandoahCollectorPolicy::should_handle_requested_gc(GCCause::Cause cause) {
+ assert(is_valid_request(cause), "only requested GCs here: %s", GCCause::to_string(cause));
+
+ if (DisableExplicitGC) {
+ return !is_explicit_gc(cause);
+ }
+ return true;
+}
+
void ShenandoahCollectorPolicy::print_gc_stats(outputStream* out) const {
out->print_cr("Under allocation pressure, concurrent cycles may cancel, and either continue cycle");
out->print_cr("under stop-the-world pause or result in stop-the-world Full GC. Increase heap size,");
out->print_cr("tune GC heuristics, set more aggressive pacing delay, or lower allocation rate");
- out->print_cr("to avoid Degenerated and Full GC cycles.");
+ out->print_cr("to avoid Degenerated and Full GC cycles. Abbreviated cycles are those which found");
+ out->print_cr("enough regions with no live objects to skip evacuation.");
out->cr();
- out->print_cr(SIZE_FORMAT_W(5) " successful concurrent GCs", _success_concurrent_gcs);
- out->print_cr(" " SIZE_FORMAT_W(5) " invoked explicitly", _explicit_concurrent);
- out->print_cr(" " SIZE_FORMAT_W(5) " invoked implicitly", _implicit_concurrent);
+ size_t completed_gcs = _success_full_gcs + _success_degenerated_gcs + _success_concurrent_gcs;
+ out->print_cr(SIZE_FORMAT_W(5) " Completed GCs", completed_gcs);
+
+ size_t explicit_requests = 0;
+ size_t implicit_requests = 0;
+ for (int c = 0; c < GCCause::_last_gc_cause; c++) {
+ size_t cause_count = _collection_cause_counts[c];
+ if (cause_count > 0) {
+ auto cause = (GCCause::Cause) c;
+ if (is_explicit_gc(cause)) {
+ explicit_requests += cause_count;
+ } else if (is_implicit_gc(cause)) {
+ implicit_requests += cause_count;
+ }
+ const char* desc = GCCause::to_string(cause);
+ out->print_cr(" " SIZE_FORMAT_W(5) " caused by %s (%.2f%%)", cause_count, desc, percent_of(cause_count, completed_gcs));
+ }
+ }
+
+ out->cr();
+ out->print_cr(SIZE_FORMAT_W(5) " Successful Concurrent GCs (%.2f%%)", _success_concurrent_gcs, percent_of(_success_concurrent_gcs, completed_gcs));
+ if (ExplicitGCInvokesConcurrent) {
+ out->print_cr(" " SIZE_FORMAT_W(5) " invoked explicitly (%.2f%%)", explicit_requests, percent_of(explicit_requests, _success_concurrent_gcs));
+ }
+ if (ShenandoahImplicitGCInvokesConcurrent) {
+ out->print_cr(" " SIZE_FORMAT_W(5) " invoked implicitly (%.2f%%)", implicit_requests, percent_of(implicit_requests, _success_concurrent_gcs));
+ }
+ out->print_cr(" " SIZE_FORMAT_W(5) " abbreviated (%.2f%%)", _abbreviated_concurrent_gcs, percent_of(_abbreviated_concurrent_gcs, _success_concurrent_gcs));
out->cr();
- out->print_cr(SIZE_FORMAT_W(5) " Degenerated GCs", _success_degenerated_gcs);
- out->print_cr(" " SIZE_FORMAT_W(5) " caused by allocation failure", _alloc_failure_degenerated);
+ size_t degenerated_gcs = _alloc_failure_degenerated_upgrade_to_full + _success_degenerated_gcs;
+ out->print_cr(SIZE_FORMAT_W(5) " Degenerated GCs (%.2f%%)", degenerated_gcs, percent_of(degenerated_gcs, completed_gcs));
+ out->print_cr(" " SIZE_FORMAT_W(5) " upgraded to Full GC (%.2f%%)", _alloc_failure_degenerated_upgrade_to_full, percent_of(_alloc_failure_degenerated_upgrade_to_full, degenerated_gcs));
+ out->print_cr(" " SIZE_FORMAT_W(5) " caused by allocation failure (%.2f%%)", _alloc_failure_degenerated, percent_of(_alloc_failure_degenerated, degenerated_gcs));
+ out->print_cr(" " SIZE_FORMAT_W(5) " abbreviated (%.2f%%)", _abbreviated_degenerated_gcs, percent_of(_abbreviated_degenerated_gcs, degenerated_gcs));
for (int c = 0; c < ShenandoahGC::_DEGENERATED_LIMIT; c++) {
- if (_degen_points[c] > 0) {
+ if (_degen_point_counts[c] > 0) {
const char* desc = ShenandoahGC::degen_point_to_string((ShenandoahGC::ShenandoahDegenPoint)c);
- out->print_cr(" " SIZE_FORMAT_W(5) " happened at %s", _degen_points[c], desc);
+ out->print_cr(" " SIZE_FORMAT_W(5) " happened at %s", _degen_point_counts[c], desc);
}
}
- out->print_cr(" " SIZE_FORMAT_W(5) " upgraded to Full GC", _alloc_failure_degenerated_upgrade_to_full);
out->cr();
- out->print_cr(SIZE_FORMAT_W(5) " Full GCs", _success_full_gcs + _alloc_failure_degenerated_upgrade_to_full);
- out->print_cr(" " SIZE_FORMAT_W(5) " invoked explicitly", _explicit_full);
- out->print_cr(" " SIZE_FORMAT_W(5) " invoked implicitly", _implicit_full);
- out->print_cr(" " SIZE_FORMAT_W(5) " caused by allocation failure", _alloc_failure_full);
- out->print_cr(" " SIZE_FORMAT_W(5) " upgraded from Degenerated GC", _alloc_failure_degenerated_upgrade_to_full);
+ out->print_cr(SIZE_FORMAT_W(5) " Full GCs (%.2f%%)", _success_full_gcs, percent_of(_success_full_gcs, completed_gcs));
+ if (!ExplicitGCInvokesConcurrent) {
+ out->print_cr(" " SIZE_FORMAT_W(5) " invoked explicitly (%.2f%%)", explicit_requests, percent_of(explicit_requests, _success_concurrent_gcs));
+ }
+ if (!ShenandoahImplicitGCInvokesConcurrent) {
+ out->print_cr(" " SIZE_FORMAT_W(5) " invoked implicitly (%.2f%%)", implicit_requests, percent_of(implicit_requests, _success_concurrent_gcs));
+ }
+ out->print_cr(" " SIZE_FORMAT_W(5) " caused by allocation failure (%.2f%%)", _alloc_failure_full, percent_of(_alloc_failure_full, _success_full_gcs));
+ out->print_cr(" " SIZE_FORMAT_W(5) " upgraded from Degenerated GC (%.2f%%)", _alloc_failure_degenerated_upgrade_to_full, percent_of(_alloc_failure_degenerated_upgrade_to_full, _success_full_gcs));
}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.hpp b/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.hpp
index a6ea6e976ae..638acce1456 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahCollectorPolicy.hpp
@@ -39,54 +39,58 @@ class ShenandoahTracer : public GCTracer, public CHeapObj {
class ShenandoahCollectorPolicy : public CHeapObj {
private:
size_t _success_concurrent_gcs;
+ size_t _abbreviated_concurrent_gcs;
size_t _success_degenerated_gcs;
+ size_t _abbreviated_degenerated_gcs;
// Written by control thread, read by mutators
volatile size_t _success_full_gcs;
+ uint _consecutive_degenerated_gcs;
size_t _alloc_failure_degenerated;
size_t _alloc_failure_degenerated_upgrade_to_full;
size_t _alloc_failure_full;
- size_t _explicit_concurrent;
- size_t _explicit_full;
- size_t _implicit_concurrent;
- size_t _implicit_full;
- size_t _degen_points[ShenandoahGC::_DEGENERATED_LIMIT];
+ size_t _collection_cause_counts[GCCause::_last_gc_cause];
+ size_t _degen_point_counts[ShenandoahGC::_DEGENERATED_LIMIT];
ShenandoahSharedFlag _in_shutdown;
-
ShenandoahTracer* _tracer;
- size_t _cycle_counter;
public:
ShenandoahCollectorPolicy();
- // TODO: This is different from gc_end: that one encompasses one VM operation.
- // These two encompass the entire cycle.
- void record_cycle_start();
-
- void record_success_concurrent();
- void record_success_degenerated();
+ // A collection cycle may be "abbreviated" if Shenandoah finds a sufficient percentage
+ // of regions that contain no live objects (ShenandoahImmediateThreshold). These cycles
+ // end after final mark, skipping the evacuation and reference-updating phases. Such
+ // cycles are very efficient and are worth tracking. Note that both degenerated and
+ // concurrent cycles can be abbreviated.
+ void record_success_concurrent(bool is_abbreviated);
+ void record_success_degenerated(bool is_abbreviated);
void record_success_full();
void record_alloc_failure_to_degenerated(ShenandoahGC::ShenandoahDegenPoint point);
void record_alloc_failure_to_full();
void record_degenerated_upgrade_to_full();
- void record_explicit_to_concurrent();
- void record_explicit_to_full();
- void record_implicit_to_concurrent();
- void record_implicit_to_full();
+ void record_collection_cause(GCCause::Cause cause);
void record_shutdown();
bool is_at_shutdown();
ShenandoahTracer* tracer() {return _tracer;}
- size_t cycle_counter() const;
-
void print_gc_stats(outputStream* out) const;
size_t full_gc_count() const {
return _success_full_gcs + _alloc_failure_degenerated_upgrade_to_full;
}
+
+ // If the heuristics find that the number of consecutive degenerated cycles is above
+ // ShenandoahFullGCThreshold, then they will initiate a Full GC upon an allocation
+ // failure.
+ inline size_t consecutive_degenerated_gc_count() const {
+ return _consecutive_degenerated_gcs;
+ }
+
+ static bool should_run_full_gc(GCCause::Cause cause);
+ static bool should_handle_requested_gc(GCCause::Cause cause);
};
#endif // SHARE_GC_SHENANDOAH_SHENANDOAHCOLLECTORPOLICY_HPP
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp b/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp
index cde8307659f..1e2120d5556 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.cpp
@@ -87,7 +87,8 @@ class ShenandoahBreakpointMarkScope : public StackObj {
ShenandoahConcurrentGC::ShenandoahConcurrentGC() :
_mark(),
- _degen_point(ShenandoahDegenPoint::_degenerated_unset) {
+ _degen_point(ShenandoahDegenPoint::_degenerated_unset),
+ _abbreviated(false) {
}
ShenandoahGC::ShenandoahDegenPoint ShenandoahConcurrentGC::degen_point() const {
@@ -188,6 +189,7 @@ bool ShenandoahConcurrentGC::collect(GCCause::Cause cause) {
entry_cleanup_complete();
} else {
vmop_entry_final_roots();
+ _abbreviated = true;
}
return true;
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.hpp b/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.hpp
index 1010ffe5da7..4cd1a841350 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahConcurrentGC.hpp
@@ -45,12 +45,16 @@ class ShenandoahConcurrentGC : public ShenandoahGC {
private:
ShenandoahConcurrentMark _mark;
ShenandoahDegenPoint _degen_point;
+ bool _abbreviated;
public:
ShenandoahConcurrentGC();
bool collect(GCCause::Cause cause);
ShenandoahDegenPoint degen_point() const;
+ // Return true if this cycle found enough immediate garbage to skip evacuation
+ bool abbreviated() const { return _abbreviated; }
+
// Cancel ongoing concurrent GC
static void cancel();
private:
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp b/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp
index 80c9189e938..bdf1e5b9128 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahControlThread.cpp
@@ -30,58 +30,33 @@
#include "gc/shenandoah/shenandoahDegeneratedGC.hpp"
#include "gc/shenandoah/shenandoahFreeSet.hpp"
#include "gc/shenandoah/shenandoahFullGC.hpp"
-#include "gc/shenandoah/shenandoahPhaseTimings.hpp"
#include "gc/shenandoah/shenandoahHeap.inline.hpp"
-#include "gc/shenandoah/shenandoahMark.inline.hpp"
#include "gc/shenandoah/shenandoahMonitoringSupport.hpp"
-#include "gc/shenandoah/shenandoahOopClosures.inline.hpp"
-#include "gc/shenandoah/shenandoahRootProcessor.inline.hpp"
+#include "gc/shenandoah/shenandoahPacer.inline.hpp"
#include "gc/shenandoah/shenandoahUtils.hpp"
-#include "gc/shenandoah/shenandoahVMOperations.hpp"
-#include "gc/shenandoah/shenandoahWorkerPolicy.hpp"
#include "gc/shenandoah/heuristics/shenandoahHeuristics.hpp"
-#include "memory/iterator.hpp"
#include "memory/metaspaceUtils.hpp"
#include "memory/metaspaceStats.hpp"
-#include "memory/universe.hpp"
+#include "memory/resourceArea.hpp"
#include "runtime/atomic.hpp"
ShenandoahControlThread::ShenandoahControlThread() :
ConcurrentGCThread(),
_alloc_failure_waiters_lock(Mutex::safepoint-2, "ShenandoahAllocFailureGC_lock", true),
_gc_waiters_lock(Mutex::safepoint-2, "ShenandoahRequestedGC_lock", true),
- _periodic_task(this),
_requested_gc_cause(GCCause::_no_cause_specified),
_degen_point(ShenandoahGC::_degenerated_outside_cycle),
_allocs_seen(0) {
set_name("Shenandoah Control Thread");
reset_gc_id();
create_and_start();
- _periodic_task.enroll();
- if (ShenandoahPacing) {
- _periodic_pacer_notify_task.enroll();
- }
-}
-
-ShenandoahControlThread::~ShenandoahControlThread() {
- // This is here so that super is called.
-}
-
-void ShenandoahPeriodicTask::task() {
- _thread->handle_force_counters_update();
- _thread->handle_counters_update();
-}
-
-void ShenandoahPeriodicPacerNotify::task() {
- assert(ShenandoahPacing, "Should not be here otherwise");
- ShenandoahHeap::heap()->pacer()->notify_waiters();
}
void ShenandoahControlThread::run_service() {
ShenandoahHeap* heap = ShenandoahHeap::heap();
- GCMode default_mode = concurrent_normal;
- GCCause::Cause default_cause = GCCause::_shenandoah_concurrent_gc;
+ const GCMode default_mode = concurrent_normal;
+ const GCCause::Cause default_cause = GCCause::_shenandoah_concurrent_gc;
int sleep = ShenandoahControlIntervalMin;
double last_shrink_time = os::elapsedTime();
@@ -91,23 +66,21 @@ void ShenandoahControlThread::run_service() {
// Having a period 10x lower than the delay would mean we hit the
// shrinking with lag of less than 1/10-th of true delay.
// ShenandoahUncommitDelay is in msecs, but shrink_period is in seconds.
- double shrink_period = (double)ShenandoahUncommitDelay / 1000 / 10;
+ const double shrink_period = (double)ShenandoahUncommitDelay / 1000 / 10;
- ShenandoahCollectorPolicy* policy = heap->shenandoah_policy();
- ShenandoahHeuristics* heuristics = heap->heuristics();
+ ShenandoahCollectorPolicy* const policy = heap->shenandoah_policy();
+ ShenandoahHeuristics* const heuristics = heap->heuristics();
while (!in_graceful_shutdown() && !should_terminate()) {
// Figure out if we have pending requests.
- bool alloc_failure_pending = _alloc_failure_gc.is_set();
- bool is_gc_requested = _gc_requested.is_set();
- GCCause::Cause requested_gc_cause = _requested_gc_cause;
- bool explicit_gc_requested = is_gc_requested && is_explicit_gc(requested_gc_cause);
- bool implicit_gc_requested = is_gc_requested && !is_explicit_gc(requested_gc_cause);
+ const bool alloc_failure_pending = _alloc_failure_gc.is_set();
+ const bool is_gc_requested = _gc_requested.is_set();
+ const GCCause::Cause requested_gc_cause = _requested_gc_cause;
- // This control loop iteration have seen this much allocations.
- size_t allocs_seen = Atomic::xchg(&_allocs_seen, (size_t)0, memory_order_relaxed);
+ // This control loop iteration has seen this much allocation.
+ const size_t allocs_seen = Atomic::xchg(&_allocs_seen, (size_t)0, memory_order_relaxed);
// Check if we have seen a new target for soft max heap size.
- bool soft_max_changed = check_soft_max_changed();
+ const bool soft_max_changed = heap->check_soft_max_changed();
// Choose which GC mode to run in. The block below should select a single mode.
GCMode mode = none;
@@ -134,36 +107,17 @@ void ShenandoahControlThread::run_service() {
mode = stw_full;
}
- } else if (explicit_gc_requested) {
+ } else if (is_gc_requested) {
cause = requested_gc_cause;
- log_info(gc)("Trigger: Explicit GC request (%s)", GCCause::to_string(cause));
-
+ log_info(gc)("Trigger: GC request (%s)", GCCause::to_string(cause));
heuristics->record_requested_gc();
- if (ExplicitGCInvokesConcurrent) {
- policy->record_explicit_to_concurrent();
- mode = default_mode;
- // Unload and clean up everything
- heap->set_unload_classes(heuristics->can_unload_classes());
- } else {
- policy->record_explicit_to_full();
+ if (ShenandoahCollectorPolicy::should_run_full_gc(cause)) {
mode = stw_full;
- }
- } else if (implicit_gc_requested) {
- cause = requested_gc_cause;
- log_info(gc)("Trigger: Implicit GC request (%s)", GCCause::to_string(cause));
-
- heuristics->record_requested_gc();
-
- if (ShenandoahImplicitGCInvokesConcurrent) {
- policy->record_implicit_to_concurrent();
+ } else {
mode = default_mode;
-
// Unload and clean up everything
heap->set_unload_classes(heuristics->can_unload_classes());
- } else {
- policy->record_implicit_to_full();
- mode = stw_full;
}
} else {
// Potential normal cycle: ask heuristics if it wants to act
@@ -178,11 +132,11 @@ void ShenandoahControlThread::run_service() {
// Blow all soft references on this cycle, if handling allocation failure,
// either implicit or explicit GC request, or we are requested to do so unconditionally.
- if (alloc_failure_pending || implicit_gc_requested || explicit_gc_requested || ShenandoahAlwaysClearSoftRefs) {
+ if (alloc_failure_pending || is_gc_requested || ShenandoahAlwaysClearSoftRefs) {
heap->soft_ref_policy()->set_should_clear_all_soft_refs(true);
}
- bool gc_requested = (mode != none);
+ const bool gc_requested = (mode != none);
assert (!gc_requested || cause != GCCause::_last_gc_cause, "GC cause should be set");
if (gc_requested) {
@@ -195,7 +149,7 @@ void ShenandoahControlThread::run_service() {
// If GC was requested, we are sampling the counters even without actual triggers
// from allocation machinery. This captures GC phases more accurately.
- set_forced_counters_update(true);
+ heap->set_forced_counters_update(true);
// If GC was requested, we better dump freeset data for performance debugging
{
@@ -218,7 +172,7 @@ void ShenandoahControlThread::run_service() {
}
// If this was the requested GC cycle, notify waiters about it
- if (explicit_gc_requested || implicit_gc_requested) {
+ if (is_gc_requested) {
notify_gc_waiters();
}
@@ -236,16 +190,16 @@ void ShenandoahControlThread::run_service() {
// Notify Universe about new heap usage. This has implications for
// global soft refs policy, and we better report it every time heap
// usage goes down.
- Universe::heap()->update_capacity_and_used_at_gc();
+ heap->update_capacity_and_used_at_gc();
// Signal that we have completed a visit to all live objects.
- Universe::heap()->record_whole_heap_examined_timestamp();
+ heap->record_whole_heap_examined_timestamp();
}
// Disable forced counters update, and update counters one more time
// to capture the state at the end of GC session.
- handle_force_counters_update();
- set_forced_counters_update(false);
+ heap->handle_force_counters_update();
+ heap->set_forced_counters_update(false);
// Retract forceful part of soft refs policy
heap->soft_ref_policy()->set_should_clear_all_soft_refs(false);
@@ -291,14 +245,14 @@ void ShenandoahControlThread::run_service() {
}
}
- double current = os::elapsedTime();
+ const double current = os::elapsedTime();
- if (ShenandoahUncommit && (explicit_gc_requested || soft_max_changed || (current - last_shrink_time > shrink_period))) {
+ if (ShenandoahUncommit && (is_gc_requested || soft_max_changed || (current - last_shrink_time > shrink_period))) {
// Explicit GC tries to uncommit everything down to min capacity.
// Soft max change tries to uncommit everything down to target capacity.
// Periodic uncommit tries to uncommit suitable regions down to min capacity.
- double shrink_before = (explicit_gc_requested || soft_max_changed) ?
+ double shrink_before = (is_gc_requested || soft_max_changed) ?
current :
current - (ShenandoahUncommitDelay / 1000.0);
@@ -306,7 +260,7 @@ void ShenandoahControlThread::run_service() {
heap->soft_max_capacity() :
heap->min_capacity();
- service_uncommit(shrink_before, shrink_until);
+ heap->maybe_uncommit(shrink_before, shrink_until);
heap->phase_timings()->flush_cycle_to_global();
last_shrink_time = current;
}
@@ -314,7 +268,7 @@ void ShenandoahControlThread::run_service() {
// Wait before performing the next action. If allocation happened during this wait,
// we exit sooner, to let heuristics re-evaluate new conditions. If we are at idle,
// back off exponentially.
- if (_heap_changed.try_unset()) {
+ if (heap->has_changed()) {
sleep = ShenandoahControlIntervalMin;
} else if ((current - last_sleep_adjust_time) * 1000 > ShenandoahControlIntervalAdjustPeriod){
sleep = MIN2(ShenandoahControlIntervalMax, MAX2(1, sleep * 2));
@@ -329,25 +283,6 @@ void ShenandoahControlThread::run_service() {
}
}
-bool ShenandoahControlThread::check_soft_max_changed() const {
- ShenandoahHeap* heap = ShenandoahHeap::heap();
- size_t new_soft_max = Atomic::load(&SoftMaxHeapSize);
- size_t old_soft_max = heap->soft_max_capacity();
- if (new_soft_max != old_soft_max) {
- new_soft_max = MAX2(heap->min_capacity(), new_soft_max);
- new_soft_max = MIN2(heap->max_capacity(), new_soft_max);
- if (new_soft_max != old_soft_max) {
- log_info(gc)("Soft Max Heap Size: " SIZE_FORMAT "%s -> " SIZE_FORMAT "%s",
- byte_size_in_proper_unit(old_soft_max), proper_unit_for_byte_size(old_soft_max),
- byte_size_in_proper_unit(new_soft_max), proper_unit_for_byte_size(new_soft_max)
- );
- heap->set_soft_max_capacity(new_soft_max);
- return true;
- }
- }
- return false;
-}
-
void ShenandoahControlThread::service_concurrent_normal_cycle(GCCause::Cause cause) {
// Normal cycle goes via all concurrent phases. If allocation failure (af) happens during
// any of the concurrent phases, it first degrades to Degenerated GC and completes GC there.
@@ -396,7 +331,7 @@ void ShenandoahControlThread::service_concurrent_normal_cycle(GCCause::Cause cau
if (gc.collect(cause)) {
// Cycle is complete
heap->heuristics()->record_success_concurrent();
- heap->shenandoah_policy()->record_success_concurrent();
+ heap->shenandoah_policy()->record_success_concurrent(gc.abbreviated());
} else {
assert(heap->cancelled_gc(), "Must have been cancelled");
check_cancellation_or_degen(gc.degen_point());
@@ -427,10 +362,6 @@ void ShenandoahControlThread::service_stw_full_cycle(GCCause::Cause cause) {
ShenandoahFullGC gc;
gc.collect(cause);
-
- ShenandoahHeap* const heap = ShenandoahHeap::heap();
- heap->heuristics()->record_success_full();
- heap->shenandoah_policy()->record_success_full();
}
void ShenandoahControlThread::service_stw_degenerated_cycle(GCCause::Cause cause, ShenandoahGC::ShenandoahDegenPoint point) {
@@ -441,58 +372,10 @@ void ShenandoahControlThread::service_stw_degenerated_cycle(GCCause::Cause cause
ShenandoahDegenGC gc(point);
gc.collect(cause);
-
- ShenandoahHeap* const heap = ShenandoahHeap::heap();
- heap->heuristics()->record_success_degenerated();
- heap->shenandoah_policy()->record_success_degenerated();
-}
-
-void ShenandoahControlThread::service_uncommit(double shrink_before, size_t shrink_until) {
- ShenandoahHeap* heap = ShenandoahHeap::heap();
-
- // Determine if there is work to do. This avoids taking heap lock if there is
- // no work available, avoids spamming logs with superfluous logging messages,
- // and minimises the amount of work while locks are taken.
-
- if (heap->committed() <= shrink_until) return;
-
- bool has_work = false;
- for (size_t i = 0; i < heap->num_regions(); i++) {
- ShenandoahHeapRegion *r = heap->get_region(i);
- if (r->is_empty_committed() && (r->empty_time() < shrink_before)) {
- has_work = true;
- break;
- }
- }
-
- if (has_work) {
- heap->entry_uncommit(shrink_before, shrink_until);
- }
-}
-
-bool ShenandoahControlThread::is_explicit_gc(GCCause::Cause cause) const {
- return GCCause::is_user_requested_gc(cause) ||
- GCCause::is_serviceability_requested_gc(cause);
}
void ShenandoahControlThread::request_gc(GCCause::Cause cause) {
- assert(GCCause::is_user_requested_gc(cause) ||
- GCCause::is_serviceability_requested_gc(cause) ||
- cause == GCCause::_metadata_GC_clear_soft_refs ||
- cause == GCCause::_codecache_GC_aggressive ||
- cause == GCCause::_codecache_GC_threshold ||
- cause == GCCause::_full_gc_alot ||
- cause == GCCause::_wb_young_gc ||
- cause == GCCause::_wb_full_gc ||
- cause == GCCause::_wb_breakpoint ||
- cause == GCCause::_scavenge_alot,
- "only requested GCs here: %s", GCCause::to_string(cause));
-
- if (is_explicit_gc(cause)) {
- if (!DisableExplicitGC) {
- handle_requested_gc(cause);
- }
- } else {
+ if (ShenandoahCollectorPolicy::should_handle_requested_gc(cause)) {
handle_requested_gc(cause);
}
}
@@ -581,43 +464,11 @@ void ShenandoahControlThread::notify_gc_waiters() {
ml.notify_all();
}
-void ShenandoahControlThread::handle_counters_update() {
- if (_do_counters_update.is_set()) {
- _do_counters_update.unset();
- ShenandoahHeap::heap()->monitoring_support()->update_counters();
- }
-}
-
-void ShenandoahControlThread::handle_force_counters_update() {
- if (_force_counters_update.is_set()) {
- _do_counters_update.unset(); // reset these too, we do update now!
- ShenandoahHeap::heap()->monitoring_support()->update_counters();
- }
-}
-
-void ShenandoahControlThread::notify_heap_changed() {
- // This is called from allocation path, and thus should be fast.
-
- // Update monitoring counters when we took a new region. This amortizes the
- // update costs on slow path.
- if (_do_counters_update.is_unset()) {
- _do_counters_update.set();
- }
- // Notify that something had changed.
- if (_heap_changed.is_unset()) {
- _heap_changed.set();
- }
-}
-
void ShenandoahControlThread::pacing_notify_alloc(size_t words) {
assert(ShenandoahPacing, "should only call when pacing is enabled");
Atomic::add(&_allocs_seen, words, memory_order_relaxed);
}
-void ShenandoahControlThread::set_forced_counters_update(bool value) {
- _force_counters_update.set_cond(value);
-}
-
void ShenandoahControlThread::reset_gc_id() {
Atomic::store(&_gc_id, (size_t)0);
}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp b/src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp
index 544c504b4fa..9da25b1a73c 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahControlThread.hpp
@@ -28,30 +28,8 @@
#include "gc/shared/gcCause.hpp"
#include "gc/shared/concurrentGCThread.hpp"
#include "gc/shenandoah/shenandoahGC.hpp"
-#include "gc/shenandoah/shenandoahHeap.hpp"
#include "gc/shenandoah/shenandoahPadding.hpp"
#include "gc/shenandoah/shenandoahSharedVariables.hpp"
-#include "runtime/task.hpp"
-#include "utilities/ostream.hpp"
-
-// Periodic task is useful for doing asynchronous things that do not require (heap) locks,
-// or synchronization with other parts of collector. These could run even when ShenandoahConcurrentThread
-// is busy driving the GC cycle.
-class ShenandoahPeriodicTask : public PeriodicTask {
-private:
- ShenandoahControlThread* _thread;
-public:
- ShenandoahPeriodicTask(ShenandoahControlThread* thread) :
- PeriodicTask(100), _thread(thread) {}
- virtual void task();
-};
-
-// Periodic task to notify blocked paced waiters.
-class ShenandoahPeriodicPacerNotify : public PeriodicTask {
-public:
- ShenandoahPeriodicPacerNotify() : PeriodicTask(PeriodicTask::min_interval) {}
- virtual void task();
-};
class ShenandoahControlThread: public ConcurrentGCThread {
friend class VMStructs;
@@ -69,8 +47,6 @@ class ShenandoahControlThread: public ConcurrentGCThread {
// to make complete explicit cycle for for demanding customers.
Monitor _alloc_failure_waiters_lock;
Monitor _gc_waiters_lock;
- ShenandoahPeriodicTask _periodic_task;
- ShenandoahPeriodicPacerNotify _periodic_pacer_notify_task;
public:
void run_service();
@@ -80,9 +56,6 @@ class ShenandoahControlThread: public ConcurrentGCThread {
ShenandoahSharedFlag _gc_requested;
ShenandoahSharedFlag _alloc_failure_gc;
ShenandoahSharedFlag _graceful_shutdown;
- ShenandoahSharedFlag _heap_changed;
- ShenandoahSharedFlag _do_counters_update;
- ShenandoahSharedFlag _force_counters_update;
GCCause::Cause _requested_gc_cause;
ShenandoahGC::ShenandoahDegenPoint _degen_point;
@@ -96,7 +69,6 @@ class ShenandoahControlThread: public ConcurrentGCThread {
void service_concurrent_normal_cycle(GCCause::Cause cause);
void service_stw_full_cycle(GCCause::Cause cause);
void service_stw_degenerated_cycle(GCCause::Cause cause, ShenandoahGC::ShenandoahDegenPoint point);
- void service_uncommit(double shrink_before, size_t shrink_until);
bool try_set_alloc_failure_gc();
void notify_alloc_failure_waiters();
@@ -112,14 +84,9 @@ class ShenandoahControlThread: public ConcurrentGCThread {
// Blocks until GC is over.
void handle_requested_gc(GCCause::Cause cause);
- bool is_explicit_gc(GCCause::Cause cause) const;
-
- bool check_soft_max_changed() const;
-
public:
// Constructor
ShenandoahControlThread();
- ~ShenandoahControlThread();
// Handle allocation failure from a mutator allocation.
// Optionally blocks while collector is handling the failure. If the GC
@@ -132,12 +99,6 @@ class ShenandoahControlThread: public ConcurrentGCThread {
void request_gc(GCCause::Cause cause);
- void handle_counters_update();
- void handle_force_counters_update();
- void set_forced_counters_update(bool value);
-
- void notify_heap_changed();
-
void pacing_notify_alloc(size_t words);
void start();
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp b/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp
index e7cf402a527..fb5283bb1d8 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.cpp
@@ -44,7 +44,8 @@
ShenandoahDegenGC::ShenandoahDegenGC(ShenandoahDegenPoint degen_point) :
ShenandoahGC(),
- _degen_point(degen_point) {
+ _degen_point(degen_point),
+ _abbreviated(false) {
}
bool ShenandoahDegenGC::collect(GCCause::Cause cause) {
@@ -193,6 +194,8 @@ void ShenandoahDegenGC::op_degenerated() {
if (heap->has_forwarded_objects()) {
op_init_updaterefs();
assert(!heap->cancelled_gc(), "STW reference update can not OOM");
+ } else {
+ _abbreviated = true;
}
case _degenerated_updaterefs:
@@ -230,6 +233,8 @@ void ShenandoahDegenGC::op_degenerated() {
op_degenerated_futile();
} else {
heap->notify_gc_progress();
+ heap->shenandoah_policy()->record_success_degenerated(_abbreviated);
+ heap->heuristics()->record_success_degenerated();
}
}
@@ -343,17 +348,11 @@ void ShenandoahDegenGC::op_cleanup_complete() {
}
void ShenandoahDegenGC::op_degenerated_fail() {
- log_info(gc)("Cannot finish degeneration, upgrading to Full GC");
- ShenandoahHeap::heap()->shenandoah_policy()->record_degenerated_upgrade_to_full();
-
- ShenandoahFullGC full_gc;
- full_gc.op_full(GCCause::_shenandoah_upgrade_to_full_gc);
+ upgrade_to_full();
}
void ShenandoahDegenGC::op_degenerated_futile() {
- ShenandoahHeap::heap()->shenandoah_policy()->record_degenerated_upgrade_to_full();
- ShenandoahFullGC full_gc;
- full_gc.op_full(GCCause::_shenandoah_upgrade_to_full_gc);
+ upgrade_to_full();
}
const char* ShenandoahDegenGC::degen_event_message(ShenandoahDegenPoint point) const {
@@ -373,3 +372,10 @@ const char* ShenandoahDegenGC::degen_event_message(ShenandoahDegenPoint point) c
return "ERROR";
}
}
+
+void ShenandoahDegenGC::upgrade_to_full() {
+ log_info(gc)("Degenerated GC upgrading to Full GC");
+ ShenandoahHeap::heap()->shenandoah_policy()->record_degenerated_upgrade_to_full();
+ ShenandoahFullGC full_gc;
+ full_gc.op_full(GCCause::_shenandoah_upgrade_to_full_gc);
+}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp b/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp
index 8f6f71d52c2..633c3f1ce73 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahDegeneratedGC.hpp
@@ -33,6 +33,7 @@ class ShenandoahDegenGC : public ShenandoahGC {
friend class VM_ShenandoahDegeneratedGC;
private:
const ShenandoahDegenPoint _degen_point;
+ bool _abbreviated;
public:
ShenandoahDegenGC(ShenandoahDegenPoint degen_point);
@@ -48,6 +49,7 @@ class ShenandoahDegenGC : public ShenandoahGC {
void op_finish_mark();
void op_prepare_evacuation();
void op_cleanup_early();
+
void op_evacuate();
void op_init_updaterefs();
void op_updaterefs();
@@ -58,6 +60,9 @@ class ShenandoahDegenGC : public ShenandoahGC {
void op_degenerated_futile();
void op_degenerated_fail();
+ // Turns this degenerated cycle into a full gc without leaving the safepoint
+ void upgrade_to_full();
+
const char* degen_event_message(ShenandoahDegenPoint point) const;
};
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp b/src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp
index 4cef5378d30..3552669abb6 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp
@@ -31,6 +31,7 @@
#include "gc/shared/tlab_globals.hpp"
#include "gc/shared/workerThread.hpp"
#include "gc/shenandoah/heuristics/shenandoahHeuristics.hpp"
+#include "gc/shenandoah/shenandoahCollectorPolicy.hpp"
#include "gc/shenandoah/shenandoahConcurrentGC.hpp"
#include "gc/shenandoah/shenandoahCollectionSet.hpp"
#include "gc/shenandoah/shenandoahFreeSet.hpp"
@@ -105,15 +106,21 @@ void ShenandoahFullGC::op_full(GCCause::Cause cause) {
// Perform full GC
do_it(cause);
+ ShenandoahHeap* const heap = ShenandoahHeap::heap();
+
metrics.snap_after();
if (metrics.is_good_progress()) {
- ShenandoahHeap::heap()->notify_gc_progress();
+ heap->notify_gc_progress();
} else {
// Nothing to do. Tell the allocation path that we have failed to make
// progress, and it can finally fail.
- ShenandoahHeap::heap()->notify_gc_no_progress();
+ heap->notify_gc_no_progress();
}
+
+ // Regardless if progress was made, we record that we completed a "successful" full GC.
+ heap->heuristics()->record_success_full();
+ heap->shenandoah_policy()->record_success_full();
}
void ShenandoahFullGC::do_it(GCCause::Cause gc_cause) {
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp
index 12eb2aad5e8..de55dde8aca 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp
@@ -432,8 +432,6 @@ jint ShenandoahHeap::initialize() {
if (ShenandoahPacing) {
_pacer = new ShenandoahPacer(this);
_pacer->setup_for_idle();
- } else {
- _pacer = nullptr;
}
_control_thread = new ShenandoahControlThread();
@@ -519,7 +517,6 @@ ShenandoahHeap::ShenandoahHeap(ShenandoahCollectorPolicy* policy) :
_stw_memory_manager("Shenandoah Pauses"),
_cycle_memory_manager("Shenandoah Cycles"),
_gc_timer(new ConcurrentGCTimer()),
- _soft_ref_policy(),
_log_min_obj_alignment_in_bytes(LogMinObjAlignmentInBytes),
_ref_processor(new ShenandoahReferenceProcessor(MAX2(_max_workers, 1U))),
_marking_context(nullptr),
@@ -756,6 +753,33 @@ bool ShenandoahHeap::is_in(const void* p) const {
return p >= heap_base && p < last_region_end;
}
+void ShenandoahHeap::maybe_uncommit(double shrink_before, size_t shrink_until) {
+ assert (ShenandoahUncommit, "should be enabled");
+
+ // Determine if there is work to do. This avoids taking heap lock if there is
+ // no work available, avoids spamming logs with superfluous logging messages,
+ // and minimises the amount of work while locks are taken.
+
+ if (committed() <= shrink_until) return;
+
+ bool has_work = false;
+ for (size_t i = 0; i < num_regions(); i++) {
+ ShenandoahHeapRegion* r = get_region(i);
+ if (r->is_empty_committed() && (r->empty_time() < shrink_before)) {
+ has_work = true;
+ break;
+ }
+ }
+
+ if (has_work) {
+ static const char* msg = "Concurrent uncommit";
+ ShenandoahConcurrentPhase gcPhase(msg, ShenandoahPhaseTimings::conc_uncommit, true /* log_heap_usage */);
+ EventMark em("%s", msg);
+
+ op_uncommit(shrink_before, shrink_until);
+ }
+}
+
void ShenandoahHeap::op_uncommit(double shrink_before, size_t shrink_until) {
assert (ShenandoahUncommit, "should be enabled");
@@ -782,8 +806,43 @@ void ShenandoahHeap::op_uncommit(double shrink_before, size_t shrink_until) {
}
if (count > 0) {
- control_thread()->notify_heap_changed();
+ notify_heap_changed();
+ }
+}
+
+bool ShenandoahHeap::check_soft_max_changed() {
+ size_t new_soft_max = Atomic::load(&SoftMaxHeapSize);
+ size_t old_soft_max = soft_max_capacity();
+ if (new_soft_max != old_soft_max) {
+ new_soft_max = MAX2(min_capacity(), new_soft_max);
+ new_soft_max = MIN2(max_capacity(), new_soft_max);
+ if (new_soft_max != old_soft_max) {
+ log_info(gc)("Soft Max Heap Size: " SIZE_FORMAT "%s -> " SIZE_FORMAT "%s",
+ byte_size_in_proper_unit(old_soft_max), proper_unit_for_byte_size(old_soft_max),
+ byte_size_in_proper_unit(new_soft_max), proper_unit_for_byte_size(new_soft_max)
+ );
+ set_soft_max_capacity(new_soft_max);
+ return true;
+ }
}
+ return false;
+}
+
+void ShenandoahHeap::notify_heap_changed() {
+ // Update monitoring counters when we took a new region. This amortizes the
+ // update costs on slow path.
+ monitoring_support()->notify_heap_changed();
+
+ // This is called from allocation path, and thus should be fast.
+ _heap_changed.try_set();
+}
+
+void ShenandoahHeap::set_forced_counters_update(bool value) {
+ monitoring_support()->set_forced_counters_update(value);
+}
+
+void ShenandoahHeap::handle_force_counters_update() {
+ monitoring_support()->handle_force_counters_update();
}
HeapWord* ShenandoahHeap::allocate_from_gclab_slow(Thread* thread, size_t size) {
@@ -915,7 +974,7 @@ HeapWord* ShenandoahHeap::allocate_memory(ShenandoahAllocRequest& req) {
}
if (in_new_region) {
- control_thread()->notify_heap_changed();
+ notify_heap_changed();
}
if (result != nullptr) {
@@ -2246,14 +2305,6 @@ void ShenandoahHeap::safepoint_synchronize_end() {
SuspendibleThreadSet::desynchronize();
}
-void ShenandoahHeap::entry_uncommit(double shrink_before, size_t shrink_until) {
- static const char *msg = "Concurrent uncommit";
- ShenandoahConcurrentPhase gc_phase(msg, ShenandoahPhaseTimings::conc_uncommit, true /* log_heap_usage */);
- EventMark em("%s", msg);
-
- op_uncommit(shrink_before, shrink_until);
-}
-
void ShenandoahHeap::try_inject_alloc_failure() {
if (ShenandoahAllocFailureALot && !cancelled_gc() && ((os::random() % 1000) > 950)) {
_inject_alloc_failure.set();
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp
index deb0a972167..d3bddc8cc76 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.hpp
@@ -207,6 +207,15 @@ class ShenandoahHeap : public CollectedHeap, public ShenandoahSpaceInfo {
void set_soft_max_capacity(size_t v);
+// ---------- Periodic Tasks
+//
+private:
+ void notify_heap_changed();
+
+public:
+ void set_forced_counters_update(bool value);
+ void handle_force_counters_update();
+
// ---------- Workers handling
//
private:
@@ -286,6 +295,9 @@ class ShenandoahHeap : public CollectedHeap, public ShenandoahSpaceInfo {
private:
bool _gc_state_changed;
ShenandoahSharedBitmap _gc_state;
+
+ // tracks if new regions have been allocated or retired since last check
+ ShenandoahSharedFlag _heap_changed;
ShenandoahSharedFlag _degenerated_gc_in_progress;
ShenandoahSharedFlag _full_gc_in_progress;
ShenandoahSharedFlag _full_gc_move_in_progress;
@@ -307,6 +319,12 @@ class ShenandoahHeap : public CollectedHeap, public ShenandoahSpaceInfo {
// a safepoint and that any changes were propagated to java threads after the safepoint.
bool has_gc_state_changed() const { return _gc_state_changed; }
+ // Returns true if allocations have occurred in new regions or if regions have been
+ // uncommitted since the previous calls. This call will reset the flag to false.
+ bool has_changed() {
+ return _heap_changed.try_unset();
+ }
+
void set_concurrent_mark_in_progress(bool in_progress);
void set_evacuation_in_progress(bool in_progress);
void set_update_refs_in_progress(bool in_progress);
@@ -355,10 +373,14 @@ class ShenandoahHeap : public CollectedHeap, public ShenandoahSpaceInfo {
void cancel_gc(GCCause::Cause cause);
public:
- // Elastic heap support
- void entry_uncommit(double shrink_before, size_t shrink_until);
+ // These will uncommit empty regions if heap::committed > shrink_until
+ // and there exists at least one region which was made empty before shrink_before.
+ void maybe_uncommit(double shrink_before, size_t shrink_until);
void op_uncommit(double shrink_before, size_t shrink_until);
+ // Returns true if the soft maximum heap has been changed using management APIs.
+ bool check_soft_max_changed();
+
private:
// GC support
// Reset bitmap, prepare regions for new GC cycle
@@ -419,15 +441,12 @@ class ShenandoahHeap : public CollectedHeap, public ShenandoahSpaceInfo {
GCMemoryManager _stw_memory_manager;
GCMemoryManager _cycle_memory_manager;
ConcurrentGCTimer* _gc_timer;
- SoftRefPolicy _soft_ref_policy;
-
// For exporting to SA
int _log_min_obj_alignment_in_bytes;
public:
ShenandoahMonitoringSupport* monitoring_support() { return _monitoring_support; }
GCMemoryManager* cycle_memory_manager() { return &_cycle_memory_manager; }
GCMemoryManager* stw_memory_manager() { return &_stw_memory_manager; }
- SoftRefPolicy* soft_ref_policy() override { return &_soft_ref_policy; }
GrowableArray memory_managers() override;
GrowableArray memory_pools() override;
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp b/src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp
index d2487705d5a..c5763608582 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeapRegion.hpp
@@ -30,7 +30,6 @@
#include "gc/shenandoah/shenandoahAllocRequest.hpp"
#include "gc/shenandoah/shenandoahAsserts.hpp"
#include "gc/shenandoah/shenandoahHeap.hpp"
-#include "gc/shenandoah/shenandoahPacer.hpp"
#include "gc/shenandoah/shenandoahPadding.hpp"
#include "utilities/sizes.hpp"
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahMark.hpp b/src/hotspot/share/gc/shenandoah/shenandoahMark.hpp
index 25eed7c66e6..8731aa7a068 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahMark.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahMark.hpp
@@ -27,9 +27,18 @@
#include "gc/shared/stringdedup/stringDedup.hpp"
#include "gc/shared/taskTerminator.hpp"
-#include "gc/shenandoah/shenandoahOopClosures.hpp"
+#include "gc/shenandoah/shenandoahHeap.hpp"
#include "gc/shenandoah/shenandoahTaskqueue.hpp"
+enum StringDedupMode {
+ NO_DEDUP, // Do not do anything for String deduplication
+ ENQUEUE_DEDUP, // Enqueue candidate Strings for deduplication, if meet age threshold
+ ALWAYS_DEDUP // Enqueue Strings for deduplication
+};
+
+class ShenandoahMarkingContext;
+class ShenandoahReferenceProcessor;
+
// Base class for mark
// Mark class does not maintain states. Instead, mark states are
// maintained by task queues, mark bitmap and SATB buffers (concurrent mark)
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp b/src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp
index db0b629f94e..9ca7272815f 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahMark.inline.hpp
@@ -32,6 +32,7 @@
#include "gc/shenandoah/shenandoahBarrierSet.inline.hpp"
#include "gc/shenandoah/shenandoahHeap.inline.hpp"
#include "gc/shenandoah/shenandoahMarkingContext.inline.hpp"
+#include "gc/shenandoah/shenandoahOopClosures.inline.hpp"
#include "gc/shenandoah/shenandoahStringDedup.inline.hpp"
#include "gc/shenandoah/shenandoahTaskqueue.inline.hpp"
#include "gc/shenandoah/shenandoahUtils.hpp"
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.cpp b/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.cpp
index edc66f657fb..ffeab9b6351 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.cpp
@@ -37,7 +37,7 @@ class ShenandoahYoungGenerationCounters : public GenerationCounters {
ShenandoahYoungGenerationCounters() :
GenerationCounters("Young", 0, 0, 0, (size_t)0, (size_t)0) {};
- virtual void update_all() {
+ void update_all() override {
// no update
}
};
@@ -46,19 +46,20 @@ class ShenandoahGenerationCounters : public GenerationCounters {
private:
ShenandoahHeap* _heap;
public:
- ShenandoahGenerationCounters(ShenandoahHeap* heap) :
+ explicit ShenandoahGenerationCounters(ShenandoahHeap* heap) :
GenerationCounters("Heap", 1, 1, heap->initial_capacity(), heap->max_capacity(), heap->capacity()),
_heap(heap)
{};
- virtual void update_all() {
+ void update_all() override {
_current_size->set_value(_heap->capacity());
}
};
ShenandoahMonitoringSupport::ShenandoahMonitoringSupport(ShenandoahHeap* heap) :
_partial_counters(nullptr),
- _full_counters(nullptr)
+ _full_counters(nullptr),
+ _counters_update_task(this)
{
// Collection counters do not fit Shenandoah very well.
// We record partial cycles as "young", and full cycles (including full STW GC) as "old".
@@ -71,6 +72,8 @@ ShenandoahMonitoringSupport::ShenandoahMonitoringSupport(ShenandoahHeap* heap) :
_space_counters = new HSpaceCounters(_heap_counters->name_space(), "Heap", 0, heap->max_capacity(), heap->initial_capacity());
_heap_region_counters = new ShenandoahHeapRegionCounters();
+
+ _counters_update_task.enroll();
}
CollectorCounters* ShenandoahMonitoringSupport::stw_collection_counters() {
@@ -103,3 +106,44 @@ void ShenandoahMonitoringSupport::update_counters() {
MetaspaceCounters::update_performance_counters();
}
}
+
+void ShenandoahMonitoringSupport::notify_heap_changed() {
+ _counters_update_task.notify_heap_changed();
+}
+
+void ShenandoahMonitoringSupport::set_forced_counters_update(bool value) {
+ _counters_update_task.set_forced_counters_update(value);
+}
+
+void ShenandoahMonitoringSupport::handle_force_counters_update() {
+ _counters_update_task.handle_force_counters_update();
+}
+
+void ShenandoahPeriodicCountersUpdateTask::task() {
+ handle_force_counters_update();
+ handle_counters_update();
+}
+
+void ShenandoahPeriodicCountersUpdateTask::handle_counters_update() {
+ if (_do_counters_update.is_set()) {
+ _do_counters_update.unset();
+ _monitoring_support->update_counters();
+ }
+}
+
+void ShenandoahPeriodicCountersUpdateTask::handle_force_counters_update() {
+ if (_force_counters_update.is_set()) {
+ _do_counters_update.unset(); // reset these too, we do update now!
+ _monitoring_support->update_counters();
+ }
+}
+
+void ShenandoahPeriodicCountersUpdateTask::notify_heap_changed() {
+ if (_do_counters_update.is_unset()) {
+ _do_counters_update.set();
+ }
+}
+
+void ShenandoahPeriodicCountersUpdateTask::set_forced_counters_update(bool value) {
+ _force_counters_update.set_cond(value);
+}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.hpp b/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.hpp
index 176cd7978d6..2d90f887707 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahMonitoringSupport.hpp
@@ -25,13 +25,35 @@
#ifndef SHARE_GC_SHENANDOAH_SHENANDOAHMONITORINGSUPPORT_HPP
#define SHARE_GC_SHENANDOAH_SHENANDOAHMONITORINGSUPPORT_HPP
+#include "gc/shenandoah/shenandoahSharedVariables.hpp"
#include "memory/allocation.hpp"
+#include "runtime/task.hpp"
class GenerationCounters;
class HSpaceCounters;
class ShenandoahHeap;
class CollectorCounters;
class ShenandoahHeapRegionCounters;
+class ShenandoahMonitoringSupport;
+
+class ShenandoahPeriodicCountersUpdateTask : public PeriodicTask {
+private:
+ ShenandoahSharedFlag _do_counters_update;
+ ShenandoahSharedFlag _force_counters_update;
+ ShenandoahMonitoringSupport* const _monitoring_support;
+
+public:
+ explicit ShenandoahPeriodicCountersUpdateTask(ShenandoahMonitoringSupport* monitoring_support) :
+ PeriodicTask(100),
+ _monitoring_support(monitoring_support) { }
+
+ void task() override;
+
+ void handle_counters_update();
+ void handle_force_counters_update();
+ void set_forced_counters_update(bool value);
+ void notify_heap_changed();
+};
class ShenandoahMonitoringSupport : public CHeapObj {
private:
@@ -44,14 +66,20 @@ class ShenandoahMonitoringSupport : public CHeapObj {
HSpaceCounters* _space_counters;
ShenandoahHeapRegionCounters* _heap_region_counters;
+ ShenandoahPeriodicCountersUpdateTask _counters_update_task;
public:
- ShenandoahMonitoringSupport(ShenandoahHeap* heap);
- CollectorCounters* stw_collection_counters();
- CollectorCounters* full_stw_collection_counters();
- CollectorCounters* concurrent_collection_counters();
- CollectorCounters* partial_collection_counters();
- void update_counters();
+ explicit ShenandoahMonitoringSupport(ShenandoahHeap* heap);
+ CollectorCounters* stw_collection_counters();
+ CollectorCounters* full_stw_collection_counters();
+ CollectorCounters* concurrent_collection_counters();
+ CollectorCounters* partial_collection_counters();
+
+ void notify_heap_changed();
+ void set_forced_counters_update(bool value);
+ void handle_force_counters_update();
+
+ void update_counters();
};
#endif // SHARE_GC_SHENANDOAH_SHENANDOAHMONITORINGSUPPORT_HPP
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahOopClosures.hpp b/src/hotspot/share/gc/shenandoah/shenandoahOopClosures.hpp
index 11c70f2726a..d8aad6ab6f0 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahOopClosures.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahOopClosures.hpp
@@ -33,12 +33,6 @@
#include "memory/iterator.hpp"
#include "runtime/javaThread.hpp"
-enum StringDedupMode {
- NO_DEDUP, // Do not do anything for String deduplication
- ENQUEUE_DEDUP, // Enqueue candidate Strings for deduplication, if meet age threshold
- ALWAYS_DEDUP // Enqueue Strings for deduplication
-};
-
class ShenandoahMarkRefsSuperClosure : public MetadataVisitingOopIterateClosure {
private:
ShenandoahObjToScanQueue* _queue;
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp b/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp
index 208656c6015..0fc6744c15a 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahPacer.cpp
@@ -338,3 +338,8 @@ void ShenandoahPacer::print_cycle_on(outputStream* out) {
}
out->cr();
}
+
+void ShenandoahPeriodicPacerNotifyTask::task() {
+ assert(ShenandoahPacing, "Should not be here otherwise");
+ _pacer->notify_waiters();
+}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahPacer.hpp b/src/hotspot/share/gc/shenandoah/shenandoahPacer.hpp
index 8dbd9c4d26f..1c2bf00eb56 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahPacer.hpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahPacer.hpp
@@ -29,8 +29,24 @@
#include "gc/shenandoah/shenandoahPadding.hpp"
#include "gc/shenandoah/shenandoahSharedVariables.hpp"
#include "memory/allocation.hpp"
+#include "runtime/task.hpp"
class ShenandoahHeap;
+class ShenandoahPacer;
+
+
+// Periodic task to notify blocked paced waiters.
+class ShenandoahPeriodicPacerNotifyTask : public PeriodicTask {
+private:
+ ShenandoahPacer* const _pacer;
+public:
+ explicit ShenandoahPeriodicPacerNotifyTask(ShenandoahPacer* pacer) :
+ PeriodicTask(PeriodicTask::min_interval),
+ _pacer(pacer) { }
+
+ void task() override;
+};
+
#define PACING_PROGRESS_UNINIT (-1)
#define PACING_PROGRESS_ZERO ( 0)
@@ -48,6 +64,7 @@ class ShenandoahPacer : public CHeapObj {
TruncatedSeq* _progress_history;
Monitor* _wait_monitor;
ShenandoahSharedFlag _need_notify_waiters;
+ ShenandoahPeriodicPacerNotifyTask _notify_waiters_task;
// Set once per phase
volatile intptr_t _epoch;
@@ -64,15 +81,18 @@ class ShenandoahPacer : public CHeapObj {
shenandoah_padding(3);
public:
- ShenandoahPacer(ShenandoahHeap* heap) :
+ explicit ShenandoahPacer(ShenandoahHeap* heap) :
_heap(heap),
_last_time(os::elapsedTime()),
_progress_history(new TruncatedSeq(5)),
_wait_monitor(new Monitor(Mutex::safepoint-1, "ShenandoahWaitMonitor_lock", true)),
+ _notify_waiters_task(this),
_epoch(0),
_tax_rate(1),
_budget(0),
- _progress(PACING_PROGRESS_UNINIT) {}
+ _progress(PACING_PROGRESS_UNINIT) {
+ _notify_waiters_task.enroll();
+ }
void setup_for_idle();
void setup_for_mark();
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp b/src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp
index bb13e9b8e22..1017210e23e 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahUnload.cpp
@@ -106,7 +106,7 @@ class ShenandoahCompiledICProtectionBehaviour : public CompiledICProtectionBehav
}
virtual bool is_safe(CompiledMethod* method) {
- if (SafepointSynchronize::is_at_safepoint()) {
+ if (SafepointSynchronize::is_at_safepoint() || method->is_unloading()) {
return true;
}
diff --git a/src/hotspot/share/gc/shenandoah/shenandoahUtils.cpp b/src/hotspot/share/gc/shenandoah/shenandoahUtils.cpp
index 711d906ec7c..64074a9672c 100644
--- a/src/hotspot/share/gc/shenandoah/shenandoahUtils.cpp
+++ b/src/hotspot/share/gc/shenandoah/shenandoahUtils.cpp
@@ -44,12 +44,12 @@ ShenandoahGCSession::ShenandoahGCSession(GCCause::Cause cause) :
_tracer(_heap->tracer()) {
assert(!ShenandoahGCPhase::is_current_phase_valid(), "No current GC phase");
+ _heap->shenandoah_policy()->record_collection_cause(cause);
_heap->set_gc_cause(cause);
_timer->register_gc_start();
_tracer->report_gc_start(cause, _timer->gc_start());
_heap->trace_heap_before_gc(_tracer);
- _heap->shenandoah_policy()->record_cycle_start();
_heap->heuristics()->record_cycle_start();
_trace_cycle.initialize(_heap->cycle_memory_manager(), cause,
"end of GC cycle",
diff --git a/src/hotspot/share/gc/x/xCollectedHeap.cpp b/src/hotspot/share/gc/x/xCollectedHeap.cpp
index e22006dfaed..cabe7c3c77b 100644
--- a/src/hotspot/share/gc/x/xCollectedHeap.cpp
+++ b/src/hotspot/share/gc/x/xCollectedHeap.cpp
@@ -50,7 +50,6 @@ XCollectedHeap* XCollectedHeap::heap() {
}
XCollectedHeap::XCollectedHeap() :
- _soft_ref_policy(),
_barrier_set(),
_initialize(&_barrier_set),
_heap(),
@@ -95,10 +94,6 @@ void XCollectedHeap::stop() {
gc_threads_do(&cl);
}
-SoftRefPolicy* XCollectedHeap::soft_ref_policy() {
- return &_soft_ref_policy;
-}
-
size_t XCollectedHeap::max_capacity() const {
return _heap.max_capacity();
}
diff --git a/src/hotspot/share/gc/x/xCollectedHeap.hpp b/src/hotspot/share/gc/x/xCollectedHeap.hpp
index 302963ca2c4..940524ab997 100644
--- a/src/hotspot/share/gc/x/xCollectedHeap.hpp
+++ b/src/hotspot/share/gc/x/xCollectedHeap.hpp
@@ -42,7 +42,6 @@ class XCollectedHeap : public CollectedHeap {
friend class ::VMStructs;
private:
- SoftRefPolicy _soft_ref_policy;
XBarrierSet _barrier_set;
XInitialize _initialize;
XHeap _heap;
@@ -65,8 +64,6 @@ class XCollectedHeap : public CollectedHeap {
void initialize_serviceability() override;
void stop() override;
- SoftRefPolicy* soft_ref_policy() override;
-
size_t max_capacity() const override;
size_t capacity() const override;
size_t used() const override;
diff --git a/src/hotspot/share/gc/x/xDirector.cpp b/src/hotspot/share/gc/x/xDirector.cpp
index 05ef3623741..e1c69bd05b7 100644
--- a/src/hotspot/share/gc/x/xDirector.cpp
+++ b/src/hotspot/share/gc/x/xDirector.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2021, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -168,7 +168,7 @@ static double select_gc_workers(double serial_gc_time, double parallelizable_gc_
return gc_workers;
}
-XDriverRequest rule_allocation_rate_dynamic() {
+static XDriverRequest rule_allocation_rate_dynamic() {
if (!XStatCycle::is_time_trustable()) {
// Rule disabled
return GCCause::_no_gc;
diff --git a/src/hotspot/share/gc/x/xNMethod.cpp b/src/hotspot/share/gc/x/xNMethod.cpp
index 613e1908502..5a368a8483b 100644
--- a/src/hotspot/share/gc/x/xNMethod.cpp
+++ b/src/hotspot/share/gc/x/xNMethod.cpp
@@ -24,7 +24,6 @@
#include "precompiled.hpp"
#include "code/relocInfo.hpp"
#include "code/nmethod.hpp"
-#include "code/icBuffer.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetNMethod.hpp"
#include "gc/shared/classUnloadingContext.hpp"
@@ -305,9 +304,7 @@ class XNMethodUnlinkClosure : public NMethodClosure {
}
// Clear compiled ICs and exception caches
- if (!nm->unload_nmethod_caches(_unloading_occurred)) {
- set_failed();
- }
+ nm->unload_nmethod_caches(_unloading_occurred);
}
bool failed() const {
@@ -318,13 +315,11 @@ class XNMethodUnlinkClosure : public NMethodClosure {
class XNMethodUnlinkTask : public XTask {
private:
XNMethodUnlinkClosure _cl;
- ICRefillVerifier* _verifier;
public:
- XNMethodUnlinkTask(bool unloading_occurred, ICRefillVerifier* verifier) :
+ XNMethodUnlinkTask(bool unloading_occurred) :
XTask("XNMethodUnlinkTask"),
- _cl(unloading_occurred),
- _verifier(verifier) {
+ _cl(unloading_occurred) {
XNMethodTable::nmethods_do_begin();
}
@@ -333,33 +328,13 @@ class XNMethodUnlinkTask : public XTask {
}
virtual void work() {
- ICRefillVerifierMark mark(_verifier);
XNMethodTable::nmethods_do(&_cl);
}
-
- bool success() const {
- return !_cl.failed();
- }
};
void XNMethod::unlink(XWorkers* workers, bool unloading_occurred) {
- for (;;) {
- ICRefillVerifier verifier;
-
- {
- XNMethodUnlinkTask task(unloading_occurred, &verifier);
- workers->run(&task);
- if (task.success()) {
- return;
- }
- }
-
- // Cleaning failed because we ran out of transitional IC stubs,
- // so we have to refill and try again. Refilling requires taking
- // a safepoint, so we temporarily leave the suspendible thread set.
- SuspendibleThreadSetLeaver sts;
- InlineCacheBuffer::refill_ic_stubs();
- }
+ XNMethodUnlinkTask task(unloading_occurred);
+ workers->run(&task);
}
void XNMethod::purge() {
diff --git a/src/hotspot/share/gc/x/xNMethodTable.cpp b/src/hotspot/share/gc/x/xNMethodTable.cpp
index f866c581684..52fcba755a7 100644
--- a/src/hotspot/share/gc/x/xNMethodTable.cpp
+++ b/src/hotspot/share/gc/x/xNMethodTable.cpp
@@ -24,7 +24,6 @@
#include "precompiled.hpp"
#include "code/relocInfo.hpp"
#include "code/nmethod.hpp"
-#include "code/icBuffer.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetNMethod.hpp"
#include "gc/x/xGlobals.hpp"
diff --git a/src/hotspot/share/gc/x/xUnload.cpp b/src/hotspot/share/gc/x/xUnload.cpp
index 27d429d3635..230dbf613a4 100644
--- a/src/hotspot/share/gc/x/xUnload.cpp
+++ b/src/hotspot/share/gc/x/xUnload.cpp
@@ -101,7 +101,7 @@ class XCompiledICProtectionBehaviour : public CompiledICProtectionBehaviour {
}
virtual bool is_safe(CompiledMethod* method) {
- if (SafepointSynchronize::is_at_safepoint()) {
+ if (SafepointSynchronize::is_at_safepoint() || method->is_unloading()) {
return true;
}
diff --git a/src/hotspot/share/gc/z/zArguments.cpp b/src/hotspot/share/gc/z/zArguments.cpp
index 192cad86e67..c71e59944f0 100644
--- a/src/hotspot/share/gc/z/zArguments.cpp
+++ b/src/hotspot/share/gc/z/zArguments.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -39,7 +39,6 @@ void ZArguments::initialize_alignments() {
void ZArguments::initialize_heap_flags_and_sizes() {
if (!FLAG_IS_CMDLINE(MaxHeapSize) &&
- !FLAG_IS_CMDLINE(MaxRAMFraction) &&
!FLAG_IS_CMDLINE(MaxRAMPercentage) &&
!FLAG_IS_CMDLINE(SoftMaxHeapSize)) {
// We are really just guessing how much memory the program needs.
diff --git a/src/hotspot/share/gc/z/zCollectedHeap.cpp b/src/hotspot/share/gc/z/zCollectedHeap.cpp
index d53cdad3c5e..12608a32c84 100644
--- a/src/hotspot/share/gc/z/zCollectedHeap.cpp
+++ b/src/hotspot/share/gc/z/zCollectedHeap.cpp
@@ -58,8 +58,7 @@ ZCollectedHeap* ZCollectedHeap::heap() {
}
ZCollectedHeap::ZCollectedHeap()
- : _soft_ref_policy(),
- _barrier_set(),
+ : _barrier_set(),
_initialize(&_barrier_set),
_heap(),
_driver_minor(new ZDriverMinor()),
@@ -106,10 +105,6 @@ void ZCollectedHeap::stop() {
gc_threads_do(&cl);
}
-SoftRefPolicy* ZCollectedHeap::soft_ref_policy() {
- return &_soft_ref_policy;
-}
-
size_t ZCollectedHeap::max_capacity() const {
return _heap.max_capacity();
}
diff --git a/src/hotspot/share/gc/z/zCollectedHeap.hpp b/src/hotspot/share/gc/z/zCollectedHeap.hpp
index 2b83f614009..48ebae5388c 100644
--- a/src/hotspot/share/gc/z/zCollectedHeap.hpp
+++ b/src/hotspot/share/gc/z/zCollectedHeap.hpp
@@ -42,7 +42,6 @@ class ZCollectedHeap : public CollectedHeap {
friend class VMStructs;
private:
- SoftRefPolicy _soft_ref_policy;
ZBarrierSet _barrier_set;
ZInitialize _initialize;
ZHeap _heap;
@@ -66,8 +65,6 @@ class ZCollectedHeap : public CollectedHeap {
void initialize_serviceability() override;
void stop() override;
- SoftRefPolicy* soft_ref_policy() override;
-
size_t max_capacity() const override;
size_t capacity() const override;
size_t used() const override;
diff --git a/src/hotspot/share/gc/z/zDirector.cpp b/src/hotspot/share/gc/z/zDirector.cpp
index 417ac33a6f8..47e24063ead 100644
--- a/src/hotspot/share/gc/z/zDirector.cpp
+++ b/src/hotspot/share/gc/z/zDirector.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -143,11 +143,11 @@ static double select_young_gc_workers(const ZDirectorStats& stats, double serial
return gc_workers;
}
-ZDriverRequest rule_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
- double serial_gc_time_passed,
- double parallel_gc_time_passed,
- bool conservative_alloc_rate,
- size_t capacity) {
+static ZDriverRequest rule_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
+ double serial_gc_time_passed,
+ double parallel_gc_time_passed,
+ bool conservative_alloc_rate,
+ size_t capacity) {
if (!stats._old_stats._cycle._is_time_trustable) {
// Rule disabled
return ZDriverRequest(GCCause::_no_gc, ZYoungGCThreads, 0);
@@ -214,9 +214,9 @@ ZDriverRequest rule_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
return ZDriverRequest(GCCause::_z_allocation_rate, actual_gc_workers, 0);
}
-ZDriverRequest rule_soft_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
- double serial_gc_time_passed,
- double parallel_gc_time_passed) {
+static ZDriverRequest rule_soft_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
+ double serial_gc_time_passed,
+ double parallel_gc_time_passed) {
return rule_minor_allocation_rate_dynamic(stats,
0.0 /* serial_gc_time_passed */,
0.0 /* parallel_gc_time_passed */,
@@ -224,9 +224,9 @@ ZDriverRequest rule_soft_minor_allocation_rate_dynamic(const ZDirectorStats& sta
stats._heap._soft_max_heap_size /* capacity */);
}
-ZDriverRequest rule_semi_hard_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
- double serial_gc_time_passed,
- double parallel_gc_time_passed) {
+static ZDriverRequest rule_semi_hard_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
+ double serial_gc_time_passed,
+ double parallel_gc_time_passed) {
return rule_minor_allocation_rate_dynamic(stats,
0.0 /* serial_gc_time_passed */,
0.0 /* parallel_gc_time_passed */,
@@ -234,9 +234,9 @@ ZDriverRequest rule_semi_hard_minor_allocation_rate_dynamic(const ZDirectorStats
ZHeap::heap()->max_capacity() /* capacity */);
}
-ZDriverRequest rule_hard_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
- double serial_gc_time_passed,
- double parallel_gc_time_passed) {
+static ZDriverRequest rule_hard_minor_allocation_rate_dynamic(const ZDirectorStats& stats,
+ double serial_gc_time_passed,
+ double parallel_gc_time_passed) {
return rule_minor_allocation_rate_dynamic(stats,
0.0 /* serial_gc_time_passed */,
0.0 /* parallel_gc_time_passed */,
diff --git a/src/hotspot/share/gc/z/zNMethod.cpp b/src/hotspot/share/gc/z/zNMethod.cpp
index 71d514face1..a7d23124b68 100644
--- a/src/hotspot/share/gc/z/zNMethod.cpp
+++ b/src/hotspot/share/gc/z/zNMethod.cpp
@@ -25,7 +25,6 @@
#include "code/codeCache.hpp"
#include "code/relocInfo.hpp"
#include "code/nmethod.hpp"
-#include "code/icBuffer.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetNMethod.hpp"
#include "gc/shared/classUnloadingContext.hpp"
@@ -334,23 +333,13 @@ oop ZNMethod::load_oop(oop* p, DecoratorSet decorators) {
class ZNMethodUnlinkClosure : public NMethodClosure {
private:
- bool _unloading_occurred;
- volatile bool _failed;
-
- void set_failed() {
- Atomic::store(&_failed, true);
- }
+ bool _unloading_occurred;
public:
ZNMethodUnlinkClosure(bool unloading_occurred)
- : _unloading_occurred(unloading_occurred),
- _failed(false) {}
+ : _unloading_occurred(unloading_occurred) {}
virtual void do_nmethod(nmethod* nm) {
- if (failed()) {
- return;
- }
-
if (nm->is_unloading()) {
// Unlink from the ZNMethodTable
ZNMethod::unregister_nmethod(nm);
@@ -386,26 +375,18 @@ class ZNMethodUnlinkClosure : public NMethodClosure {
}
// Clear compiled ICs and exception caches
- if (!nm->unload_nmethod_caches(_unloading_occurred)) {
- set_failed();
- }
- }
-
- bool failed() const {
- return Atomic::load(&_failed);
+ nm->unload_nmethod_caches(_unloading_occurred);
}
};
class ZNMethodUnlinkTask : public ZTask {
private:
ZNMethodUnlinkClosure _cl;
- ICRefillVerifier* _verifier;
public:
- ZNMethodUnlinkTask(bool unloading_occurred, ICRefillVerifier* verifier)
+ ZNMethodUnlinkTask(bool unloading_occurred)
: ZTask("ZNMethodUnlinkTask"),
- _cl(unloading_occurred),
- _verifier(verifier) {
+ _cl(unloading_occurred) {
ZNMethodTable::nmethods_do_begin(false /* secondary */);
}
@@ -414,33 +395,13 @@ class ZNMethodUnlinkTask : public ZTask {
}
virtual void work() {
- ICRefillVerifierMark mark(_verifier);
ZNMethodTable::nmethods_do(false /* secondary */, &_cl);
}
-
- bool success() const {
- return !_cl.failed();
- }
};
void ZNMethod::unlink(ZWorkers* workers, bool unloading_occurred) {
- for (;;) {
- ICRefillVerifier verifier;
-
- {
- ZNMethodUnlinkTask task(unloading_occurred, &verifier);
- workers->run(&task);
- if (task.success()) {
- return;
- }
- }
-
- // Cleaning failed because we ran out of transitional IC stubs,
- // so we have to refill and try again. Refilling requires taking
- // a safepoint, so we temporarily leave the suspendible thread set.
- SuspendibleThreadSetLeaver sts_leaver;
- InlineCacheBuffer::refill_ic_stubs();
- }
+ ZNMethodUnlinkTask task(unloading_occurred);
+ workers->run(&task);
}
void ZNMethod::purge() {
diff --git a/src/hotspot/share/gc/z/zNMethodTable.cpp b/src/hotspot/share/gc/z/zNMethodTable.cpp
index f75af7af616..a4b56292c52 100644
--- a/src/hotspot/share/gc/z/zNMethodTable.cpp
+++ b/src/hotspot/share/gc/z/zNMethodTable.cpp
@@ -24,7 +24,6 @@
#include "precompiled.hpp"
#include "code/relocInfo.hpp"
#include "code/nmethod.hpp"
-#include "code/icBuffer.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/barrierSetNMethod.hpp"
#include "gc/z/zHash.inline.hpp"
diff --git a/src/hotspot/share/gc/z/zUnload.cpp b/src/hotspot/share/gc/z/zUnload.cpp
index 76c275df355..3ab4cd5b19f 100644
--- a/src/hotspot/share/gc/z/zUnload.cpp
+++ b/src/hotspot/share/gc/z/zUnload.cpp
@@ -104,7 +104,7 @@ class ZCompiledICProtectionBehaviour : public CompiledICProtectionBehaviour {
}
virtual bool is_safe(CompiledMethod* method) {
- if (SafepointSynchronize::is_at_safepoint()) {
+ if (SafepointSynchronize::is_at_safepoint() || method->is_unloading()) {
return true;
}
diff --git a/src/hotspot/share/include/jvm.h b/src/hotspot/share/include/jvm.h
index 5d6ab27a3a1..b49c15d73b6 100644
--- a/src/hotspot/share/include/jvm.h
+++ b/src/hotspot/share/include/jvm.h
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1152,10 +1152,10 @@ JNIEXPORT void JNICALL
JVM_VirtualThreadUnmount(JNIEnv* env, jobject vthread, jboolean hide);
JNIEXPORT void JNICALL
-JVM_VirtualThreadHideFrames(JNIEnv* env, jobject vthread, jboolean hide);
+JVM_VirtualThreadHideFrames(JNIEnv* env, jclass clazz, jboolean hide);
JNIEXPORT void JNICALL
-JVM_VirtualThreadDisableSuspend(JNIEnv* env, jobject vthread, jboolean enter);
+JVM_VirtualThreadDisableSuspend(JNIEnv* env, jclass clazz, jboolean enter);
/*
* Core reflection support.
diff --git a/src/hotspot/share/interpreter/bytecodeTracer.cpp b/src/hotspot/share/interpreter/bytecodeTracer.cpp
index 1855d557960..624f2b621c1 100644
--- a/src/hotspot/share/interpreter/bytecodeTracer.cpp
+++ b/src/hotspot/share/interpreter/bytecodeTracer.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -192,10 +192,13 @@ void BytecodeTracer::print_method_codes(const methodHandle& method, int from, in
BytecodeStream s(method);
s.set_interval(from, to);
- ttyLocker ttyl; // keep the following output coherent
+ // Keep output to st coherent: collect all lines and print at once.
+ ResourceMark rm;
+ stringStream ss;
while (s.next() >= 0) {
- method_printer.trace(method, s.bcp(), st);
+ method_printer.trace(method, s.bcp(), &ss);
}
+ st->print("%s", ss.as_string());
}
void BytecodePrinter::print_constant(int cp_index, outputStream* st) {
@@ -589,6 +592,10 @@ void BytecodePrinter::print_attributes(int bci, outputStream* st) {
void BytecodePrinter::bytecode_epilog(int bci, outputStream* st) {
MethodData* mdo = method()->method_data();
if (mdo != nullptr) {
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
ProfileData* data = mdo->bci_to_data(bci);
if (data != nullptr) {
st->print(" %d ", mdo->dp_to_di(data->dp()));
diff --git a/src/hotspot/share/interpreter/interpreterRuntime.cpp b/src/hotspot/share/interpreter/interpreterRuntime.cpp
index 70439459d35..ff7c453e01e 100644
--- a/src/hotspot/share/interpreter/interpreterRuntime.cpp
+++ b/src/hotspot/share/interpreter/interpreterRuntime.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -547,7 +547,12 @@ JRT_ENTRY(address, InterpreterRuntime::exception_handler_for_exception(JavaThrea
#if INCLUDE_JVMCI
if (EnableJVMCI && h_method->method_data() != nullptr) {
ResourceMark rm(current);
- ProfileData* pdata = h_method->method_data()->allocate_bci_to_data(current_bci, nullptr);
+ MethodData* mdo = h_method->method_data();
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
+ ProfileData* pdata = mdo->allocate_bci_to_data(current_bci, nullptr);
if (pdata != nullptr && pdata->is_BitData()) {
BitData* bit_data = (BitData*) pdata;
bit_data->set_exception_seen();
diff --git a/src/hotspot/share/interpreter/linkResolver.cpp b/src/hotspot/share/interpreter/linkResolver.cpp
index ed85a07f708..38bef3e1e98 100644
--- a/src/hotspot/share/interpreter/linkResolver.cpp
+++ b/src/hotspot/share/interpreter/linkResolver.cpp
@@ -1085,13 +1085,6 @@ void LinkResolver::resolve_static_call(CallInfo& result,
resolved_method = linktime_resolve_static_method(new_info, CHECK);
}
- if (resolved_method->is_continuation_native_intrinsic()
- && resolved_method->from_interpreted_entry() == nullptr) { // does a load_acquire
- methodHandle mh(THREAD, resolved_method);
- // Generate a compiled form of the enterSpecial intrinsic.
- AdapterHandlerLibrary::create_native_wrapper(mh);
- }
-
// setup result
result.set_static(resolved_klass, methodHandle(THREAD, resolved_method), CHECK);
JFR_ONLY(Jfr::on_resolution(result, CHECK);)
diff --git a/src/hotspot/share/jfr/jni/jfrJniMethod.cpp b/src/hotspot/share/jfr/jni/jfrJniMethod.cpp
index db534b6df3c..eee46cbbdde 100644
--- a/src/hotspot/share/jfr/jni/jfrJniMethod.cpp
+++ b/src/hotspot/share/jfr/jni/jfrJniMethod.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -36,6 +36,7 @@
#include "jfr/recorder/repository/jfrEmergencyDump.hpp"
#include "jfr/recorder/service/jfrEventThrottler.hpp"
#include "jfr/recorder/service/jfrOptionSet.hpp"
+#include "jfr/recorder/service/jfrRecorderService.hpp"
#include "jfr/recorder/stacktrace/jfrStackFilter.hpp"
#include "jfr/recorder/stacktrace/jfrStackFilterRegistry.hpp"
#include "jfr/recorder/stacktrace/jfrStackTraceRepository.hpp"
@@ -348,7 +349,8 @@ JVM_ENTRY_NO_ENV(void, jfr_set_force_instrumentation(JNIEnv* env, jclass jvm, jb
JVM_END
JVM_ENTRY_NO_ENV(void, jfr_emit_old_object_samples(JNIEnv* env, jclass jvm, jlong cutoff_ticks, jboolean emit_all, jboolean skip_bfs))
- LeakProfiler::emit_events(cutoff_ticks, emit_all == JNI_TRUE, skip_bfs == JNI_TRUE);
+ JfrRecorderService service;
+ service.emit_leakprofiler_events(cutoff_ticks, emit_all == JNI_TRUE, skip_bfs == JNI_TRUE);
JVM_END
JVM_ENTRY_NO_ENV(void, jfr_exclude_thread(JNIEnv* env, jclass jvm, jobject t))
diff --git a/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleCheckpoint.cpp b/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleCheckpoint.cpp
index 9f6679c93eb..6a1bbfaf83f 100644
--- a/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleCheckpoint.cpp
+++ b/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleCheckpoint.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -205,6 +205,15 @@ static bool stack_trace_precondition(const ObjectSample* sample) {
return sample->has_stack_trace_id() && !sample->is_dead();
}
+static void add_to_leakp_set(const ObjectSample* sample) {
+ assert(sample != nullptr, "invariant");
+ oop object = sample->object();
+ if (object == nullptr) {
+ return;
+ }
+ JfrTraceId::load_leakp(object->klass());
+}
+
class StackTraceBlobInstaller {
private:
BlobCache _cache;
@@ -219,6 +228,7 @@ class StackTraceBlobInstaller {
}
void sample_do(ObjectSample* sample) {
if (stack_trace_precondition(sample)) {
+ add_to_leakp_set(sample);
install(sample);
}
}
diff --git a/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleWriter.cpp b/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleWriter.cpp
index 2c2d7fe7b3d..87851c71cdb 100644
--- a/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleWriter.cpp
+++ b/src/hotspot/share/jfr/leakprofiler/checkpoint/objectSampleWriter.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2014, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2014, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -199,7 +199,7 @@ static ArrayInfo* array_infos = nullptr;
static FieldTable* field_infos = nullptr;
static RootDescriptionInfo* root_infos = nullptr;
-int __write_sample_info__(JfrCheckpointWriter* writer, const void* si) {
+static int __write_sample_info__(JfrCheckpointWriter* writer, const void* si) {
assert(writer != nullptr, "invariant");
assert(si != nullptr, "invariant");
const OldObjectSampleInfo* const oosi = (const OldObjectSampleInfo*)si;
@@ -224,7 +224,7 @@ static void write_sample_infos(JfrCheckpointWriter& writer) {
}
}
-int __write_reference_info__(JfrCheckpointWriter* writer, const void* ri) {
+static int __write_reference_info__(JfrCheckpointWriter* writer, const void* ri) {
assert(writer != nullptr, "invariant");
assert(ri != nullptr, "invariant");
const ReferenceInfo* const ref_info = (const ReferenceInfo*)ri;
@@ -246,7 +246,7 @@ static void write_reference_infos(JfrCheckpointWriter& writer) {
}
}
-int __write_array_info__(JfrCheckpointWriter* writer, const void* ai) {
+static int __write_array_info__(JfrCheckpointWriter* writer, const void* ai) {
assert(writer != nullptr, "invariant");
assert(ai != nullptr, "invariant");
const ObjectSampleArrayInfo* const osai = (const ObjectSampleArrayInfo*)ai;
@@ -283,7 +283,7 @@ static void write_array_infos(JfrCheckpointWriter& writer) {
}
}
-int __write_field_info__(JfrCheckpointWriter* writer, const void* fi) {
+static int __write_field_info__(JfrCheckpointWriter* writer, const void* fi) {
assert(writer != nullptr, "invariant");
assert(fi != nullptr, "invariant");
const FieldTable::FieldInfoEntry* field_info_entry = (const FieldTable::FieldInfoEntry*)fi;
@@ -340,7 +340,7 @@ static const char* description(const ObjectSampleRootDescriptionInfo* osdi) {
return description.description();
}
-int __write_root_description_info__(JfrCheckpointWriter* writer, const void* di) {
+static int __write_root_description_info__(JfrCheckpointWriter* writer, const void* di) {
assert(writer != nullptr, "invariant");
assert(di != nullptr, "invariant");
const ObjectSampleRootDescriptionInfo* const osdi = (const ObjectSampleRootDescriptionInfo*)di;
@@ -367,11 +367,11 @@ typedef JfrTypeWriterImplHost RootDescriptionWriter;
-int _edge_reference_compare_(uintptr_t lhs, uintptr_t rhs) {
+static int _edge_reference_compare_(uintptr_t lhs, uintptr_t rhs) {
return lhs > rhs ? 1 : (lhs < rhs) ? -1 : 0;
}
-int _root_desc_compare_(const ObjectSampleRootDescriptionInfo*const & lhs, const ObjectSampleRootDescriptionInfo* const& rhs) {
+static int _root_desc_compare_(const ObjectSampleRootDescriptionInfo*const & lhs, const ObjectSampleRootDescriptionInfo* const& rhs) {
const uintptr_t lhs_ref = lhs->_data._root_edge->reference().addr();
const uintptr_t rhs_ref = rhs->_data._root_edge->reference().addr();
return _edge_reference_compare_(lhs_ref, rhs_ref);
diff --git a/src/hotspot/share/jfr/leakprofiler/sampling/objectSample.cpp b/src/hotspot/share/jfr/leakprofiler/sampling/objectSample.cpp
index a732191af3d..21d3338f515 100644
--- a/src/hotspot/share/jfr/leakprofiler/sampling/objectSample.cpp
+++ b/src/hotspot/share/jfr/leakprofiler/sampling/objectSample.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2019, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2019, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -24,6 +24,7 @@
#include "precompiled.hpp"
#include "jfr/leakprofiler/sampling/objectSample.hpp"
#include "jfr/leakprofiler/sampling/objectSampler.hpp"
+#include "jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp"
#include "oops/weakHandle.inline.hpp"
#include "runtime/handles.inline.hpp"
@@ -36,7 +37,7 @@ void ObjectSample::reset() {
}
oop ObjectSample::object() const {
- return _object.resolve();
+ return is_dead() ? nullptr :_object.resolve();
}
bool ObjectSample::is_dead() const {
@@ -48,6 +49,7 @@ const oop* ObjectSample::object_addr() const {
}
void ObjectSample::set_object(oop object) {
+ assert(object != nullptr, "invariant");
assert(_object.is_empty(), "should be empty");
Handle h(Thread::current(), object);
_object = WeakHandle(ObjectSampler::oop_storage(), h);
diff --git a/src/hotspot/share/jfr/metadata/metadata.xml b/src/hotspot/share/jfr/metadata/metadata.xml
index 53fa17555c6..78cd95840b2 100644
--- a/src/hotspot/share/jfr/metadata/metadata.xml
+++ b/src/hotspot/share/jfr/metadata/metadata.xml
@@ -927,6 +927,11 @@
+
+
-
+
diff --git a/src/hotspot/share/jfr/periodic/jfrPeriodic.cpp b/src/hotspot/share/jfr/periodic/jfrPeriodic.cpp
index 0648a5e64db..65b23e3433a 100644
--- a/src/hotspot/share/jfr/periodic/jfrPeriodic.cpp
+++ b/src/hotspot/share/jfr/periodic/jfrPeriodic.cpp
@@ -527,6 +527,13 @@ TRACE_REQUEST_FUNC(PhysicalMemory) {
event.commit();
}
+TRACE_REQUEST_FUNC(SwapSpace) {
+ EventSwapSpace event;
+ event.set_totalSize(os::total_swap_space());
+ event.set_freeSize(os::free_swap_space());
+ event.commit();
+}
+
TRACE_REQUEST_FUNC(JavaThreadStatistics) {
EventJavaThreadStatistics event;
event.set_activeCount(ThreadService::get_live_thread_count());
diff --git a/src/hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp b/src/hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp
index 0ab787a7f3a..5cef25c54cb 100644
--- a/src/hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp
+++ b/src/hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -668,7 +668,7 @@ JfrThreadSampling::~JfrThreadSampling() {
}
#ifdef ASSERT
-void assert_periods(const JfrThreadSampler* sampler, int64_t java_period_millis, int64_t native_period_millis) {
+static void assert_periods(const JfrThreadSampler* sampler, int64_t java_period_millis, int64_t native_period_millis) {
assert(sampler != nullptr, "invariant");
assert(sampler->get_java_period() == java_period_millis, "invariant");
assert(sampler->get_native_period() == native_period_millis, "invariant");
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSet.cpp b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSet.cpp
index 3a86f204e61..3ef7abd8a43 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSet.cpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSet.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -126,7 +126,6 @@ static traceid artifact_tag(const T* ptr, bool leakp) {
SET_LEAKP(ptr);
}
assert(IS_LEAKP(ptr), "invariant");
- return artifact_id(ptr);
}
if (not_used(ptr)) {
SET_TRANSIENT(ptr);
@@ -153,7 +152,7 @@ static inline bool should_do_cld_klass(const Klass* cld_klass, bool leakp) {
static inline bool should_enqueue(const Klass* cld_klass) {
assert(cld_klass != nullptr, "invariant");
- if (previous_epoch()) {
+ if (unloading() || previous_epoch()) {
return false;
}
CldPtr cld = get_cld(cld_klass);
@@ -253,7 +252,11 @@ class ModuleFieldSelector {
static TypePtr select(KlassPtr klass) {
assert(klass != nullptr, "invariant");
PkgPtr pkg = klass->package();
- return pkg != nullptr ? pkg->module() : nullptr;
+ if (pkg == nullptr) {
+ return nullptr;
+ }
+ assert(current_epoch() ? IS_SERIALIZED(pkg) : true, "invariant");
+ return pkg->module();
}
};
@@ -272,7 +275,11 @@ class ModuleCldFieldSelector {
static TypePtr select(KlassPtr klass) {
assert(klass != nullptr, "invariant");
ModPtr mod = ModuleFieldSelector::select(klass);
- return mod != nullptr ? mod->loader_data() : nullptr;
+ if (mod == nullptr) {
+ return nullptr;
+ }
+ assert(current_epoch() ? IS_SERIALIZED(mod) : true, "invariant");
+ return mod->loader_data();
}
};
@@ -283,18 +290,7 @@ class SerializePredicate {
SerializePredicate(bool class_unload) : _class_unload(class_unload) {}
bool operator()(T const& value) {
assert(value != nullptr, "invariant");
- return _class_unload ? _artifacts->should_do_unloading_artifact(value) : IS_NOT_SERIALIZED(value);
- }
-};
-
-template <>
-class SerializePredicate {
- bool _class_unload;
-public:
- SerializePredicate(bool class_unload) : _class_unload(class_unload) {}
- bool operator()(const Klass* klass) {
- assert(klass != nullptr, "invariant");
- return _class_unload ? true : IS_NOT_SERIALIZED(klass);
+ return _class_unload ? true : IS_NOT_SERIALIZED(value);
}
};
@@ -352,13 +348,19 @@ static void do_write_klass(JfrCheckpointWriter* writer, CldPtr cld, KlassPtr kla
writer->write(package_id(klass, leakp));
writer->write(klass->modifier_flags());
writer->write(klass->is_hidden());
- if (!leakp) {
- set_serialized(klass);
+ if (leakp) {
+ assert(IS_LEAKP(klass), "invariant");
+ CLEAR_LEAKP(klass);
+ assert(IS_NOT_LEAKP(klass), "invariant");
+ return;
}
+ assert(used(klass), "invariant");
+ assert(unloading() ? true : IS_NOT_SERIALIZED(klass), "invariant");
+ set_serialized(klass);
}
static inline bool should_write_cld_klass(KlassPtr klass, bool leakp) {
- return klass != nullptr && (leakp || IS_NOT_SERIALIZED(klass));
+ return klass != nullptr && (leakp ? IS_LEAKP(klass) : unloading() ? true : IS_NOT_SERIALIZED(klass));
}
static void write_klass(JfrCheckpointWriter* writer, KlassPtr klass, bool leakp, int& elements) {
@@ -373,10 +375,10 @@ static void write_klass(JfrCheckpointWriter* writer, KlassPtr klass, bool leakp,
write_klass(writer, cld_klass, leakp, elements);
}
}
- KlassPtr mod_klass = get_module_cld_klass(klass, leakp);
- if (should_write_cld_klass(mod_klass, leakp)) {
+ KlassPtr mod_cld_klass = get_module_cld_klass(klass, leakp);
+ if (should_write_cld_klass(mod_cld_klass, leakp)) {
// Write the klass for the module cld.
- write_klass(writer, mod_klass, leakp, elements);
+ write_klass(writer, mod_cld_klass, leakp, elements);
}
}
@@ -398,7 +400,6 @@ int write__klass(JfrCheckpointWriter* writer, const void* k) {
int write__klass__leakp(JfrCheckpointWriter* writer, const void* k) {
assert(k != nullptr, "invariant");
KlassPtr klass = static_cast(k);
- CLEAR_LEAKP(klass);
int elements = 0;
write_klass(writer, klass, true, elements);
return elements;
@@ -978,15 +979,12 @@ class MethodIteratorHost {
MethodUsedPredicate _method_used_predicate;
MethodFlagPredicate _method_flag_predicate;
public:
- MethodIteratorHost(JfrCheckpointWriter* writer,
- bool current_epoch = false,
- bool class_unload = false,
- bool skip_header = false) :
- _method_cb(writer, class_unload, skip_header),
- _klass_cb(writer, class_unload, skip_header),
- _klass_used_predicate(current_epoch),
- _method_used_predicate(current_epoch),
- _method_flag_predicate(current_epoch) {}
+ MethodIteratorHost(JfrCheckpointWriter* writer) :
+ _method_cb(writer, unloading(), false),
+ _klass_cb(writer, unloading(), false),
+ _klass_used_predicate(current_epoch()),
+ _method_used_predicate(current_epoch()),
+ _method_flag_predicate(current_epoch()) {}
bool operator()(KlassPtr klass) {
if (_method_used_predicate(klass)) {
@@ -1037,14 +1035,13 @@ typedef LeakPredicate LeakMethodPredicate;
typedef JfrPredicatedTypeWriterImplHost LeakMethodWriterImplTarget;
typedef JfrTypeWriterHost LeakMethodWriterImpl;
typedef MethodIteratorHost LeakMethodWriter;
-typedef MethodIteratorHost LeakMethodWriter;
typedef CompositeFunctor CompositeMethodWriter;
static void write_methods_with_leakp(MethodWriter& mw) {
assert(_writer != nullptr, "invariant");
assert(_leakp_writer != nullptr, "invariant");
assert(previous_epoch(), "invariant");
- LeakMethodWriter lpmw(_leakp_writer, current_epoch(), unloading());
+ LeakMethodWriter lpmw(_leakp_writer);
CompositeMethodWriter cmw(&lpmw, &mw);
_artifacts->iterate_klasses(cmw);
_artifacts->tally(mw);
@@ -1052,7 +1049,7 @@ static void write_methods_with_leakp(MethodWriter& mw) {
static void write_methods() {
assert(_writer != nullptr, "invariant");
- MethodWriter mw(_writer, current_epoch(), unloading());
+ MethodWriter mw(_writer);
if (_leakp_writer == nullptr) {
_artifacts->iterate_klasses(mw);
_artifacts->tally(mw);
@@ -1065,7 +1062,7 @@ static void write_methods_on_clear() {
assert(_writer != nullptr, "invariant");
assert(_leakp_writer != nullptr, "invariant");
assert(previous_epoch(), "invariant");
- MethodWriter mw(_writer, current_epoch(), unloading());
+ MethodWriter mw(_writer);
write_methods_with_leakp(mw);
}
@@ -1092,14 +1089,14 @@ static int write_symbol(JfrCheckpointWriter* writer, SymbolEntryPtr entry, bool
return 1;
}
-int write__symbol(JfrCheckpointWriter* writer, const void* e) {
+static int write__symbol(JfrCheckpointWriter* writer, const void* e) {
assert(e != nullptr, "invariant");
SymbolEntryPtr entry = static_cast(e);
set_serialized(entry);
return write_symbol(writer, entry, false);
}
-int write__symbol__leakp(JfrCheckpointWriter* writer, const void* e) {
+static int write__symbol__leakp(JfrCheckpointWriter* writer, const void* e) {
assert(e != nullptr, "invariant");
SymbolEntryPtr entry = static_cast(e);
return write_symbol(writer, entry, true);
@@ -1113,14 +1110,14 @@ static int write_string(JfrCheckpointWriter* writer, StringEntryPtr entry, bool
return 1;
}
-int write__string(JfrCheckpointWriter* writer, const void* e) {
+static int write__string(JfrCheckpointWriter* writer, const void* e) {
assert(e != nullptr, "invariant");
StringEntryPtr entry = static_cast(e);
set_serialized(entry);
return write_string(writer, entry, false);
}
-int write__string__leakp(JfrCheckpointWriter* writer, const void* e) {
+static int write__string__leakp(JfrCheckpointWriter* writer, const void* e) {
assert(e != nullptr, "invariant");
StringEntryPtr entry = static_cast(e);
return write_string(writer, entry, true);
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.cpp b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.cpp
index 883821f853c..3db940156a8 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.cpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -54,10 +54,6 @@ void JfrArtifactSet::initialize(bool class_unload) {
_klass_list = new GrowableArray(initial_klass_list_size);
_klass_loader_set = new GrowableArray(initial_klass_loader_set_size);
_klass_loader_leakp_set = new GrowableArray(initial_klass_loader_set_size);
-
- if (class_unload) {
- _unloading_set = new GrowableArray(initial_klass_list_size);
- }
}
void JfrArtifactSet::clear() {
@@ -117,18 +113,9 @@ bool JfrArtifactSet::should_do_cld_klass(const Klass* k, bool leakp) {
return not_in_set(leakp ? _klass_loader_leakp_set : _klass_loader_set, k);
}
-bool JfrArtifactSet::should_do_unloading_artifact(const void* ptr) {
- assert(ptr != nullptr, "invariant");
- assert(_class_unload, "invariant");
- assert(_unloading_set != nullptr, "invariant");
- // The incoming pointers are of all kinds of different types.
- // However, we are only interested in set membership.
- // Treat them uniformly as const Klass* for simplicity and code reuse.
- return not_in_set(_unloading_set, static_cast(ptr));
-}
-
void JfrArtifactSet::register_klass(const Klass* k) {
assert(k != nullptr, "invariant");
+ assert(IS_SERIALIZED(k), "invariant");
assert(_klass_list != nullptr, "invariant");
_klass_list->append(k);
}
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.hpp b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.hpp
index 24424fdef3a..237745b13d9 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.hpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/jfrTypeSetUtils.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -80,6 +80,7 @@ class KlassToFieldEnvelope {
public:
KlassToFieldEnvelope(Letter* letter) : _letter(letter) {}
bool operator()(const Klass* klass) {
+ assert(IS_SERIALIZED(klass), "invariant");
typename FieldSelector::TypePtr t = FieldSelector::select(klass);
return t != nullptr ? (*_letter)(t) : true;
}
@@ -130,7 +131,7 @@ class SymbolPredicate {
class KlassUsedPredicate {
bool _current_epoch;
-public:
+ public:
KlassUsedPredicate(bool current_epoch) : _current_epoch(current_epoch) {}
bool operator()(const Klass* klass) {
return _current_epoch ? USED_THIS_EPOCH(klass) : USED_PREVIOUS_EPOCH(klass);
@@ -201,7 +202,6 @@ class JfrArtifactSet : public JfrCHeapObj {
GrowableArray* _klass_list;
GrowableArray* _klass_loader_set;
GrowableArray* _klass_loader_leakp_set;
- GrowableArray* _unloading_set;
size_t _total_count;
bool _class_unload;
@@ -229,23 +229,8 @@ class JfrArtifactSet : public JfrCHeapObj {
size_t total_count() const;
void register_klass(const Klass* k);
bool should_do_cld_klass(const Klass* k, bool leakp);
- bool should_do_unloading_artifact(const void* ptr);
void increment_checkpoint_id();
- template
- void iterate_klasses(Functor& functor) const {
- for (int i = 0; i < _klass_list->length(); ++i) {
- if (!functor(_klass_list->at(i))) {
- return;
- }
- }
- for (int i = 0; i < _klass_loader_set->length(); ++i) {
- if (!functor(_klass_loader_set->at(i))) {
- return;
- }
- }
- }
-
template
void iterate_symbols(T& functor) {
_symbol_table->iterate_symbols(functor);
@@ -261,6 +246,24 @@ class JfrArtifactSet : public JfrCHeapObj {
_total_count += writer.count();
}
+ template
+ void iterate_klasses(Functor& functor) const {
+ if (iterate(functor, _klass_list)) {
+ iterate(functor, _klass_loader_set);
+ }
+ }
+
+ private:
+ template
+ bool iterate(Functor& functor, GrowableArray* list) const {
+ assert(list != nullptr, "invariant");
+ for (int i = 0; i < list->length(); ++i) {
+ if (!functor(list->at(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
};
class KlassArtifactRegistrator {
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.hpp b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.hpp
index 36b9d491d30..d9211e8cb8c 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.hpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -92,6 +92,7 @@ class JfrTraceId : public AllStatic {
static traceid load(const ModuleEntry* module);
static traceid load(const PackageEntry* package);
static traceid load(const ClassLoaderData* cld);
+ static traceid load_leakp(const Klass* klass); // leak profiler
static traceid load_leakp(const Klass* klass, const Method* method); // leak profiler
static traceid load_leakp_previous_epoch(const Klass* klass, const Method* method); // leak profiler
static traceid load_no_enqueue(const Method* method);
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp
index 4f1ce813ea6..aa99a8383eb 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceId.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -68,6 +68,10 @@ inline traceid JfrTraceId::load(const ClassLoaderData* cld) {
return JfrTraceIdLoadBarrier::load(cld);
}
+inline traceid JfrTraceId::load_leakp(const Klass* klass) {
+ return JfrTraceIdLoadBarrier::load_leakp(klass);
+}
+
inline traceid JfrTraceId::load_leakp(const Klass* klass, const Method* method) {
return JfrTraceIdLoadBarrier::load_leakp(klass, method);
}
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.hpp b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.hpp
index 0795c98940e..ff984d05c19 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.hpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -87,6 +87,7 @@ class JfrTraceIdLoadBarrier : AllStatic {
static traceid load(const Method* method);
static traceid load(const ModuleEntry* module);
static traceid load(const PackageEntry* package);
+ static traceid load_leakp(const Klass* klass); // leak profiler
static traceid load_leakp(const Klass* klass, const Method* method); // leak profiler
static traceid load_leakp_previuos_epoch(const Klass* klass, const Method* method); // leak profiler
static void do_klasses(void f(Klass*), bool previous_epoch = false);
diff --git a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.inline.hpp b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.inline.hpp
index 13853e14a13..ed302335274 100644
--- a/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.inline.hpp
+++ b/src/hotspot/share/jfr/recorder/checkpoint/types/traceid/jfrTraceIdLoadBarrier.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -82,7 +82,7 @@ inline traceid JfrTraceIdLoadBarrier::load(const Klass* klass) {
if (should_tag(klass)) {
load_barrier(klass);
}
- assert(USED_THIS_EPOCH(klass), "invariant");
+ assert(METHOD_AND_CLASS_USED_THIS_EPOCH(klass), "invariant");
return TRACE_ID(klass);
}
@@ -148,6 +148,13 @@ inline traceid JfrTraceIdLoadBarrier::load(const PackageEntry* package) {
return set_used_and_get(package);
}
+inline traceid JfrTraceIdLoadBarrier::load_leakp(const Klass* klass) {
+ assert(klass != nullptr, "invariant");
+ load(klass); // Ensure tagged and enqueued.
+ SET_LEAKP(klass);
+ return TRACE_ID(klass);
+}
+
inline traceid JfrTraceIdLoadBarrier::load_leakp(const Klass* klass, const Method* method) {
assert(klass != nullptr, "invariant");
assert(METHOD_AND_CLASS_USED_THIS_EPOCH(klass), "invariant");
diff --git a/src/hotspot/share/jfr/recorder/service/jfrRecorderService.cpp b/src/hotspot/share/jfr/recorder/service/jfrRecorderService.cpp
index fe674bc9610..a9f0cecb53c 100644
--- a/src/hotspot/share/jfr/recorder/service/jfrRecorderService.cpp
+++ b/src/hotspot/share/jfr/recorder/service/jfrRecorderService.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -691,3 +691,15 @@ void JfrRecorderService::evaluate_chunk_size_for_rotation() {
DEBUG_ONLY(JfrJavaSupport::check_java_thread_in_native(JavaThread::current()));
JfrChunkRotation::evaluate(_chunkwriter);
}
+
+void JfrRecorderService::emit_leakprofiler_events(int64_t cutoff_ticks, bool emit_all, bool skip_bfs) {
+ DEBUG_ONLY(JfrJavaSupport::check_java_thread_in_vm(JavaThread::current()));
+ // Take the rotation lock to exclude flush() during event emits. This is because event emit
+ // also creates a number checkpoint events. Those checkpoint events require a future typeset checkpoint
+ // event for completeness, i.e. to be generated before being flushed to a segment.
+ // The upcoming flush() or rotation() after event emit completes this typeset checkpoint
+ // and serializes all event emit checkpoint events to the same segment.
+ JfrRotationLock lock;
+ LeakProfiler::emit_events(cutoff_ticks, emit_all, skip_bfs);
+}
+
diff --git a/src/hotspot/share/jfr/recorder/service/jfrRecorderService.hpp b/src/hotspot/share/jfr/recorder/service/jfrRecorderService.hpp
index 4ba775d8970..89c0437dd13 100644
--- a/src/hotspot/share/jfr/recorder/service/jfrRecorderService.hpp
+++ b/src/hotspot/share/jfr/recorder/service/jfrRecorderService.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2016, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -70,6 +70,7 @@ class JfrRecorderService : public StackObj {
void flushpoint();
void process_full_buffers();
void evaluate_chunk_size_for_rotation();
+ void emit_leakprofiler_events(int64_t cutoff_ticks, bool emit_all, bool skip_bfs);
static bool is_recording();
};
diff --git a/src/hotspot/share/jfr/support/jfrMethodData.cpp b/src/hotspot/share/jfr/support/jfrMethodData.cpp
index a9213333204..4595f26f4aa 100644
--- a/src/hotspot/share/jfr/support/jfrMethodData.cpp
+++ b/src/hotspot/share/jfr/support/jfrMethodData.cpp
@@ -56,6 +56,10 @@ static bool mark_mdo(Method* method, int bci, JavaThread* jt) {
assert(jt != nullptr, "invariant");
MethodData* const mdo = get_mdo(method, jt);
assert(mdo != nullptr, "invariant");
+
+ // Lock to access ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
// Get the datalayout for the invocation bci.
BitData* const bit_data = get_bit_data(mdo, bci);
// Returns true if this callsite is not yet linked and
diff --git a/src/hotspot/share/jfr/support/jfrTraceIdExtension.hpp b/src/hotspot/share/jfr/support/jfrTraceIdExtension.hpp
index d319e307884..353e5c3f07c 100644
--- a/src/hotspot/share/jfr/support/jfrTraceIdExtension.hpp
+++ b/src/hotspot/share/jfr/support/jfrTraceIdExtension.hpp
@@ -1,5 +1,5 @@
/*
-* Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+* Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -95,6 +95,10 @@ class JfrTraceFlag {
} \
uint8_t* trace_meta_addr() const { \
return _trace_flags.meta_addr(); \
+ } \
+ void copy_trace_flags(uint8_t src_flags) const { \
+ uint8_t flags = *_trace_flags.flags_addr(); \
+ _trace_flags.set_flags(flags | src_flags); \
}
#endif // SHARE_JFR_SUPPORT_JFRTRACEIDEXTENSION_HPP
diff --git a/src/hotspot/share/jvmci/jvmciCodeInstaller.cpp b/src/hotspot/share/jvmci/jvmciCodeInstaller.cpp
index 51281b0f556..a5a20a1310e 100644
--- a/src/hotspot/share/jvmci/jvmciCodeInstaller.cpp
+++ b/src/hotspot/share/jvmci/jvmciCodeInstaller.cpp
@@ -896,8 +896,8 @@ int CodeInstaller::estimate_stubs_size(HotSpotCompiledCodeStream* stream, JVMCI_
// Estimate the number of static call stubs that might be emitted.
u2 static_call_stubs = stream->read_u2("numStaticCallStubs");
u2 trampoline_stubs = stream->read_u2("numTrampolineStubs");
- int size = static_call_stubs * CompiledStaticCall::to_interp_stub_size();
- size += trampoline_stubs * CompiledStaticCall::to_trampoline_stub_size();
+ int size = static_call_stubs * CompiledDirectCall::to_interp_stub_size();
+ size += trampoline_stubs * CompiledDirectCall::to_trampoline_stub_size();
return size;
}
@@ -1243,7 +1243,7 @@ void CodeInstaller::site_Call(CodeBuffer& buffer, u1 tag, jint pc_offset, HotSpo
CodeInstaller::pd_relocate_JavaMethod(buffer, method, pc_offset, JVMCI_CHECK);
if (_next_call_type == INVOKESTATIC || _next_call_type == INVOKESPECIAL) {
// Need a static call stub for transitions from compiled to interpreted.
- if (CompiledStaticCall::emit_to_interp_stub(buffer, _instructions->start() + pc_offset) == nullptr) {
+ if (CompiledDirectCall::emit_to_interp_stub(buffer, _instructions->start() + pc_offset) == nullptr) {
JVMCI_ERROR("could not emit to_interp stub - code cache is full");
}
}
diff --git a/src/hotspot/share/jvmci/jvmciCompilerToVM.cpp b/src/hotspot/share/jvmci/jvmciCompilerToVM.cpp
index 1a1e0ba7fd6..ca375073313 100644
--- a/src/hotspot/share/jvmci/jvmciCompilerToVM.cpp
+++ b/src/hotspot/share/jvmci/jvmciCompilerToVM.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -61,7 +61,7 @@
#include "runtime/globals_extension.hpp"
#include "runtime/interfaceSupport.inline.hpp"
#include "runtime/jniHandles.inline.hpp"
-#include "runtime/reflectionUtils.hpp"
+#include "runtime/reflection.hpp"
#include "runtime/stackFrameStream.inline.hpp"
#include "runtime/timerTrace.hpp"
#include "runtime/vframe.inline.hpp"
@@ -179,20 +179,11 @@ Handle JavaArgumentUnboxer::next_arg(BasicType expectedType) {
CompilerThreadCanCallJava ccj(thread, __is_hotspot); \
JVMCIENV_FROM_JNI(JVMCI::compilation_tick(thread), env); \
-static JavaThread* get_current_thread(bool allow_null=true) {
- Thread* thread = Thread::current_or_null_safe();
- if (thread == nullptr) {
- assert(allow_null, "npe");
- return nullptr;
- }
- return JavaThread::cast(thread);
-}
-
// Entry to native method implementation that transitions
// current thread to '_thread_in_vm'.
#define C2V_VMENTRY(result_type, name, signature) \
JNIEXPORT result_type JNICALL c2v_ ## name signature { \
- JavaThread* thread = get_current_thread(); \
+ JavaThread* thread = JavaThread::current_or_null(); \
if (thread == nullptr) { \
env->ThrowNew(JNIJVMCI::InternalError::clazz(), \
err_msg("Cannot call into HotSpot from JVMCI shared library without attaching current thread")); \
@@ -203,7 +194,7 @@ static JavaThread* get_current_thread(bool allow_null=true) {
#define C2V_VMENTRY_(result_type, name, signature, result) \
JNIEXPORT result_type JNICALL c2v_ ## name signature { \
- JavaThread* thread = get_current_thread(); \
+ JavaThread* thread = JavaThread::current_or_null(); \
if (thread == nullptr) { \
env->ThrowNew(JNIJVMCI::InternalError::clazz(), \
err_msg("Cannot call into HotSpot from JVMCI shared library without attaching current thread")); \
@@ -219,7 +210,7 @@ static JavaThread* get_current_thread(bool allow_null=true) {
// current thread to '_thread_in_vm'.
#define C2V_VMENTRY_PREFIX(result_type, name, signature) \
JNIEXPORT result_type JNICALL c2v_ ## name signature { \
- JavaThread* thread = get_current_thread();
+ JavaThread* thread = JavaThread::current_or_null();
#define C2V_END }
@@ -1389,7 +1380,7 @@ C2V_END
/*
* Used by matches() to convert a ResolvedJavaMethod[] to an array of Method*.
*/
-GrowableArray* init_resolved_methods(jobjectArray methods, JVMCIEnv* JVMCIENV) {
+static GrowableArray* init_resolved_methods(jobjectArray methods, JVMCIEnv* JVMCIENV) {
objArrayOop methods_oop = (objArrayOop) JNIHandles::resolve(methods);
GrowableArray* resolved_methods = new GrowableArray(methods_oop->length());
for (int i = 0; i < methods_oop->length(); i++) {
@@ -1408,7 +1399,7 @@ GrowableArray* init_resolved_methods(jobjectArray methods, JVMCIEnv* JV
* The ResolvedJavaMethod[] array is converted to a Method* array that is then cached in the resolved_methods_ref in/out parameter.
* In case of a match, the matching ResolvedJavaMethod is returned in matched_jvmci_method_ref.
*/
-bool matches(jobjectArray methods, Method* method, GrowableArray** resolved_methods_ref, Handle* matched_jvmci_method_ref, Thread* THREAD, JVMCIEnv* JVMCIENV) {
+static bool matches(jobjectArray methods, Method* method, GrowableArray** resolved_methods_ref, Handle* matched_jvmci_method_ref, Thread* THREAD, JVMCIEnv* JVMCIENV) {
GrowableArray* resolved_methods = *resolved_methods_ref;
if (resolved_methods == nullptr) {
resolved_methods = init_resolved_methods(methods, JVMCIENV);
@@ -1429,7 +1420,7 @@ bool matches(jobjectArray methods, Method* method, GrowableArray** reso
/*
* Resolves an interface call to a concrete method handle.
*/
-methodHandle resolve_interface_call(Klass* spec_klass, Symbol* name, Symbol* signature, JavaCallArguments* args, TRAPS) {
+static methodHandle resolve_interface_call(Klass* spec_klass, Symbol* name, Symbol* signature, JavaCallArguments* args, TRAPS) {
CallInfo callinfo;
Handle receiver = args->receiver();
Klass* recvrKlass = receiver.is_null() ? (Klass*)nullptr : receiver->klass();
@@ -1444,7 +1435,7 @@ methodHandle resolve_interface_call(Klass* spec_klass, Symbol* name, Symbol* sig
/*
* Used by c2v_iterateFrames to make a new vframeStream at the given compiled frame id (stack pointer) and vframe id.
*/
-void resync_vframestream_to_compiled_frame(vframeStream& vfst, intptr_t* stack_pointer, int vframe_id, JavaThread* thread, TRAPS) {
+static void resync_vframestream_to_compiled_frame(vframeStream& vfst, intptr_t* stack_pointer, int vframe_id, JavaThread* thread, TRAPS) {
vfst = vframeStream(thread);
while (vfst.frame_id() != stack_pointer && !vfst.at_end()) {
vfst.next();
@@ -1467,7 +1458,7 @@ void resync_vframestream_to_compiled_frame(vframeStream& vfst, intptr_t* stack_p
/*
* Used by c2v_iterateFrames. Returns an array of any unallocated scope objects or null if none.
*/
-GrowableArray* get_unallocated_objects_or_null(GrowableArray* scope_objects) {
+static GrowableArray* get_unallocated_objects_or_null(GrowableArray* scope_objects) {
GrowableArray* unallocated = nullptr;
for (int i = 0; i < scope_objects->length(); i++) {
ObjectValue* sv = (ObjectValue*) scope_objects->at(i);
@@ -1889,7 +1880,10 @@ C2V_END
C2V_VMENTRY_0(jint, methodDataExceptionSeen, (JNIEnv* env, jobject, jlong method_data_pointer, jint bci))
MethodData* mdo = (MethodData*) method_data_pointer;
- MutexLocker mu(mdo->extra_data_lock());
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker mu(mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
DataLayout* data = mdo->extra_data_base();
DataLayout* end = mdo->args_data_limit();
for (;; data = mdo->next_extra(data)) {
@@ -2613,7 +2607,7 @@ static void attachSharedLibraryThread(JNIEnv* env, jbyteArray name, jboolean as_
if (res != JNI_OK) {
JNI_THROW("attachSharedLibraryThread", InternalError, err_msg("Trying to attach thread returned %d", res));
}
- JavaThread* thread = get_current_thread(false);
+ JavaThread* thread = JavaThread::thread_from_jni_environment(hotspotEnv);
const char* attach_error;
{
// Transition to VM
@@ -3071,7 +3065,7 @@ C2V_VMENTRY_0(jint, registerCompilerPhase, (JNIEnv* env, jobject, jstring jphase
C2V_END
C2V_VMENTRY(void, notifyCompilerPhaseEvent, (JNIEnv* env, jobject, jlong startTime, jint phase, jint compileId, jint level))
- EventCompilerPhase event;
+ EventCompilerPhase event(UNTIMED);
if (event.should_commit()) {
CompilerEvent::PhaseEvent::post(event, startTime, phase, compileId, level);
}
diff --git a/src/hotspot/share/jvmci/jvmciCompilerToVMInit.cpp b/src/hotspot/share/jvmci/jvmciCompilerToVMInit.cpp
index 89fa78b10f3..8822aed0cbe 100644
--- a/src/hotspot/share/jvmci/jvmciCompilerToVMInit.cpp
+++ b/src/hotspot/share/jvmci/jvmciCompilerToVMInit.cpp
@@ -310,7 +310,6 @@ JVMCIObjectArray CompilerToVM::initialize_intrinsics(JVMCI_TRAPS) {
do_bool_flag(RestrictContended) \
do_intx_flag(StackReservedPages) \
do_intx_flag(StackShadowPages) \
- do_bool_flag(TLABStats) \
do_uintx_flag(TLABWasteIncrement) \
do_intx_flag(TypeProfileWidth) \
do_bool_flag(UseAESIntrinsics) \
diff --git a/src/hotspot/share/jvmci/jvmciRuntime.cpp b/src/hotspot/share/jvmci/jvmciRuntime.cpp
index 5e9999f2733..3a19745fe09 100644
--- a/src/hotspot/share/jvmci/jvmciRuntime.cpp
+++ b/src/hotspot/share/jvmci/jvmciRuntime.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -54,7 +54,7 @@
#include "runtime/java.hpp"
#include "runtime/jniHandles.inline.hpp"
#include "runtime/mutex.hpp"
-#include "runtime/reflectionUtils.hpp"
+#include "runtime/reflection.hpp"
#include "runtime/sharedRuntime.hpp"
#include "runtime/synchronizer.hpp"
#if INCLUDE_G1GC
diff --git a/src/hotspot/share/jvmci/vmStructs_jvmci.cpp b/src/hotspot/share/jvmci/vmStructs_jvmci.cpp
index 2ad442db934..f47c4cdd940 100644
--- a/src/hotspot/share/jvmci/vmStructs_jvmci.cpp
+++ b/src/hotspot/share/jvmci/vmStructs_jvmci.cpp
@@ -24,6 +24,7 @@
#include "precompiled.hpp"
#include "code/codeCache.hpp"
+#include "code/compiledIC.hpp"
#include "compiler/compileBroker.hpp"
#include "gc/shared/collectedHeap.hpp"
#include "jvmci/jvmciCodeInstaller.hpp"
@@ -144,6 +145,11 @@
\
nonstatic_field(CompileTask, _num_inlined_bytecodes, int) \
\
+ volatile_nonstatic_field(CompiledICData, _speculated_method, Method*) \
+ volatile_nonstatic_field(CompiledICData, _speculated_klass, uintptr_t) \
+ nonstatic_field(CompiledICData, _itable_defc_klass, Klass*) \
+ nonstatic_field(CompiledICData, _itable_refc_klass, Klass*) \
+ \
nonstatic_field(ConstantPool, _tags, Array*) \
nonstatic_field(ConstantPool, _pool_holder, InstanceKlass*) \
nonstatic_field(ConstantPool, _length, int) \
@@ -430,6 +436,7 @@
declare_toplevel_type(oopDesc) \
declare_type(arrayOopDesc, oopDesc) \
\
+ declare_toplevel_type(CompiledICData) \
declare_toplevel_type(MetaspaceObj) \
declare_type(Metadata, MetaspaceObj) \
declare_type(Klass, Metadata) \
diff --git a/src/hotspot/share/memory/iterator.hpp b/src/hotspot/share/memory/iterator.hpp
index aa092e8536d..06c1bd7415b 100644
--- a/src/hotspot/share/memory/iterator.hpp
+++ b/src/hotspot/share/memory/iterator.hpp
@@ -233,16 +233,6 @@ class ObjectToOopClosure: public ObjectClosure {
ObjectToOopClosure(OopIterateClosure* cl) : _cl(cl) {}
};
-// SpaceClosure is used for iterating over spaces
-
-class Space;
-
-class SpaceClosure : public StackObj {
- public:
- // Called for each space
- virtual void do_space(Space* s) = 0;
-};
-
// CodeBlobClosure is used for iterating through code blobs
// in the code cache or on thread stacks
diff --git a/src/hotspot/share/memory/metaspace/virtualSpaceNode.cpp b/src/hotspot/share/memory/metaspace/virtualSpaceNode.cpp
index dc5f7a6cdf1..84f9c9ecc43 100644
--- a/src/hotspot/share/memory/metaspace/virtualSpaceNode.cpp
+++ b/src/hotspot/share/memory/metaspace/virtualSpaceNode.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2018, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2018, 2021 SAP SE. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -56,12 +56,12 @@ namespace metaspace {
#define LOGFMT_ARGS p2i(this), p2i(_base)
#ifdef ASSERT
-void check_pointer_is_aligned_to_commit_granule(const MetaWord* p) {
+static void check_pointer_is_aligned_to_commit_granule(const MetaWord* p) {
assert(is_aligned(p, Settings::commit_granule_bytes()),
"Pointer not aligned to commit granule size: " PTR_FORMAT ".",
p2i(p));
}
-void check_word_size_is_aligned_to_commit_granule(size_t word_size) {
+static void check_word_size_is_aligned_to_commit_granule(size_t word_size) {
assert(is_aligned(word_size, Settings::commit_granule_words()),
"Not aligned to commit granule size: " SIZE_FORMAT ".", word_size);
}
diff --git a/src/hotspot/share/memory/universe.cpp b/src/hotspot/share/memory/universe.cpp
index dfeb6b11f6f..2a3d532725f 100644
--- a/src/hotspot/share/memory/universe.cpp
+++ b/src/hotspot/share/memory/universe.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -298,7 +298,7 @@ void Universe::check_alignment(uintx size, uintx alignment, const char* name) {
}
}
-void initialize_basic_type_klass(Klass* k, TRAPS) {
+static void initialize_basic_type_klass(Klass* k, TRAPS) {
Klass* ok = vmClasses::Object_klass();
#if INCLUDE_CDS
if (UseSharedSpaces) {
@@ -924,11 +924,11 @@ void universe_oopstorage_init() {
Universe::oopstorage_init();
}
-void initialize_known_method(LatestMethodCache* method_cache,
- InstanceKlass* ik,
- const char* method,
- Symbol* signature,
- bool is_static, TRAPS)
+static void initialize_known_method(LatestMethodCache* method_cache,
+ InstanceKlass* ik,
+ const char* method,
+ Symbol* signature,
+ bool is_static, TRAPS)
{
TempNewSymbol name = SymbolTable::new_symbol(method);
Method* m = nullptr;
diff --git a/src/hotspot/share/oops/compiledICHolder.cpp b/src/hotspot/share/oops/compiledICHolder.cpp
deleted file mode 100644
index 8bfa55bcce7..00000000000
--- a/src/hotspot/share/oops/compiledICHolder.cpp
+++ /dev/null
@@ -1,75 +0,0 @@
-/*
- * Copyright (c) 1998, 2022, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#include "precompiled.hpp"
-#include "oops/compiledICHolder.hpp"
-#include "runtime/atomic.hpp"
-
-#ifdef ASSERT
-volatile int CompiledICHolder::_live_count;
-volatile int CompiledICHolder::_live_not_claimed_count;
-#endif
-
-CompiledICHolder::CompiledICHolder(Metadata* metadata, Klass* klass, bool is_method)
- : _holder_metadata(metadata), _holder_klass(klass), _next(nullptr), _is_metadata_method(is_method) {
-#ifdef ASSERT
- Atomic::inc(&_live_count);
- Atomic::inc(&_live_not_claimed_count);
-#endif // ASSERT
-}
-
-#ifdef ASSERT
-CompiledICHolder::~CompiledICHolder() {
- assert(_live_count > 0, "underflow");
- Atomic::dec(&_live_count);
-}
-#endif // ASSERT
-
-// Printing
-
-void CompiledICHolder::print_on(outputStream* st) const {
- st->print("%s", internal_name());
- st->print(" - metadata: "); holder_metadata()->print_value_on(st); st->cr();
- st->print(" - klass: "); holder_klass()->print_value_on(st); st->cr();
-}
-
-void CompiledICHolder::print_value_on(outputStream* st) const {
- st->print("%s", internal_name());
-}
-
-
-// Verification
-
-void CompiledICHolder::verify_on(outputStream* st) {
- guarantee(holder_metadata()->is_method() || holder_metadata()->is_klass(), "should be method or klass");
- guarantee(holder_klass()->is_klass(), "should be klass");
-}
-
-#ifdef ASSERT
-
-void CompiledICHolder::claim() {
- Atomic::dec(&_live_not_claimed_count);
-}
-
-#endif // ASSERT
diff --git a/src/hotspot/share/oops/compiledICHolder.hpp b/src/hotspot/share/oops/compiledICHolder.hpp
deleted file mode 100644
index 4509c8f578b..00000000000
--- a/src/hotspot/share/oops/compiledICHolder.hpp
+++ /dev/null
@@ -1,92 +0,0 @@
-/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
- * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
- *
- * This code is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 only, as
- * published by the Free Software Foundation.
- *
- * This code is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
- * version 2 for more details (a copy is included in the LICENSE file that
- * accompanied this code).
- *
- * You should have received a copy of the GNU General Public License version
- * 2 along with this work; if not, write to the Free Software Foundation,
- * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
- *
- * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
- * or visit www.oracle.com if you need additional information or have any
- * questions.
- *
- */
-
-#ifndef SHARE_OOPS_COMPILEDICHOLDER_HPP
-#define SHARE_OOPS_COMPILEDICHOLDER_HPP
-
-#include "oops/oop.hpp"
-#include "utilities/macros.hpp"
-#include "oops/klass.hpp"
-#include "oops/method.hpp"
-
-// A CompiledICHolder* is a helper object for the inline cache implementation.
-// It holds:
-// (1) (method+klass pair) when converting from compiled to an interpreted call
-// (2) (klass+klass pair) when calling itable stub from megamorphic compiled call
-//
-// These are always allocated in the C heap and are freed during a
-// safepoint by the ICBuffer logic. It's unsafe to free them earlier
-// since they might be in use.
-//
-
-
-class CompiledICHolder : public CHeapObj {
- friend class VMStructs;
- private:
-#ifdef ASSERT
- static volatile int _live_count; // allocated
- static volatile int _live_not_claimed_count; // allocated but not yet in use so not
- // reachable by iterating over nmethods
-#endif
-
- Metadata* _holder_metadata;
- Klass* _holder_klass; // to avoid name conflict with oopDesc::_klass
- CompiledICHolder* _next;
- bool _is_metadata_method;
-
- public:
- // Constructor
- CompiledICHolder(Metadata* metadata, Klass* klass, bool is_method = true);
- ~CompiledICHolder() NOT_DEBUG_RETURN;
-
-#ifdef ASSERT
- static int live_count() { return _live_count; }
- static int live_not_claimed_count() { return _live_not_claimed_count; }
-#endif
-
- // accessors
- Klass* holder_klass() const { return _holder_klass; }
- Metadata* holder_metadata() const { return _holder_metadata; }
-
- static ByteSize holder_metadata_offset() { return byte_offset_of(CompiledICHolder, _holder_metadata); }
- static ByteSize holder_klass_offset() { return byte_offset_of(CompiledICHolder, _holder_klass); }
-
- CompiledICHolder* next() { return _next; }
- void set_next(CompiledICHolder* n) { _next = n; }
-
- inline bool is_loader_alive();
-
- // Verify
- void verify_on(outputStream* st);
-
- // Printing
- void print_on(outputStream* st) const;
- void print_value_on(outputStream* st) const;
-
- const char* internal_name() const { return "{compiledICHolder}"; }
-
- void claim() NOT_DEBUG_RETURN;
-};
-
-#endif // SHARE_OOPS_COMPILEDICHOLDER_HPP
diff --git a/src/hotspot/share/oops/constMethod.cpp b/src/hotspot/share/oops/constMethod.cpp
index 8b4b509323f..a6a93d6f48a 100644
--- a/src/hotspot/share/oops/constMethod.cpp
+++ b/src/hotspot/share/oops/constMethod.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -384,7 +384,7 @@ AnnotationArray** ConstMethod::default_annotations_addr() const {
return (AnnotationArray**)constMethod_end() - offset;
}
-Array* copy_annotations(ClassLoaderData* loader_data, AnnotationArray* from, TRAPS) {
+static Array* copy_annotations(ClassLoaderData* loader_data, AnnotationArray* from, TRAPS) {
int length = from->length();
Array* a = MetadataFactory::new_array(loader_data, length, 0, CHECK_NULL);
memcpy((void*)a->adr_at(0), (void*)from->adr_at(0), length);
diff --git a/src/hotspot/share/oops/cpCache.cpp b/src/hotspot/share/oops/cpCache.cpp
index daa094baa7e..43e30a9ca3c 100644
--- a/src/hotspot/share/oops/cpCache.cpp
+++ b/src/hotspot/share/oops/cpCache.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -453,7 +453,7 @@ void ConstantPoolCache::set_archived_references(int root_index) {
#endif
#if INCLUDE_JVMTI
-void log_adjust(const char* entry_type, Method* old_method, Method* new_method, bool* trace_name_printed) {
+static void log_adjust(const char* entry_type, Method* old_method, Method* new_method, bool* trace_name_printed) {
ResourceMark rm;
if (!(*trace_name_printed)) {
diff --git a/src/hotspot/share/oops/instanceKlass.cpp b/src/hotspot/share/oops/instanceKlass.cpp
index 2178a506e5c..e236e583d24 100644
--- a/src/hotspot/share/oops/instanceKlass.cpp
+++ b/src/hotspot/share/oops/instanceKlass.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -85,7 +85,7 @@
#include "runtime/mutexLocker.hpp"
#include "runtime/orderAccess.hpp"
#include "runtime/os.inline.hpp"
-#include "runtime/reflectionUtils.hpp"
+#include "runtime/reflection.hpp"
#include "runtime/threads.hpp"
#include "services/classLoadingService.hpp"
#include "services/finalizerService.hpp"
@@ -2531,7 +2531,6 @@ void InstanceKlass::clean_method_data() {
for (int m = 0; m < methods()->length(); m++) {
MethodData* mdo = methods()->at(m)->method_data();
if (mdo != nullptr) {
- ConditionalMutexLocker ml(mdo->extra_data_lock(), !SafepointSynchronize::is_at_safepoint());
mdo->clean_method_data(/*always_clean*/false);
}
}
diff --git a/src/hotspot/share/oops/klass.hpp b/src/hotspot/share/oops/klass.hpp
index 96f8f8efdb6..0c9c05bed32 100644
--- a/src/hotspot/share/oops/klass.hpp
+++ b/src/hotspot/share/oops/klass.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -582,7 +582,7 @@ class Klass : public Metadata {
if (has_archived_mirror_index()) {
// _java_mirror is not a valid OopHandle but rather an encoded reference in the shared heap
return false;
- } else if (_java_mirror.ptr_raw() == nullptr) {
+ } else if (_java_mirror.is_empty()) {
return false;
} else {
return true;
diff --git a/src/hotspot/share/oops/klassVtable.cpp b/src/hotspot/share/oops/klassVtable.cpp
index ece1ad3c6de..cbc379709f1 100644
--- a/src/hotspot/share/oops/klassVtable.cpp
+++ b/src/hotspot/share/oops/klassVtable.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1442,7 +1442,7 @@ class InterfaceVisiterClosure : public StackObj {
};
// Visit all interfaces with at least one itable method
-void visit_all_interfaces(Array* transitive_intf, InterfaceVisiterClosure *blk) {
+static void visit_all_interfaces(Array* transitive_intf, InterfaceVisiterClosure *blk) {
// Handle array argument
for(int i = 0; i < transitive_intf->length(); i++) {
InstanceKlass* intf = transitive_intf->at(i);
diff --git a/src/hotspot/share/oops/method.cpp b/src/hotspot/share/oops/method.cpp
index 5cd59a802be..a36974fc3f0 100644
--- a/src/hotspot/share/oops/method.cpp
+++ b/src/hotspot/share/oops/method.cpp
@@ -72,6 +72,7 @@
#include "runtime/safepointVerifiers.hpp"
#include "runtime/sharedRuntime.hpp"
#include "runtime/signature.hpp"
+#include "runtime/threads.hpp"
#include "runtime/vm_version.hpp"
#include "utilities/align.hpp"
#include "utilities/quickSort.hpp"
@@ -540,38 +541,38 @@ bool Method::was_executed_more_than(int n) {
}
}
-void Method::print_invocation_count() {
+void Method::print_invocation_count(outputStream* st) {
//---< compose+print method return type, klass, name, and signature >---
- if (is_static()) tty->print("static ");
- if (is_final()) tty->print("final ");
- if (is_synchronized()) tty->print("synchronized ");
- if (is_native()) tty->print("native ");
- tty->print("%s::", method_holder()->external_name());
- name()->print_symbol_on(tty);
- signature()->print_symbol_on(tty);
+ if (is_static()) { st->print("static "); }
+ if (is_final()) { st->print("final "); }
+ if (is_synchronized()) { st->print("synchronized "); }
+ if (is_native()) { st->print("native "); }
+ st->print("%s::", method_holder()->external_name());
+ name()->print_symbol_on(st);
+ signature()->print_symbol_on(st);
if (WizardMode) {
// dump the size of the byte codes
- tty->print(" {%d}", code_size());
+ st->print(" {%d}", code_size());
}
- tty->cr();
+ st->cr();
// Counting based on signed int counters tends to overflow with
// longer-running workloads on fast machines. The counters under
// consideration here, however, are limited in range by counting
// logic. See InvocationCounter:count_limit for example.
// No "overflow precautions" need to be implemented here.
- tty->print_cr (" interpreter_invocation_count: " INT32_FORMAT_W(11), interpreter_invocation_count());
- tty->print_cr (" invocation_counter: " INT32_FORMAT_W(11), invocation_count());
- tty->print_cr (" backedge_counter: " INT32_FORMAT_W(11), backedge_count());
+ st->print_cr (" interpreter_invocation_count: " INT32_FORMAT_W(11), interpreter_invocation_count());
+ st->print_cr (" invocation_counter: " INT32_FORMAT_W(11), invocation_count());
+ st->print_cr (" backedge_counter: " INT32_FORMAT_W(11), backedge_count());
if (method_data() != nullptr) {
- tty->print_cr (" decompile_count: " UINT32_FORMAT_W(11), method_data()->decompile_count());
+ st->print_cr (" decompile_count: " UINT32_FORMAT_W(11), method_data()->decompile_count());
}
#ifndef PRODUCT
if (CountCompiledCalls) {
- tty->print_cr (" compiled_invocation_count: " INT64_FORMAT_W(11), compiled_invocation_count());
+ st->print_cr (" compiled_invocation_count: " INT64_FORMAT_W(11), compiled_invocation_count());
}
#endif
}
@@ -1249,10 +1250,17 @@ void Method::link_method(const methodHandle& h_method, TRAPS) {
// ONLY USE the h_method now as make_adapter may have blocked
if (h_method->is_continuation_native_intrinsic()) {
- // the entry points to this method will be set in set_code, called when first resolving this method
_from_interpreted_entry = nullptr;
_from_compiled_entry = nullptr;
_i2i_entry = nullptr;
+ if (Continuations::enabled()) {
+ assert(!Threads::is_vm_complete(), "should only be called during vm init");
+ AdapterHandlerLibrary::create_native_wrapper(h_method);
+ if (!h_method->has_compiled_code()) {
+ THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(), "Initial size of CodeCache is too small");
+ }
+ assert(_from_interpreted_entry == get_i2c_entry(), "invariant");
+ }
}
}
diff --git a/src/hotspot/share/oops/method.hpp b/src/hotspot/share/oops/method.hpp
index 89d5ccbb168..e4bff457504 100644
--- a/src/hotspot/share/oops/method.hpp
+++ b/src/hotspot/share/oops/method.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -454,7 +454,7 @@ class Method : public Metadata {
void mask_for(int bci, InterpreterOopMap* mask);
// operations on invocation counter
- void print_invocation_count();
+ void print_invocation_count(outputStream* st);
// byte codes
void set_code(address code) { return constMethod()->set_code(code); }
diff --git a/src/hotspot/share/oops/methodData.cpp b/src/hotspot/share/oops/methodData.cpp
index 1107a53a8ed..008de899bdb 100644
--- a/src/hotspot/share/oops/methodData.cpp
+++ b/src/hotspot/share/oops/methodData.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2000, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1218,7 +1218,7 @@ void MethodData::post_initialize(BytecodeStream* stream) {
MethodData::MethodData(const methodHandle& method)
: _method(method()),
// Holds Compile_lock
- _extra_data_lock(Mutex::safepoint-2, "MDOExtraData_lock"),
+ _extra_data_lock(Mutex::nosafepoint, "MDOExtraData_lock"),
_compiler_counters(),
_parameters_type_data_di(parameters_uninitialized) {
initialize();
@@ -1379,6 +1379,8 @@ address MethodData::bci_to_dp(int bci) {
// Translate a bci to its corresponding data, or null.
ProfileData* MethodData::bci_to_data(int bci) {
+ check_extra_data_locked();
+
DataLayout* data = data_layout_before(bci);
for ( ; is_valid(data); data = next_data_layout(data)) {
if (data->bci() == bci) {
@@ -1429,7 +1431,9 @@ DataLayout* MethodData::next_extra(DataLayout* dp) {
return (DataLayout*)((address)dp + DataLayout::compute_size_in_bytes(nb_cells));
}
-ProfileData* MethodData::bci_to_extra_data_helper(int bci, Method* m, DataLayout*& dp, bool concurrent) {
+ProfileData* MethodData::bci_to_extra_data_find(int bci, Method* m, DataLayout*& dp) {
+ check_extra_data_locked();
+
DataLayout* end = args_data_limit();
for (;; dp = next_extra(dp)) {
@@ -1450,14 +1454,9 @@ ProfileData* MethodData::bci_to_extra_data_helper(int bci, Method* m, DataLayout
case DataLayout::speculative_trap_data_tag:
if (m != nullptr) {
SpeculativeTrapData* data = new SpeculativeTrapData(dp);
- // data->method() may be null in case of a concurrent
- // allocation. Maybe it's for the same method. Try to use that
- // entry in that case.
if (dp->bci() == bci) {
- if (data->method() == nullptr) {
- assert(concurrent, "impossible because no concurrent allocation");
- return nullptr;
- } else if (data->method() == m) {
+ assert(data->method() != nullptr, "method must be set");
+ if (data->method() == m) {
return data;
}
}
@@ -1473,6 +1472,8 @@ ProfileData* MethodData::bci_to_extra_data_helper(int bci, Method* m, DataLayout
// Translate a bci to its corresponding extra data, or null.
ProfileData* MethodData::bci_to_extra_data(int bci, Method* m, bool create_if_missing) {
+ check_extra_data_locked();
+
// This code assumes an entry for a SpeculativeTrapData is 2 cells
assert(2*DataLayout::compute_size_in_bytes(BitData::static_cell_count()) ==
DataLayout::compute_size_in_bytes(SpeculativeTrapData::static_cell_count()),
@@ -1486,23 +1487,14 @@ ProfileData* MethodData::bci_to_extra_data(int bci, Method* m, bool create_if_mi
DataLayout* dp = extra_data_base();
DataLayout* end = args_data_limit();
- // Allocation in the extra data space has to be atomic because not
- // all entries have the same size and non atomic concurrent
- // allocation would result in a corrupted extra data space.
- ProfileData* result = bci_to_extra_data_helper(bci, m, dp, true);
- if (result != nullptr) {
+ // Find if already exists
+ ProfileData* result = bci_to_extra_data_find(bci, m, dp);
+ if (result != nullptr || dp >= end) {
return result;
}
- if (create_if_missing && dp < end) {
- MutexLocker ml(&_extra_data_lock);
- // Check again now that we have the lock. Another thread may
- // have added extra data entries.
- ProfileData* result = bci_to_extra_data_helper(bci, m, dp, false);
- if (result != nullptr || dp >= end) {
- return result;
- }
-
+ if (create_if_missing) {
+ // Not found -> Allocate
assert(dp->tag() == DataLayout::no_tag || (dp->tag() == DataLayout::speculative_trap_data_tag && m != nullptr), "should be free");
assert(next_extra(dp)->tag() == DataLayout::no_tag || next_extra(dp)->tag() == DataLayout::arg_info_data_tag, "should be free or arg info");
u1 tag = m == nullptr ? DataLayout::bit_data_tag : DataLayout::speculative_trap_data_tag;
@@ -1729,6 +1721,8 @@ void MethodData::metaspace_pointers_do(MetaspaceClosure* it) {
}
void MethodData::clean_extra_data_helper(DataLayout* dp, int shift, bool reset) {
+ check_extra_data_locked();
+
if (shift == 0) {
return;
}
@@ -1770,6 +1764,8 @@ class CleanExtraDataMethodClosure : public CleanExtraDataClosure {
// Remove SpeculativeTrapData entries that reference an unloaded or
// redefined method
void MethodData::clean_extra_data(CleanExtraDataClosure* cl) {
+ check_extra_data_locked();
+
DataLayout* dp = extra_data_base();
DataLayout* end = args_data_limit();
@@ -1814,6 +1810,8 @@ void MethodData::clean_extra_data(CleanExtraDataClosure* cl) {
// Verify there's no unloaded or redefined method referenced by a
// SpeculativeTrapData entry
void MethodData::verify_extra_data_clean(CleanExtraDataClosure* cl) {
+ check_extra_data_locked();
+
#ifdef ASSERT
DataLayout* dp = extra_data_base();
DataLayout* end = args_data_limit();
@@ -1851,6 +1849,10 @@ void MethodData::clean_method_data(bool always_clean) {
}
CleanExtraDataKlassClosure cl(always_clean);
+
+ // Lock to modify extra data, and prevent Safepoint from breaking the lock
+ MutexLocker ml(extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
clean_extra_data(&cl);
verify_extra_data_clean(&cl);
}
@@ -1860,6 +1862,10 @@ void MethodData::clean_method_data(bool always_clean) {
void MethodData::clean_weak_method_links() {
ResourceMark rm;
CleanExtraDataMethodClosure cl;
+
+ // Lock to modify extra data, and prevent Safepoint from breaking the lock
+ MutexLocker ml(extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
clean_extra_data(&cl);
verify_extra_data_clean(&cl);
}
@@ -1873,3 +1879,16 @@ void MethodData::release_C_heap_structures() {
FailedSpeculation::free_failed_speculations(get_failed_speculations_address());
#endif
}
+
+#ifdef ASSERT
+void MethodData::check_extra_data_locked() const {
+ // Cast const away, just to be able to verify the lock
+ // Usually we only want non-const accesses on the lock,
+ // so this here is an exception.
+ MethodData* self = (MethodData*)this;
+ assert(self->extra_data_lock()->owned_by_self(), "must have lock");
+ assert(!Thread::current()->is_Java_thread() ||
+ JavaThread::current()->is_in_no_safepoint_scope(),
+ "JavaThread must have NoSafepointVerifier inside lock scope");
+}
+#endif
diff --git a/src/hotspot/share/oops/methodData.hpp b/src/hotspot/share/oops/methodData.hpp
index bff0f619f4b..4fa42eec960 100644
--- a/src/hotspot/share/oops/methodData.hpp
+++ b/src/hotspot/share/oops/methodData.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2000, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -2168,7 +2168,7 @@ class MethodData : public Metadata {
// What is the index of the first data entry?
int first_di() const { return 0; }
- ProfileData* bci_to_extra_data_helper(int bci, Method* m, DataLayout*& dp, bool concurrent);
+ ProfileData* bci_to_extra_data_find(int bci, Method* m, DataLayout*& dp);
// Find or create an extra ProfileData:
ProfileData* bci_to_extra_data(int bci, Method* m, bool create_if_missing);
@@ -2310,20 +2310,12 @@ class MethodData : public Metadata {
intx arg_local() { return _arg_local; }
intx arg_stack() { return _arg_stack; }
intx arg_returned() { return _arg_returned; }
- uint arg_modified(int a) { ArgInfoData *aid = arg_info();
- assert(aid != nullptr, "arg_info must be not null");
- assert(a >= 0 && a < aid->number_of_args(), "valid argument number");
- return aid->arg_modified(a); }
-
+ uint arg_modified(int a);
void set_eflags(intx v) { _eflags = v; }
void set_arg_local(intx v) { _arg_local = v; }
void set_arg_stack(intx v) { _arg_stack = v; }
void set_arg_returned(intx v) { _arg_returned = v; }
- void set_arg_modified(int a, uint v) { ArgInfoData *aid = arg_info();
- assert(aid != nullptr, "arg_info must be not null");
- assert(a >= 0 && a < aid->number_of_args(), "valid argument number");
- aid->set_arg_modified(a, v); }
-
+ void set_arg_modified(int a, uint v);
void clear_escape_info() { _eflags = _arg_local = _arg_stack = _arg_returned = 0; }
// Location and size of data area
@@ -2371,6 +2363,8 @@ class MethodData : public Metadata {
// Same, but try to create an extra_data record if one is needed:
ProfileData* allocate_bci_to_data(int bci, Method* m) {
+ check_extra_data_locked();
+
ProfileData* data = nullptr;
// If m not null, try to allocate a SpeculativeTrapData entry
if (m == nullptr) {
@@ -2397,7 +2391,10 @@ class MethodData : public Metadata {
// Add a handful of extra data records, for trap tracking.
// Only valid after 'set_size' is called at the end of MethodData::initialize
- DataLayout* extra_data_base() const { return limit_data_position(); }
+ DataLayout* extra_data_base() const {
+ check_extra_data_locked();
+ return limit_data_position();
+ }
DataLayout* extra_data_limit() const { return (DataLayout*)((address)this + size_in_bytes()); }
// pointers to sections in extra data
DataLayout* args_data_limit() const { return parameters_data_base(); }
@@ -2412,7 +2409,7 @@ class MethodData : public Metadata {
DataLayout* exception_handler_data_base() const { return data_layout_at(_exception_handler_data_di); }
DataLayout* exception_handler_data_limit() const { return extra_data_limit(); }
- int extra_data_size() const { return (int)((address)extra_data_limit() - (address)extra_data_base()); }
+ int extra_data_size() const { return (int)((address)extra_data_limit() - (address)limit_data_position()); }
static DataLayout* next_extra(DataLayout* dp);
// Return (uint)-1 for overflow.
@@ -2529,6 +2526,7 @@ class MethodData : public Metadata {
void clean_method_data(bool always_clean);
void clean_weak_method_links();
Mutex* extra_data_lock() { return &_extra_data_lock; }
+ void check_extra_data_locked() const NOT_DEBUG_RETURN;
};
#endif // SHARE_OOPS_METHODDATA_HPP
diff --git a/src/hotspot/share/oops/methodData.inline.hpp b/src/hotspot/share/oops/methodData.inline.hpp
index e8084380a27..a59271431b5 100644
--- a/src/hotspot/share/oops/methodData.inline.hpp
+++ b/src/hotspot/share/oops/methodData.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2018, 2019, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -28,6 +28,7 @@
#include "oops/methodData.hpp"
#include "runtime/atomic.hpp"
+#include "runtime/mutexLocker.hpp"
inline void DataLayout::release_set_cell_at(int index, intptr_t value) {
Atomic::release_store(&_cells[index], value);
@@ -53,4 +54,22 @@ inline void RetData::release_set_bci(uint row, int bci) {
release_set_int_at(bci0_offset + row * ret_row_cell_count, bci);
}
+inline uint MethodData::arg_modified(int a) {
+ // Lock and avoid breaking lock with Safepoint
+ MutexLocker ml(extra_data_lock(), Mutex::_no_safepoint_check_flag);
+ ArgInfoData* aid = arg_info();
+ assert(aid != nullptr, "arg_info must be not null");
+ assert(a >= 0 && a < aid->number_of_args(), "valid argument number");
+ return aid->arg_modified(a);
+}
+
+inline void MethodData::set_arg_modified(int a, uint v) {
+ // Lock and avoid breaking lock with Safepoint
+ MutexLocker ml(extra_data_lock(), Mutex::_no_safepoint_check_flag);
+ ArgInfoData* aid = arg_info();
+ assert(aid != nullptr, "arg_info must be not null");
+ assert(a >= 0 && a < aid->number_of_args(), "valid argument number");
+ aid->set_arg_modified(a, v);
+}
+
#endif // SHARE_OOPS_METHODDATA_INLINE_HPP
diff --git a/src/hotspot/share/oops/oopHandle.hpp b/src/hotspot/share/oops/oopHandle.hpp
index ae631fd7d53..983ddf9ae5c 100644
--- a/src/hotspot/share/oops/oopHandle.hpp
+++ b/src/hotspot/share/oops/oopHandle.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -75,15 +75,4 @@ class OopHandle {
oop* ptr_raw() const { return _obj; }
};
-// Convert OopHandle to oop*
-
-template<>
-struct PrimitiveConversions::Translate : public std::true_type {
- typedef OopHandle Value;
- typedef oop* Decayed;
-
- static Decayed decay(Value x) { return x.ptr_raw(); }
- static Value recover(Decayed x) { return OopHandle(x); }
-};
-
#endif // SHARE_OOPS_OOPHANDLE_HPP
diff --git a/src/hotspot/share/oops/oopHandle.inline.hpp b/src/hotspot/share/oops/oopHandle.inline.hpp
index 95db9210e96..23a13cb35e3 100644
--- a/src/hotspot/share/oops/oopHandle.inline.hpp
+++ b/src/hotspot/share/oops/oopHandle.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2018, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2018, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -57,9 +57,8 @@ inline void OopHandle::release(OopStorage* storage) {
}
inline void OopHandle::replace(oop obj) {
- oop* ptr = ptr_raw();
- assert(ptr != nullptr, "should not use replace");
- NativeAccess<>::oop_store(ptr, obj);
+ assert(!is_empty(), "should not use replace");
+ NativeAccess<>::oop_store(_obj, obj);
}
inline oop OopHandle::xchg(oop new_value) {
diff --git a/src/hotspot/share/oops/oopsHierarchy.hpp b/src/hotspot/share/oops/oopsHierarchy.hpp
index b01153da46d..4bc5a7d4c17 100644
--- a/src/hotspot/share/oops/oopsHierarchy.hpp
+++ b/src/hotspot/share/oops/oopsHierarchy.hpp
@@ -179,9 +179,6 @@ class MethodData;
// class Metadata
class Method;
class ConstantPool;
-// class CHeapObj
-class CompiledICHolder;
-
// The klass hierarchy is separate from the oop hierarchy.
diff --git a/src/hotspot/share/opto/addnode.cpp b/src/hotspot/share/opto/addnode.cpp
index 1b921933004..443086b6737 100644
--- a/src/hotspot/share/opto/addnode.cpp
+++ b/src/hotspot/share/opto/addnode.cpp
@@ -764,7 +764,7 @@ Node* OrINode::Identity(PhaseGVN* phase) {
}
// Find shift value for Integer or Long OR.
-Node* rotate_shift(PhaseGVN* phase, Node* lshift, Node* rshift, int mask) {
+static Node* rotate_shift(PhaseGVN* phase, Node* lshift, Node* rshift, int mask) {
// val << norm_con_shift | val >> ({32|64} - norm_con_shift) => rotate_left val, norm_con_shift
const TypeInt* lshift_t = phase->type(lshift)->isa_int();
const TypeInt* rshift_t = phase->type(rshift)->isa_int();
@@ -1080,7 +1080,7 @@ const Type* XorLNode::Value(PhaseGVN* phase) const {
return AddNode::Value(phase);
}
-Node* build_min_max_int(Node* a, Node* b, bool is_max) {
+static Node* build_min_max_int(Node* a, Node* b, bool is_max) {
if (is_max) {
return new MaxINode(a, b);
} else {
@@ -1312,7 +1312,7 @@ const Type *MinINode::add_ring( const Type *t0, const Type *t1 ) const {
//
// Note: we assume that SubL was already replaced by an AddL, and that the stride
// has its sign flipped: SubL(limit, stride) -> AddL(limit, -stride).
-Node* fold_subI_no_underflow_pattern(Node* n, PhaseGVN* phase) {
+static Node* fold_subI_no_underflow_pattern(Node* n, PhaseGVN* phase) {
assert(n->Opcode() == Op_MaxL || n->Opcode() == Op_MinL, "sanity");
// Check that the two clamps have the correct values.
jlong clamp = (n->Opcode() == Op_MaxL) ? min_jint : max_jint;
diff --git a/src/hotspot/share/opto/c2_CodeStubs.hpp b/src/hotspot/share/opto/c2_CodeStubs.hpp
index 3df1e4b726d..83d17081070 100644
--- a/src/hotspot/share/opto/c2_CodeStubs.hpp
+++ b/src/hotspot/share/opto/c2_CodeStubs.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2022, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -97,6 +97,26 @@ class C2EntryBarrierStub: public C2CodeStub {
void emit(C2_MacroAssembler& masm);
};
+class C2FastUnlockLightweightStub : public C2CodeStub {
+private:
+ Register _obj;
+ Register _mark;
+ Register _t;
+ Register _thread;
+ Label _push_and_slow_path;
+ Label _check_successor;
+ Label _unlocked_continuation;
+public:
+ C2FastUnlockLightweightStub(Register obj, Register mark, Register t, Register thread) : C2CodeStub(),
+ _obj(obj), _mark(mark), _t(t), _thread(thread) {}
+ int max_size() const;
+ void emit(C2_MacroAssembler& masm);
+ Label& push_and_slow_path() { return _push_and_slow_path; }
+ Label& check_successor() { return _check_successor; }
+ Label& unlocked_continuation() { return _unlocked_continuation; }
+ Label& slow_path_continuation() { return continuation(); }
+};
+
#ifdef _LP64
class C2HandleAnonOMOwnerStub : public C2CodeStub {
private:
diff --git a/src/hotspot/share/opto/c2_globals.hpp b/src/hotspot/share/opto/c2_globals.hpp
index 885c0dd75a7..68d4f89b8f2 100644
--- a/src/hotspot/share/opto/c2_globals.hpp
+++ b/src/hotspot/share/opto/c2_globals.hpp
@@ -56,6 +56,9 @@
product(bool, StressIncrementalInlining, false, DIAGNOSTIC, \
"Randomize the incremental inlining decision") \
\
+ product(bool, StressMacroExpansion, false, DIAGNOSTIC, \
+ "Randomize macro node expansion order") \
+ \
product(uint, StressSeed, 0, DIAGNOSTIC, \
"Seed for randomized stress testing (if unset, a random one is " \
"generated). The seed is recorded in the compilation log, if " \
@@ -339,9 +342,6 @@
product(bool, UseSuperWord, true, \
"Transform scalar operations into superword operations") \
\
- develop(bool, SuperWordRTDepCheck, false, \
- "Enable runtime dependency checks.") \
- \
product(bool, SuperWordReductions, true, \
"Enable reductions support in superword.") \
\
diff --git a/src/hotspot/share/opto/c2compiler.cpp b/src/hotspot/share/opto/c2compiler.cpp
index b7c761d9b98..25dfee91a91 100644
--- a/src/hotspot/share/opto/c2compiler.cpp
+++ b/src/hotspot/share/opto/c2compiler.cpp
@@ -245,7 +245,6 @@ bool C2Compiler::is_intrinsic_supported(vmIntrinsics::ID id) {
if (!Matcher::match_rule_supported(Op_StrComp)) return false;
break;
case vmIntrinsics::_equalsL:
- case vmIntrinsics::_equalsU:
if (!Matcher::match_rule_supported(Op_StrEquals)) return false;
break;
case vmIntrinsics::_vectorizedHashCode:
diff --git a/src/hotspot/share/opto/callGenerator.cpp b/src/hotspot/share/opto/callGenerator.cpp
index ef7c572f582..7c022e27511 100644
--- a/src/hotspot/share/opto/callGenerator.cpp
+++ b/src/hotspot/share/opto/callGenerator.cpp
@@ -684,6 +684,9 @@ void CallGenerator::do_late_inline_helper() {
C->print_inlining_update_delayed(this);
return;
}
+ if (C->print_inlining() && (is_mh_late_inline() || is_virtual_late_inline())) {
+ C->print_inlining_update_delayed(this);
+ }
// Setup default node notes to be picked up by the inlining
Node_Notes* old_nn = C->node_notes_at(call->_idx);
diff --git a/src/hotspot/share/opto/callnode.cpp b/src/hotspot/share/opto/callnode.cpp
index 6e75cf67109..3379a9b01e0 100644
--- a/src/hotspot/share/opto/callnode.cpp
+++ b/src/hotspot/share/opto/callnode.cpp
@@ -1463,9 +1463,10 @@ void SafePointNode::disconnect_from_root(PhaseIterGVN *igvn) {
//============== SafePointScalarObjectNode ==============
-SafePointScalarObjectNode::SafePointScalarObjectNode(const TypeOopPtr* tp, Node* alloc, uint first_index, uint n_fields) :
+SafePointScalarObjectNode::SafePointScalarObjectNode(const TypeOopPtr* tp, Node* alloc, uint first_index, uint depth, uint n_fields) :
TypeNode(tp, 1), // 1 control input -- seems required. Get from root.
_first_index(first_index),
+ _depth(depth),
_n_fields(n_fields),
_alloc(alloc)
{
diff --git a/src/hotspot/share/opto/callnode.hpp b/src/hotspot/share/opto/callnode.hpp
index b659cb67082..efa84850bb2 100644
--- a/src/hotspot/share/opto/callnode.hpp
+++ b/src/hotspot/share/opto/callnode.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -508,7 +508,7 @@ class SafePointNode : public MultiNode {
class SafePointScalarObjectNode: public TypeNode {
uint _first_index; // First input edge relative index of a SafePoint node where
// states of the scalarized object fields are collected.
- // It is relative to the last (youngest) jvms->_scloff.
+ uint _depth; // Depth of the JVM state the _first_index field refers to
uint _n_fields; // Number of non-static fields of the scalarized object.
Node* _alloc; // Just for debugging purposes.
@@ -519,7 +519,7 @@ class SafePointScalarObjectNode: public TypeNode {
uint first_index() const { return _first_index; }
public:
- SafePointScalarObjectNode(const TypeOopPtr* tp, Node* alloc, uint first_index, uint n_fields);
+ SafePointScalarObjectNode(const TypeOopPtr* tp, Node* alloc, uint first_index, uint depth, uint n_fields);
virtual int Opcode() const;
virtual uint ideal_reg() const;
@@ -529,7 +529,7 @@ class SafePointScalarObjectNode: public TypeNode {
uint first_index(JVMState* jvms) const {
assert(jvms != nullptr, "missed JVMS");
- return jvms->scloff() + _first_index;
+ return jvms->of_depth(_depth)->scloff() + _first_index;
}
uint n_fields() const { return _n_fields; }
diff --git a/src/hotspot/share/opto/compile.cpp b/src/hotspot/share/opto/compile.cpp
index acce455b726..83b922492b3 100644
--- a/src/hotspot/share/opto/compile.cpp
+++ b/src/hotspot/share/opto/compile.cpp
@@ -844,7 +844,8 @@ Compile::Compile( ciEnv* ci_env, ciMethod* target, int osr_bci,
// If any phase is randomized for stress testing, seed random number
// generation and log the seed for repeatability.
- if (StressLCM || StressGCM || StressIGVN || StressCCP || StressIncrementalInlining) {
+ if (StressLCM || StressGCM || StressIGVN || StressCCP ||
+ StressIncrementalInlining || StressMacroExpansion) {
if (FLAG_IS_DEFAULT(StressSeed) || (FLAG_IS_ERGO(StressSeed) && directive->RepeatCompilationOption)) {
_stress_seed = static_cast(Ticks::now().nanoseconds());
FLAG_SET_ERGO(StressSeed, _stress_seed);
@@ -2451,12 +2452,13 @@ void Compile::Optimize() {
{
TracePhase tp("macroExpand", &timers[_t_macroExpand]);
+ print_method(PHASE_BEFORE_MACRO_EXPANSION, 3);
PhaseMacroExpand mex(igvn);
if (mex.expand_macro_nodes()) {
assert(failing(), "must bail out w/ explicit message");
return;
}
- print_method(PHASE_MACRO_EXPANSION, 2);
+ print_method(PHASE_AFTER_MACRO_EXPANSION, 2);
}
{
@@ -5121,6 +5123,16 @@ void CloneMap::dump(node_idx_t key, outputStream* st) const {
}
}
+void Compile::shuffle_macro_nodes() {
+ if (_macro_nodes.length() < 2) {
+ return;
+ }
+ for (uint i = _macro_nodes.length() - 1; i >= 1; i--) {
+ uint j = C->random() % (i + 1);
+ swap(_macro_nodes.at(i), _macro_nodes.at(j));
+ }
+}
+
// Move Allocate nodes to the start of the list
void Compile::sort_macro_nodes() {
int count = macro_count();
@@ -5140,7 +5152,7 @@ void Compile::sort_macro_nodes() {
void Compile::print_method(CompilerPhaseType cpt, int level, Node* n) {
if (failing()) { return; }
- EventCompilerPhase event;
+ EventCompilerPhase event(UNTIMED);
if (event.should_commit()) {
CompilerEvent::PhaseEvent::post(event, C->_latest_stage_start_counter, cpt, C->_compile_id, level);
}
@@ -5179,7 +5191,7 @@ void Compile::begin_method() {
// Only used from CompileWrapper
void Compile::end_method() {
- EventCompilerPhase event;
+ EventCompilerPhase event(UNTIMED);
if (event.should_commit()) {
CompilerEvent::PhaseEvent::post(event, C->_latest_stage_start_counter, PHASE_END, C->_compile_id, 1);
}
diff --git a/src/hotspot/share/opto/compile.hpp b/src/hotspot/share/opto/compile.hpp
index 42f866df781..a5e4b69d263 100644
--- a/src/hotspot/share/opto/compile.hpp
+++ b/src/hotspot/share/opto/compile.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -794,6 +794,7 @@ class Compile : public Phase {
void remove_useless_unstable_if_traps(Unique_Node_List &useful);
void process_for_unstable_if_traps(PhaseIterGVN& igvn);
+ void shuffle_macro_nodes();
void sort_macro_nodes();
void mark_parse_predicate_nodes_useless(PhaseIterGVN& igvn);
diff --git a/src/hotspot/share/opto/doCall.cpp b/src/hotspot/share/opto/doCall.cpp
index 122da9b447c..0a5e27ed5b1 100644
--- a/src/hotspot/share/opto/doCall.cpp
+++ b/src/hotspot/share/opto/doCall.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -50,15 +50,15 @@
#include "jfr/jfr.hpp"
#endif
-void print_trace_type_profile(outputStream* out, int depth, ciKlass* prof_klass, int site_count, int receiver_count) {
+static void print_trace_type_profile(outputStream* out, int depth, ciKlass* prof_klass, int site_count, int receiver_count) {
CompileTask::print_inline_indent(depth, out);
out->print(" \\-> TypeProfile (%d/%d counts) = ", receiver_count, site_count);
prof_klass->name()->print_symbol_on(out);
out->cr();
}
-void trace_type_profile(Compile* C, ciMethod* method, int depth, int bci, ciMethod* prof_method,
- ciKlass* prof_klass, int site_count, int receiver_count) {
+static void trace_type_profile(Compile* C, ciMethod* method, int depth, int bci, ciMethod* prof_method,
+ ciKlass* prof_klass, int site_count, int receiver_count) {
if (TraceTypeProfile || C->print_inlining()) {
outputStream* out = tty;
if (!C->print_inlining()) {
diff --git a/src/hotspot/share/opto/graphKit.cpp b/src/hotspot/share/opto/graphKit.cpp
index 23891215281..0c35b2145aa 100644
--- a/src/hotspot/share/opto/graphKit.cpp
+++ b/src/hotspot/share/opto/graphKit.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2001, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2001, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -3468,7 +3468,10 @@ FastLockNode* GraphKit::shared_lock(Node* obj) {
assert(dead_locals_are_killed(), "should kill locals before sync. point");
// Box the stack location
- Node* box = _gvn.transform(new BoxLockNode(next_monitor()));
+ Node* box = new BoxLockNode(next_monitor());
+ // Check for bailout after new BoxLockNode
+ if (failing()) { return nullptr; }
+ box = _gvn.transform(box);
Node* mem = reset_memory();
FastLockNode * flock = _gvn.transform(new FastLockNode(0, obj, box) )->as_FastLock();
diff --git a/src/hotspot/share/opto/intrinsicnode.cpp b/src/hotspot/share/opto/intrinsicnode.cpp
index 7d8894c8bdf..010579722bb 100644
--- a/src/hotspot/share/opto/intrinsicnode.cpp
+++ b/src/hotspot/share/opto/intrinsicnode.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -174,7 +174,7 @@ Node* CompressBitsNode::Ideal(PhaseGVN* phase, bool can_reshape) {
return nullptr;
}
-Node* compress_expand_identity(PhaseGVN* phase, Node* n) {
+static Node* compress_expand_identity(PhaseGVN* phase, Node* n) {
BasicType bt = n->bottom_type()->basic_type();
// compress(x, 0) == 0, expand(x, 0) == 0
if(phase->type(n->in(2))->higher_equal(TypeInteger::zero(bt))) return n->in(2);
diff --git a/src/hotspot/share/opto/library_call.cpp b/src/hotspot/share/opto/library_call.cpp
index 6bb3e16475a..efbf900f3d7 100644
--- a/src/hotspot/share/opto/library_call.cpp
+++ b/src/hotspot/share/opto/library_call.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -311,7 +311,6 @@ bool LibraryCallKit::try_to_inline(int predicate) {
case vmIntrinsics::_indexOfL_char: return inline_string_indexOfChar(StrIntrinsicNode::L);
case vmIntrinsics::_equalsL: return inline_string_equals(StrIntrinsicNode::LL);
- case vmIntrinsics::_equalsU: return inline_string_equals(StrIntrinsicNode::UU);
case vmIntrinsics::_vectorizedHashCode: return inline_vectorizedHashCode();
@@ -2941,7 +2940,7 @@ bool LibraryCallKit::inline_native_notify_jvmti_hide() {
{
// unconditionally update the temporary VTMS transition bit in current JavaThread
Node* thread = ideal.thread();
- Node* hide = _gvn.transform(argument(1)); // hide argument for temporary VTMS transition notification
+ Node* hide = _gvn.transform(argument(0)); // hide argument for temporary VTMS transition notification
Node* addr = basic_plus_adr(thread, in_bytes(JavaThread::is_in_tmp_VTMS_transition_offset()));
const TypePtr *addr_type = _gvn.type(addr)->isa_ptr();
@@ -2964,7 +2963,7 @@ bool LibraryCallKit::inline_native_notify_jvmti_sync() {
{
// unconditionally update the is_disable_suspend bit in current JavaThread
Node* thread = ideal.thread();
- Node* arg = _gvn.transform(argument(1)); // argument for notification
+ Node* arg = _gvn.transform(argument(0)); // argument for notification
Node* addr = basic_plus_adr(thread, in_bytes(JavaThread::is_disable_suspend_offset()));
const TypePtr *addr_type = _gvn.type(addr)->isa_ptr();
diff --git a/src/hotspot/share/opto/locknode.cpp b/src/hotspot/share/opto/locknode.cpp
index 6354df1ec59..d73150684a7 100644
--- a/src/hotspot/share/opto/locknode.cpp
+++ b/src/hotspot/share/opto/locknode.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1999, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1999, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -44,6 +44,10 @@ BoxLockNode::BoxLockNode( int slot ) : Node( Compile::current()->root() ),
init_class_id(Class_BoxLock);
init_flags(Flag_rematerialize);
OptoReg::Name reg = OptoReg::stack2reg(_slot);
+ if (!RegMask::can_represent(reg, Compile::current()->sync_stack_slots())) {
+ Compile::current()->record_method_not_compilable("must be able to represent all monitor slots in reg mask");
+ return;
+ }
_inmask.Insert(reg);
}
diff --git a/src/hotspot/share/opto/loopTransform.cpp b/src/hotspot/share/opto/loopTransform.cpp
index a23285fe389..b9e14f595c2 100644
--- a/src/hotspot/share/opto/loopTransform.cpp
+++ b/src/hotspot/share/opto/loopTransform.cpp
@@ -37,6 +37,7 @@
#include "opto/mulnode.hpp"
#include "opto/movenode.hpp"
#include "opto/opaquenode.hpp"
+#include "opto/phase.hpp"
#include "opto/predicates.hpp"
#include "opto/rootnode.hpp"
#include "opto/runtime.hpp"
@@ -1101,12 +1102,11 @@ void IdealLoopTree::policy_unroll_slp_analysis(CountedLoopNode *cl, PhaseIdealLo
// Enable this functionality target by target as needed
if (SuperWordLoopUnrollAnalysis) {
if (!cl->was_slp_analyzed()) {
- SuperWord sw(phase);
- sw.transform_loop(this, false);
+ Compile::TracePhase tp("autoVectorize", &Phase::timers[Phase::_t_autoVectorize]);
- // If the loop is slp canonical analyze it
- if (sw.early_return() == false) {
- sw.unrolling_analysis(_local_loop_unroll_factor);
+ VLoop vloop(this, true);
+ if (vloop.check_preconditions()) {
+ SuperWord::unrolling_analysis(vloop, _local_loop_unroll_factor);
}
}
diff --git a/src/hotspot/share/opto/loopnode.cpp b/src/hotspot/share/opto/loopnode.cpp
index ac822453356..fdfbab09b8f 100644
--- a/src/hotspot/share/opto/loopnode.cpp
+++ b/src/hotspot/share/opto/loopnode.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -45,7 +45,7 @@
#include "opto/predicates.hpp"
#include "opto/rootnode.hpp"
#include "opto/runtime.hpp"
-#include "opto/superword.hpp"
+#include "opto/vectorization.hpp"
#include "runtime/sharedRuntime.hpp"
#include "utilities/checkedCast.hpp"
#include "utilities/powerOfTwo.hpp"
@@ -4359,8 +4359,8 @@ void PhaseIdealLoop::collect_useful_template_assertion_predicates_for_loop(Ideal
}
void PhaseIdealLoop::eliminate_useless_template_assertion_predicates(Unique_Node_List& useful_predicates) {
- for (int i = 0; i < C->template_assertion_predicate_count(); i++) {
- Node* opaque4 = C->template_assertion_predicate_opaq_node(i);
+ for (int i = C->template_assertion_predicate_count(); i > 0; i--) {
+ Node* opaque4 = C->template_assertion_predicate_opaq_node(i - 1);
assert(opaque4->Opcode() == Op_Opaque4, "must be");
if (!useful_predicates.member(opaque4)) { // not in the useful list
_igvn.replace_node(opaque4, opaque4->in(2));
@@ -4863,29 +4863,30 @@ void PhaseIdealLoop::build_and_optimize() {
C->set_major_progress();
}
- // Convert scalar to superword operations at the end of all loop opts.
+ // Auto-vectorize main-loop
if (C->do_superword() && C->has_loops() && !C->major_progress()) {
- // SuperWord transform
- SuperWord sw(this);
+ Compile::TracePhase tp("autoVectorize", &timers[_t_autoVectorize]);
+
+ // Shared data structures for all AutoVectorizations, to reduce allocations
+ // of large arrays.
+ VSharedData vshared;
for (LoopTreeIterator iter(_ltree_root); !iter.done(); iter.next()) {
IdealLoopTree* lpt = iter.current();
- if (lpt->is_counted()) {
- CountedLoopNode *cl = lpt->_head->as_CountedLoop();
- if (cl->is_main_loop()) {
- if (!sw.transform_loop(lpt, true)) {
- // Instigate more unrolling for optimization when vectorization fails.
- if (cl->has_passed_slp()) {
- C->set_major_progress();
- cl->set_notpassed_slp();
- cl->mark_do_unroll_only();
- }
- }
+ AutoVectorizeStatus status = auto_vectorize(lpt, vshared);
+
+ if (status == AutoVectorizeStatus::TriedAndFailed) {
+ // We tried vectorization, but failed. From now on only unroll the loop.
+ CountedLoopNode* cl = lpt->_head->as_CountedLoop();
+ if (cl->has_passed_slp()) {
+ C->set_major_progress();
+ cl->set_notpassed_slp();
+ cl->mark_do_unroll_only();
}
}
}
}
- // Move UnorderedReduction out of counted loop. Can be introduced by SuperWord.
+ // Move UnorderedReduction out of counted loop. Can be introduced by AutoVectorization.
if (C->has_loops() && !C->major_progress()) {
for (LoopTreeIterator iter(_ltree_root); !iter.done(); iter.next()) {
IdealLoopTree* lpt = iter.current();
@@ -5070,7 +5071,7 @@ bool PhaseIdealLoop::verify_loop_ctrl(Node* n, const PhaseIdealLoop* phase_verif
}
}
-int compare_tree(IdealLoopTree* const& a, IdealLoopTree* const& b) {
+static int compare_tree(IdealLoopTree* const& a, IdealLoopTree* const& b) {
assert(a != nullptr && b != nullptr, "must be");
return a->_head->_idx - b->_head->_idx;
}
@@ -5962,30 +5963,6 @@ CountedLoopEndNode* CountedLoopNode::find_pre_loop_end() {
return pre_end;
}
- CountedLoopNode* CountedLoopNode::pre_loop_head() const {
- assert(is_main_loop(), "Only main loop has pre loop");
- assert(_pre_loop_end != nullptr && _pre_loop_end->loopnode() != nullptr,
- "should find head from pre loop end");
- return _pre_loop_end->loopnode();
- }
-
- CountedLoopEndNode* CountedLoopNode::pre_loop_end() {
-#ifdef ASSERT
- assert(is_main_loop(), "Only main loop has pre loop");
- assert(_pre_loop_end != nullptr, "should be set when fetched");
- Node* found_pre_end = find_pre_loop_end();
- assert(_pre_loop_end == found_pre_end && _pre_loop_end == pre_loop_head()->loopexit(),
- "should find the pre loop end and must be the same result");
-#endif
- return _pre_loop_end;
- }
-
- void CountedLoopNode::set_pre_loop_end(CountedLoopEndNode* pre_loop_end) {
- assert(is_main_loop(), "Only main loop has pre loop");
- assert(pre_loop_end, "must be valid");
- _pre_loop_end = pre_loop_end;
- }
-
//------------------------------get_late_ctrl----------------------------------
// Compute latest legal control.
Node *PhaseIdealLoop::get_late_ctrl( Node *n, Node *early ) {
diff --git a/src/hotspot/share/opto/loopnode.hpp b/src/hotspot/share/opto/loopnode.hpp
index 51a67674f33..b1a0d95ddf2 100644
--- a/src/hotspot/share/opto/loopnode.hpp
+++ b/src/hotspot/share/opto/loopnode.hpp
@@ -43,6 +43,7 @@ class PredicateBlock;
class PathFrequency;
class PhaseIdealLoop;
class VectorSet;
+class VSharedData;
class Invariance;
struct small_cache;
@@ -231,14 +232,11 @@ class CountedLoopNode : public BaseCountedLoopNode {
// vector mapped unroll factor here
int _slp_maximum_unroll_factor;
- // Cached CountedLoopEndNode of pre loop for main loops
- CountedLoopEndNode* _pre_loop_end;
-
public:
CountedLoopNode(Node *entry, Node *backedge)
: BaseCountedLoopNode(entry, backedge), _main_idx(0), _trip_count(max_juint),
_unrolled_count_log2(0), _node_count_before_unroll(0),
- _slp_maximum_unroll_factor(0), _pre_loop_end(nullptr) {
+ _slp_maximum_unroll_factor(0) {
init_class_id(Class_CountedLoop);
// Initialize _trip_count to the largest possible value.
// Will be reset (lower) if the loop's trip count is known.
@@ -330,9 +328,6 @@ class CountedLoopNode : public BaseCountedLoopNode {
Node* is_canonical_loop_entry();
CountedLoopEndNode* find_pre_loop_end();
- CountedLoopNode* pre_loop_head() const;
- CountedLoopEndNode* pre_loop_end();
- void set_pre_loop_end(CountedLoopEndNode* pre_loop_end);
#ifndef PRODUCT
virtual void dump_spec(outputStream *st) const;
@@ -1103,6 +1098,7 @@ class PhaseIdealLoop : public PhaseTransform {
// Compute the Ideal Node to Loop mapping
PhaseIdealLoop(PhaseIterGVN& igvn, LoopOptsMode mode) :
PhaseTransform(Ideal_Loop),
+ _loop_or_ctrl(igvn.C->comp_arena()),
_igvn(igvn),
_verify_me(nullptr),
_verify_only(false),
@@ -1117,6 +1113,7 @@ class PhaseIdealLoop : public PhaseTransform {
// or only verify that the graph is valid if verify_me is null.
PhaseIdealLoop(PhaseIterGVN& igvn, const PhaseIdealLoop* verify_me = nullptr) :
PhaseTransform(Ideal_Loop),
+ _loop_or_ctrl(igvn.C->comp_arena()),
_igvn(igvn),
_verify_me(verify_me),
_verify_only(verify_me == nullptr),
@@ -1437,6 +1434,14 @@ class PhaseIdealLoop : public PhaseTransform {
bool partial_peel( IdealLoopTree *loop, Node_List &old_new );
bool duplicate_loop_backedge(IdealLoopTree *loop, Node_List &old_new);
+ // AutoVectorize the loop: replace scalar ops with vector ops.
+ enum AutoVectorizeStatus {
+ Impossible, // This loop has the wrong shape to even try vectorization.
+ Success, // We just successfully vectorized the loop.
+ TriedAndFailed, // We tried to vectorize, but failed.
+ };
+ AutoVectorizeStatus auto_vectorize(IdealLoopTree* lpt, VSharedData &vshared);
+
// Move UnorderedReduction out of loop if possible
void move_unordered_reduction_out_of_loop(IdealLoopTree* loop);
diff --git a/src/hotspot/share/opto/loopopts.cpp b/src/hotspot/share/opto/loopopts.cpp
index 80a3d1a072b..c5d8ed39d9d 100644
--- a/src/hotspot/share/opto/loopopts.cpp
+++ b/src/hotspot/share/opto/loopopts.cpp
@@ -41,6 +41,7 @@
#include "opto/rootnode.hpp"
#include "opto/subnode.hpp"
#include "opto/subtypenode.hpp"
+#include "opto/superword.hpp"
#include "opto/vectornode.hpp"
#include "utilities/macros.hpp"
@@ -4209,6 +4210,36 @@ bool PhaseIdealLoop::duplicate_loop_backedge(IdealLoopTree *loop, Node_List &old
return true;
}
+// AutoVectorize the loop: replace scalar ops with vector ops.
+PhaseIdealLoop::AutoVectorizeStatus
+PhaseIdealLoop::auto_vectorize(IdealLoopTree* lpt, VSharedData &vshared) {
+ // Counted loop only
+ if (!lpt->is_counted()) {
+ return AutoVectorizeStatus::Impossible;
+ }
+
+ // Main-loop only
+ CountedLoopNode* cl = lpt->_head->as_CountedLoop();
+ if (!cl->is_main_loop()) {
+ return AutoVectorizeStatus::Impossible;
+ }
+
+ VLoop vloop(lpt, false);
+ if (!vloop.check_preconditions()) {
+ return AutoVectorizeStatus::TriedAndFailed;
+ }
+
+ // Ensure the shared data is cleared before each use
+ vshared.clear();
+
+ SuperWord sw(vloop, vshared);
+ if (!sw.transform_loop()) {
+ return AutoVectorizeStatus::TriedAndFailed;
+ }
+
+ return AutoVectorizeStatus::Success;
+}
+
// Having ReductionNodes in the loop is expensive. They need to recursively
// fold together the vector values, for every vectorized loop iteration. If
// we encounter the following pattern, we can vector accumulate the values
diff --git a/src/hotspot/share/opto/macro.cpp b/src/hotspot/share/opto/macro.cpp
index 960206137f6..c2aa017197e 100644
--- a/src/hotspot/share/opto/macro.cpp
+++ b/src/hotspot/share/opto/macro.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2005, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2005, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -167,7 +167,8 @@ void PhaseMacroExpand::eliminate_gc_barrier(Node* p2x) {
// Search for a memory operation for the specified memory slice.
static Node *scan_mem_chain(Node *mem, int alias_idx, int offset, Node *start_mem, Node *alloc, PhaseGVN *phase) {
Node *orig_mem = mem;
- Node *alloc_mem = alloc->in(TypeFunc::Memory);
+ Node *alloc_mem = alloc->as_Allocate()->proj_out_or_null(TypeFunc::Memory, /*io_use:*/false);
+ assert(alloc_mem != nullptr, "Allocation without a memory projection.");
const TypeOopPtr *tinst = phase->C->get_adr_type(alias_idx)->isa_oopptr();
while (true) {
if (mem == alloc_mem || mem == start_mem ) {
@@ -371,7 +372,8 @@ Node *PhaseMacroExpand::value_from_mem_phi(Node *mem, BasicType ft, const Type *
return nullptr; // Give up: phi tree too deep
}
Node *start_mem = C->start()->proj_out_or_null(TypeFunc::Memory);
- Node *alloc_mem = alloc->in(TypeFunc::Memory);
+ Node *alloc_mem = alloc->proj_out_or_null(TypeFunc::Memory, /*io_use:*/false);
+ assert(alloc_mem != nullptr, "Allocation without a memory projection.");
uint length = mem->req();
GrowableArray values(length, length, nullptr);
@@ -456,7 +458,8 @@ Node *PhaseMacroExpand::value_from_mem(Node *sfpt_mem, Node *sfpt_ctl, BasicType
int offset = adr_t->offset();
Node *start_mem = C->start()->proj_out_or_null(TypeFunc::Memory);
Node *alloc_ctrl = alloc->in(TypeFunc::Control);
- Node *alloc_mem = alloc->in(TypeFunc::Memory);
+ Node *alloc_mem = alloc->proj_out_or_null(TypeFunc::Memory, /*io_use:*/false);
+ assert(alloc_mem != nullptr, "Allocation without a memory projection.");
VectorSet visited;
bool done = sfpt_mem == alloc_mem;
@@ -766,7 +769,7 @@ SafePointScalarObjectNode* PhaseMacroExpand::create_scalarized_object_descriptio
}
}
- SafePointScalarObjectNode* sobj = new SafePointScalarObjectNode(res_type, alloc, first_ind, nfields);
+ SafePointScalarObjectNode* sobj = new SafePointScalarObjectNode(res_type, alloc, first_ind, sfpt->jvms()->depth(), nfields);
sobj->init_req(0, C->root());
transform_later(sobj);
@@ -2430,6 +2433,9 @@ void PhaseMacroExpand::eliminate_macro_nodes() {
//------------------------------expand_macro_nodes----------------------
// Returns true if a failure occurred.
bool PhaseMacroExpand::expand_macro_nodes() {
+ if (StressMacroExpansion) {
+ C->shuffle_macro_nodes();
+ }
// Last attempt to eliminate macro nodes.
eliminate_macro_nodes();
if (C->failing()) return true;
@@ -2511,6 +2517,9 @@ bool PhaseMacroExpand::expand_macro_nodes() {
}
assert(!success || (C->macro_count() == (old_macro_count - 1)), "elimination must have deleted one node from macro list");
progress = progress || success;
+ if (success) {
+ C->print_method(PHASE_AFTER_MACRO_EXPANSION_STEP, 5, n);
+ }
}
}
@@ -2571,6 +2580,7 @@ bool PhaseMacroExpand::expand_macro_nodes() {
}
assert(C->macro_count() == (old_macro_count - 1), "expansion must have deleted one node from macro list");
if (C->failing()) return true;
+ C->print_method(PHASE_AFTER_MACRO_EXPANSION_STEP, 5, n);
// Clean up the graph so we're less likely to hit the maximum node
// limit
@@ -2613,6 +2623,7 @@ bool PhaseMacroExpand::expand_macro_nodes() {
}
assert(C->macro_count() < macro_count, "must have deleted a node from macro list");
if (C->failing()) return true;
+ C->print_method(PHASE_AFTER_MACRO_EXPANSION_STEP, 5, n);
// Clean up the graph so we're less likely to hit the maximum node
// limit
diff --git a/src/hotspot/share/opto/matcher.hpp b/src/hotspot/share/opto/matcher.hpp
index a4a20975e88..84e48086f92 100644
--- a/src/hotspot/share/opto/matcher.hpp
+++ b/src/hotspot/share/opto/matcher.hpp
@@ -331,7 +331,7 @@ class Matcher : public PhaseTransform {
// Identify extra cases that we might want to vectorize automatically
// And exclude cases which are not profitable to auto-vectorize.
- static bool match_rule_supported_superword(int opcode, int vlen, BasicType bt);
+ static bool match_rule_supported_auto_vectorization(int opcode, int vlen, BasicType bt);
// identify extra cases that we might want to provide match rules for
// e.g. Op_ vector nodes and other intrinsics while guarding with vlen
@@ -355,7 +355,7 @@ class Matcher : public PhaseTransform {
Matcher::min_vector_size(bt) <= size);
}
// Limits on max vector size (number of elements) for auto-vectorization.
- static int superword_max_vector_size(const BasicType bt);
+ static int max_vector_size_auto_vectorization(const BasicType bt);
// Actual max scalable vector register length.
static int scalable_vector_reg_size(const BasicType bt);
diff --git a/src/hotspot/share/opto/node.cpp b/src/hotspot/share/opto/node.cpp
index 44c37768cdc..dee4ce80d35 100644
--- a/src/hotspot/share/opto/node.cpp
+++ b/src/hotspot/share/opto/node.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1632,7 +1632,7 @@ void visit_nodes(Node* start, Callback callback, bool traverse_output, bool only
}
// BFS traverse from start, return node with idx
-Node* find_node_by_idx(Node* start, uint idx, bool traverse_output, bool only_ctrl) {
+static Node* find_node_by_idx(Node* start, uint idx, bool traverse_output, bool only_ctrl) {
ResourceMark rm;
Node* result = nullptr;
auto callback = [&] (Node* n) {
@@ -1648,11 +1648,11 @@ Node* find_node_by_idx(Node* start, uint idx, bool traverse_output, bool only_ct
return result;
}
-int node_idx_cmp(const Node** n1, const Node** n2) {
+static int node_idx_cmp(const Node** n1, const Node** n2) {
return (*n1)->_idx - (*n2)->_idx;
}
-void find_nodes_by_name(Node* start, const char* name) {
+static void find_nodes_by_name(Node* start, const char* name) {
ResourceMark rm;
GrowableArray ns;
auto callback = [&] (const Node* n) {
@@ -1667,7 +1667,7 @@ void find_nodes_by_name(Node* start, const char* name) {
}
}
-void find_nodes_by_dump(Node* start, const char* pattern) {
+static void find_nodes_by_dump(Node* start, const char* pattern) {
ResourceMark rm;
GrowableArray ns;
auto callback = [&] (const Node* n) {
diff --git a/src/hotspot/share/opto/output.cpp b/src/hotspot/share/opto/output.cpp
index 9481b91ce39..640a24693de 100644
--- a/src/hotspot/share/opto/output.cpp
+++ b/src/hotspot/share/opto/output.cpp
@@ -507,8 +507,8 @@ void PhaseOutput::shorten_branches(uint* blk_starts) {
mcall->method_set((intptr_t)mcall->entry_point());
if (mcall->is_MachCallJava() && mcall->as_MachCallJava()->_method) {
- stub_size += CompiledStaticCall::to_interp_stub_size();
- reloc_size += CompiledStaticCall::reloc_to_interp_stub();
+ stub_size += CompiledDirectCall::to_interp_stub_size();
+ reloc_size += CompiledDirectCall::reloc_to_interp_stub();
}
} else if (mach->is_MachSafePoint()) {
// If call/safepoint are adjacent, account for possible
@@ -3412,6 +3412,12 @@ void PhaseOutput::install_code(ciMethod* target,
_code_offsets.set_value(CodeOffsets::Verified_Entry, 0);
_code_offsets.set_value(CodeOffsets::OSR_Entry, _first_block_size);
} else {
+ if (!target->is_static()) {
+ // The UEP of an nmethod ensures that the VEP is padded. However, the padding of the UEP is placed
+ // before the inline cache check, so we don't have to execute any nop instructions when dispatching
+ // through the UEP, yet we can ensure that the VEP is aligned appropriately.
+ _code_offsets.set_value(CodeOffsets::Entry, _first_block_size - MacroAssembler::ic_check_size());
+ }
_code_offsets.set_value(CodeOffsets::Verified_Entry, _first_block_size);
_code_offsets.set_value(CodeOffsets::OSR_Entry, 0);
}
diff --git a/src/hotspot/share/opto/parse.hpp b/src/hotspot/share/opto/parse.hpp
index 40314ecc29d..d5f12c8bc81 100644
--- a/src/hotspot/share/opto/parse.hpp
+++ b/src/hotspot/share/opto/parse.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -564,7 +564,6 @@ class Parse : public GraphKit {
float branch_prediction(float &cnt, BoolTest::mask btest, int target_bci, Node* test);
bool seems_never_taken(float prob) const;
bool path_is_suitable_for_uncommon_trap(float prob) const;
- bool seems_stable_comparison() const;
void do_ifnull(BoolTest::mask btest, Node* c);
void do_if(BoolTest::mask btest, Node* c);
diff --git a/src/hotspot/share/opto/parse1.cpp b/src/hotspot/share/opto/parse1.cpp
index 8ee99ae9a65..7c0f5cadeeb 100644
--- a/src/hotspot/share/opto/parse1.cpp
+++ b/src/hotspot/share/opto/parse1.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -224,7 +224,10 @@ void Parse::load_interpreter_state(Node* osr_buf) {
Node *monitors_addr = basic_plus_adr(osr_buf, osr_buf, (max_locals+mcnt*2-1)*wordSize);
for (index = 0; index < mcnt; index++) {
// Make a BoxLockNode for the monitor.
- Node *box = _gvn.transform(new BoxLockNode(next_monitor()));
+ Node *box = new BoxLockNode(next_monitor());
+ // Check for bailout after new BoxLockNode
+ if (failing()) { return; }
+ box = _gvn.transform(box);
// Displaced headers and locked objects are interleaved in the
@@ -1260,6 +1263,8 @@ void Parse::do_method_entry() {
kill_dead_locals();
// Build the FastLockNode
_synch_lock = shared_lock(lock_obj);
+ // Check for bailout in shared_lock
+ if (failing()) { return; }
}
// Feed profiling data for parameters to the type system so it can
diff --git a/src/hotspot/share/opto/parse2.cpp b/src/hotspot/share/opto/parse2.cpp
index 0c22409aeee..d43b8335ffd 100644
--- a/src/hotspot/share/opto/parse2.cpp
+++ b/src/hotspot/share/opto/parse2.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -1354,21 +1354,6 @@ bool Parse::seems_never_taken(float prob) const {
return prob < PROB_MIN;
}
-// True if the comparison seems to be the kind that will not change its
-// statistics from true to false. See comments in adjust_map_after_if.
-// This question is only asked along paths which are already
-// classified as untaken (by seems_never_taken), so really,
-// if a path is never taken, its controlling comparison is
-// already acting in a stable fashion. If the comparison
-// seems stable, we will put an expensive uncommon trap
-// on the untaken path.
-bool Parse::seems_stable_comparison() const {
- if (C->too_many_traps(method(), bci(), Deoptimization::Reason_unstable_if)) {
- return false;
- }
- return true;
-}
-
//-------------------------------repush_if_args--------------------------------
// Push arguments of an "if" bytecode back onto the stack by adjusting _sp.
inline int Parse::repush_if_args() {
@@ -1572,7 +1557,8 @@ bool Parse::path_is_suitable_for_uncommon_trap(float prob) const {
if (!UseInterpreter) {
return false;
}
- return (seems_never_taken(prob) && seems_stable_comparison());
+ return seems_never_taken(prob) &&
+ !C->too_many_traps(method(), bci(), Deoptimization::Reason_unstable_if);
}
void Parse::maybe_add_predicate_after_if(Block* path) {
@@ -2778,6 +2764,7 @@ void Parse::do_one_bytecode() {
}
#ifndef PRODUCT
+ if (failing()) { return; }
constexpr int perBytecode = 6;
if (C->should_print_igv(perBytecode)) {
IdealGraphPrinter* printer = C->igv_printer();
diff --git a/src/hotspot/share/opto/phase.cpp b/src/hotspot/share/opto/phase.cpp
index 90b9a7d527c..d3e34a4dc69 100644
--- a/src/hotspot/share/opto/phase.cpp
+++ b/src/hotspot/share/opto/phase.cpp
@@ -81,6 +81,7 @@ void Phase::print_timers() {
tty->print_cr (" Prune Useless: %7.3f s", timers[_t_vector_pru].seconds());
tty->print_cr (" Renumber Live: %7.3f s", timers[_t_renumberLive].seconds());
tty->print_cr (" IdealLoop: %7.3f s", timers[_t_idealLoop].seconds());
+ tty->print_cr (" AutoVectorize: %7.3f s", timers[_t_autoVectorize].seconds());
tty->print_cr (" IdealLoop Verify: %7.3f s", timers[_t_idealLoopVerify].seconds());
tty->print_cr (" Cond Const Prop: %7.3f s", timers[_t_ccp].seconds());
tty->print_cr (" GVN 2: %7.3f s", timers[_t_iterGVN2].seconds());
diff --git a/src/hotspot/share/opto/phase.hpp b/src/hotspot/share/opto/phase.hpp
index 7e100e9b0a5..db2327f378f 100644
--- a/src/hotspot/share/opto/phase.hpp
+++ b/src/hotspot/share/opto/phase.hpp
@@ -82,6 +82,7 @@ class Phase : public StackObj {
_t_vector_pru,
_t_renumberLive,
_t_idealLoop,
+ _t_autoVectorize,
_t_idealLoopVerify,
_t_ccp,
_t_iterGVN2,
diff --git a/src/hotspot/share/opto/phasetype.hpp b/src/hotspot/share/opto/phasetype.hpp
index 43a6491fee5..6448b8331cc 100644
--- a/src/hotspot/share/opto/phasetype.hpp
+++ b/src/hotspot/share/opto/phasetype.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -26,6 +26,7 @@
#define SHARE_OPTO_PHASETYPE_HPP
#include "utilities/bitMap.inline.hpp"
+#include "utilities/stringUtils.hpp"
#define COMPILER_PHASES(flags) \
flags(BEFORE_STRINGOPTS, "Before StringOpts") \
@@ -83,7 +84,9 @@
flags(CCP1, "PhaseCCP 1") \
flags(ITER_GVN2, "Iter GVN 2") \
flags(PHASEIDEALLOOP_ITERATIONS, "PhaseIdealLoop iterations") \
- flags(MACRO_EXPANSION, "Macro expand") \
+ flags(BEFORE_MACRO_EXPANSION , "Before Macro Expansion") \
+ flags(AFTER_MACRO_EXPANSION_STEP, "After Macro Expansion Step") \
+ flags(AFTER_MACRO_EXPANSION, "After Macro Expansion") \
flags(BARRIER_EXPANSION, "Barrier expand") \
flags(OPTIMIZE_FINISHED, "Optimize finished") \
flags(BEFORE_MATCHING, "Before matching") \
@@ -139,47 +142,6 @@ static CompilerPhaseType find_phase(const char* str) {
return PHASE_NONE;
}
-class PhaseNameIter {
- private:
- char* _token;
- char* _saved_ptr;
- char* _list;
-
- public:
- PhaseNameIter(ccstrlist option) {
- _list = (char*) canonicalize(option);
- _saved_ptr = _list;
- _token = strtok_r(_saved_ptr, ",", &_saved_ptr);
- }
-
- ~PhaseNameIter() {
- FREE_C_HEAP_ARRAY(char, _list);
- }
-
- const char* operator*() const { return _token; }
-
- PhaseNameIter& operator++() {
- _token = strtok_r(nullptr, ",", &_saved_ptr);
- return *this;
- }
-
- ccstrlist canonicalize(ccstrlist option_value) {
- char* canonicalized_list = NEW_C_HEAP_ARRAY(char, strlen(option_value) + 1, mtCompiler);
- int i = 0;
- char current;
- while ((current = option_value[i]) != '\0') {
- if (current == '\n' || current == ' ') {
- canonicalized_list[i] = ',';
- } else {
- canonicalized_list[i] = current;
- }
- i++;
- }
- canonicalized_list[i] = '\0';
- return canonicalized_list;
- }
-};
-
class PhaseNameValidator {
private:
CHeapBitMap _phase_name_set;
@@ -192,7 +154,7 @@ class PhaseNameValidator {
_valid(true),
_bad(nullptr)
{
- for (PhaseNameIter iter(option); *iter != nullptr && _valid; ++iter) {
+ for (StringUtils::CommaSeparatedStringIterator iter(option); *iter != nullptr && _valid; ++iter) {
CompilerPhaseType cpt = find_phase(*iter);
if (PHASE_NONE == cpt) {
diff --git a/src/hotspot/share/opto/reg_split.cpp b/src/hotspot/share/opto/reg_split.cpp
index 8e5247e93b6..35b2618c64c 100644
--- a/src/hotspot/share/opto/reg_split.cpp
+++ b/src/hotspot/share/opto/reg_split.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2000, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -289,7 +289,7 @@ int PhaseChaitin::split_USE(MachSpillCopyNode::SpillType spill_type, Node *def,
//------------------------------clone_node----------------------------
// Clone node with anti dependence check.
-Node* clone_node(Node* def, Block *b, Compile* C) {
+static Node* clone_node(Node* def, Block *b, Compile* C) {
if (def->needs_anti_dependence_check()) {
#ifdef ASSERT
if (PrintOpto && WizardMode) {
diff --git a/src/hotspot/share/opto/regmask.hpp b/src/hotspot/share/opto/regmask.hpp
index 75b3b6aa5a2..f77c14e50ea 100644
--- a/src/hotspot/share/opto/regmask.hpp
+++ b/src/hotspot/share/opto/regmask.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2022, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -29,6 +29,7 @@
#include "opto/optoreg.hpp"
#include "utilities/count_leading_zeros.hpp"
#include "utilities/count_trailing_zeros.hpp"
+#include "utilities/globalDefinitions.hpp"
class LRG;
@@ -359,16 +360,15 @@ class RegMask {
static const RegMask Empty; // Common empty mask
static const RegMask All; // Common all mask
- static bool can_represent(OptoReg::Name reg) {
- // NOTE: -1 in computation reflects the usage of the last
- // bit of the regmask as an infinite stack flag and
- // -7 is to keep mask aligned for largest value (VecZ).
- return (int)reg < (int)(CHUNK_SIZE - 1);
+ static bool can_represent(OptoReg::Name reg, unsigned int size = 1) {
+ // NOTE: MAX2(1U,size) in computation reflects the usage of the last
+ // bit of the regmask as an infinite stack flag.
+ return (int)reg < (int)(CHUNK_SIZE - MAX2(1U,size));
}
static bool can_represent_arg(OptoReg::Name reg) {
- // NOTE: -SlotsPerVecZ in computation reflects the need
+ // NOTE: SlotsPerVecZ in computation reflects the need
// to keep mask aligned for largest value (VecZ).
- return (int)reg < (int)(CHUNK_SIZE - SlotsPerVecZ);
+ return can_represent(reg, SlotsPerVecZ);
}
};
diff --git a/src/hotspot/share/opto/runtime.cpp b/src/hotspot/share/opto/runtime.cpp
index 8f3367ce000..c5f22e8ebbb 100644
--- a/src/hotspot/share/opto/runtime.cpp
+++ b/src/hotspot/share/opto/runtime.cpp
@@ -28,7 +28,6 @@
#include "code/codeCache.hpp"
#include "code/compiledMethod.inline.hpp"
#include "code/compiledIC.hpp"
-#include "code/icBuffer.hpp"
#include "code/nmethod.hpp"
#include "code/pcDesc.hpp"
#include "code/scopeDesc.hpp"
diff --git a/src/hotspot/share/opto/subnode.cpp b/src/hotspot/share/opto/subnode.cpp
index f6790172351..5ecd7b4251f 100644
--- a/src/hotspot/share/opto/subnode.cpp
+++ b/src/hotspot/share/opto/subnode.cpp
@@ -155,7 +155,7 @@ static bool ok_to_convert(Node* inc, Node* var) {
static bool is_cloop_condition(BoolNode* bol) {
for (DUIterator_Fast imax, i = bol->fast_outs(imax); i < imax; i++) {
Node* out = bol->fast_out(i);
- if (out->is_CountedLoopEnd()) {
+ if (out->is_BaseCountedLoopEnd()) {
return true;
}
}
diff --git a/src/hotspot/share/opto/superword.cpp b/src/hotspot/share/opto/superword.cpp
index b1d1169f357..eb92f58024c 100644
--- a/src/hotspot/share/opto/superword.cpp
+++ b/src/hotspot/share/opto/superword.cpp
@@ -38,158 +38,50 @@
#include "opto/movenode.hpp"
#include "utilities/powerOfTwo.hpp"
-//
-// S U P E R W O R D T R A N S F O R M
-//=============================================================================
-
-//------------------------------SuperWord---------------------------
-SuperWord::SuperWord(PhaseIdealLoop* phase) :
- _phase(phase),
- _arena(phase->C->comp_arena()),
- _igvn(phase->_igvn),
+SuperWord::SuperWord(const VLoop &vloop, VSharedData &vshared) :
+ _vloop(vloop),
+ _arena(mtCompiler),
_packset(arena(), 8, 0, nullptr), // packs for the current block
- _bb_idx(arena(), (int)(1.10 * phase->C->unique()), 0, 0), // node idx to index in bb
- _block(arena(), 8, 0, nullptr), // nodes in current block
- _data_entry(arena(), 8, 0, nullptr), // nodes with all inputs from outside
+ _bb_idx(vshared.node_idx_to_loop_body_idx()), // node idx to index in bb
+ _block(arena(), vloop.estimated_body_length(), 0, nullptr), // nodes in current block
_mem_slice_head(arena(), 8, 0, nullptr), // memory slice heads
_mem_slice_tail(arena(), 8, 0, nullptr), // memory slice tails
- _node_info(arena(), 8, 0, SWNodeInfo::initial), // info needed per node
- _clone_map(phase->C->clone_map()), // map of nodes created in cloning
+ _node_info(arena(), vloop.estimated_body_length(), 0, SWNodeInfo::initial), // info needed per node
+ _clone_map(phase()->C->clone_map()), // map of nodes created in cloning
_align_to_ref(nullptr), // memory reference to align vectors to
- _disjoint_ptrs(arena(), 8, 0, OrderedPair::initial), // runtime disambiguated pointer pairs
- _dg(_arena), // dependence graph
- _visited(arena()), // visited node set
- _post_visited(arena()), // post visited node set
- _n_idx_list(arena(), 8), // scratch list of (node,index) pairs
- _nlist(arena(), 8, 0, nullptr), // scratch list of nodes
- _stk(arena(), 8, 0, nullptr), // scratch stack of nodes
- _lpt(nullptr), // loop tree node
- _lp(nullptr), // CountedLoopNode
+ _dg(arena()), // dependence graph
+ _nlist(arena(), vloop.estimated_body_length(), 0, nullptr), // scratch list of nodes
_loop_reductions(arena()), // reduction nodes in the current loop
- _bb(nullptr), // basic block
- _iv(nullptr), // induction var
_race_possible(false), // cases where SDMU is true
- _early_return(true), // analysis evaluations routine
- _do_vector_loop(phase->C->do_vector_loop()), // whether to do vectorization/simd style
+ _do_vector_loop(phase()->C->do_vector_loop()), // whether to do vectorization/simd style
_num_work_vecs(0), // amount of vector work we have
_num_reductions(0) // amount of reduction work we have
{
-#ifndef PRODUCT
- _vector_loop_debug = 0;
- if (_phase->C->method() != nullptr) {
- _vector_loop_debug = phase->C->directive()->VectorizeDebugOption;
- }
-
-#endif
}
-//------------------------------transform_loop---------------------------
-bool SuperWord::transform_loop(IdealLoopTree* lpt, bool do_optimization) {
- assert(_phase->C->do_superword(), "SuperWord option should be enabled");
- // SuperWord only works with power of two vector sizes.
- int vector_width = Matcher::vector_width_in_bytes(T_BYTE);
- if (vector_width < 2 || !is_power_of_2(vector_width)) {
- return false;
- }
-
- assert(lpt->_head->is_CountedLoop(), "must be");
- CountedLoopNode *cl = lpt->_head->as_CountedLoop();
-
- if (!cl->is_valid_counted_loop(T_INT)) {
- return false; // skip malformed counted loop
- }
-
- // Initialize simple data used by reduction marking early.
- set_lpt(lpt);
- set_lp(cl);
- // For now, define one block which is the entire loop body.
- set_bb(cl);
-
- if (SuperWordReductions) {
- mark_reductions();
- }
-
- // skip any loop that has not been assigned max unroll by analysis
- if (do_optimization) {
- if (SuperWordLoopUnrollAnalysis && cl->slp_max_unroll() == 0) {
- return false;
- }
- }
-
- // Check for no control flow in body (other than exit)
- Node *cl_exit = cl->loopexit();
- if (cl->is_main_loop() && (cl_exit->in(0) != lpt->_head)) {
- #ifndef PRODUCT
- if (TraceSuperWord) {
- tty->print_cr("SuperWord::transform_loop: loop too complicated, cl_exit->in(0) != lpt->_head");
- tty->print("cl_exit %d", cl_exit->_idx); cl_exit->dump();
- tty->print("cl_exit->in(0) %d", cl_exit->in(0)->_idx); cl_exit->in(0)->dump();
- tty->print("lpt->_head %d", lpt->_head->_idx); lpt->_head->dump();
- lpt->dump_head();
- }
- #endif
- return false;
- }
-
- // Make sure the are no extra control users of the loop backedge
- if (cl->back_control()->outcnt() != 1) {
- return false;
- }
-
- // Skip any loops already optimized by slp
- if (cl->is_vectorized_loop()) {
- return false;
- }
-
- if (cl->is_unroll_only()) {
- return false;
- }
-
- if (cl->is_main_loop()) {
- // Check for pre-loop ending with CountedLoopEnd(Bool(Cmp(x,Opaque1(limit))))
- CountedLoopEndNode* pre_end = cl->find_pre_loop_end();
- if (pre_end == nullptr) {
- return false;
- }
- Node* pre_opaq1 = pre_end->limit();
- if (pre_opaq1->Opcode() != Op_Opaque1) {
- return false;
- }
- cl->set_pre_loop_end(pre_end);
- }
-
- init(); // initialize data structures
-
- bool success = true;
- if (do_optimization) {
- assert(_packset.length() == 0, "packset must be empty");
- success = SLP_extract();
- }
- return success;
-}
+void SuperWord::unrolling_analysis(const VLoop &vloop, int &local_loop_unroll_factor) {
+ IdealLoopTree* lpt = vloop.lpt();
+ CountedLoopNode* cl = vloop.cl();
+ Node* cl_exit = vloop.cl_exit();
+ PhaseIdealLoop* phase = vloop.phase();
-//------------------------------early unrolling analysis------------------------------
-void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
bool is_slp = true;
- size_t ignored_size = lpt()->_body.size();
+ size_t ignored_size = lpt->_body.size();
int *ignored_loop_nodes = NEW_RESOURCE_ARRAY(int, ignored_size);
Node_Stack nstack((int)ignored_size);
- CountedLoopNode *cl = lpt()->_head->as_CountedLoop();
- Node *cl_exit = cl->loopexit_or_null();
// First clear the entries
- for (uint i = 0; i < lpt()->_body.size(); i++) {
+ for (uint i = 0; i < lpt->_body.size(); i++) {
ignored_loop_nodes[i] = -1;
}
- int max_vector = Matcher::superword_max_vector_size(T_BYTE);
+ int max_vector = Matcher::max_vector_size_auto_vectorization(T_BYTE);
// Process the loop, some/all of the stack entries will not be in order, ergo
// need to preprocess the ignored initial state before we process the loop
- for (uint i = 0; i < lpt()->_body.size(); i++) {
- Node* n = lpt()->_body.at(i);
+ for (uint i = 0; i < lpt->_body.size(); i++) {
+ Node* n = lpt->_body.at(i);
if (n == cl->incr() ||
- is_marked_reduction(n) ||
n->is_AddP() ||
n->is_Cmp() ||
n->is_Bool() ||
@@ -203,7 +95,7 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
if (n->is_If()) {
IfNode *iff = n->as_If();
if (iff->_fcnt != COUNT_UNKNOWN && iff->_prob != PROB_UNKNOWN) {
- if (lpt()->is_loop_exit(iff)) {
+ if (lpt->is_loop_exit(iff)) {
ignored_loop_nodes[i] = n->_idx;
continue;
}
@@ -247,10 +139,10 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
if (n->is_Mem()) {
MemNode* current = n->as_Mem();
Node* adr = n->in(MemNode::Address);
- Node* n_ctrl = _phase->get_ctrl(adr);
+ Node* n_ctrl = phase->get_ctrl(adr);
// save a queue of post process nodes
- if (n_ctrl != nullptr && lpt()->is_member(_phase->get_loop(n_ctrl))) {
+ if (n_ctrl != nullptr && lpt->is_member(phase->get_loop(n_ctrl))) {
// Process the memory expression
int stack_idx = 0;
bool have_side_effects = true;
@@ -258,15 +150,15 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
nstack.push(adr, stack_idx++);
} else {
// Mark the components of the memory operation in nstack
- VPointer p1(current, phase(), lpt(), &nstack, true);
+ VPointer p1(current, vloop, &nstack);
have_side_effects = p1.node_stack()->is_nonempty();
}
// Process the pointer stack
while (have_side_effects) {
Node* pointer_node = nstack.node();
- for (uint j = 0; j < lpt()->_body.size(); j++) {
- Node* cur_node = lpt()->_body.at(j);
+ for (uint j = 0; j < lpt->_body.size(); j++) {
+ Node* cur_node = lpt->_body.at(j);
if (cur_node == pointer_node) {
ignored_loop_nodes[j] = cur_node->_idx;
break;
@@ -283,11 +175,11 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
// Now we try to find the maximum supported consistent vector which the machine
// description can use
bool flag_small_bt = false;
- for (uint i = 0; i < lpt()->_body.size(); i++) {
+ for (uint i = 0; i < lpt->_body.size(); i++) {
if (ignored_loop_nodes[i] != -1) continue;
BasicType bt;
- Node* n = lpt()->_body.at(i);
+ Node* n = lpt->_body.at(i);
if (n->is_Mem()) {
bt = n->as_Mem()->memory_type();
} else {
@@ -296,15 +188,17 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
if (is_java_primitive(bt) == false) continue;
- int cur_max_vector = Matcher::superword_max_vector_size(bt);
+ int cur_max_vector = Matcher::max_vector_size_auto_vectorization(bt);
// If a max vector exists which is not larger than _local_loop_unroll_factor
// stop looking, we already have the max vector to map to.
if (cur_max_vector < local_loop_unroll_factor) {
is_slp = false;
+#ifndef PRODUCT
if (TraceSuperWordLoopUnrollAnalysis) {
tty->print_cr("slp analysis fails: unroll limit greater than max vector\n");
}
+#endif
break;
}
@@ -325,11 +219,11 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
for (uint j = start; j < end; j++) {
Node* in = n->in(j);
// Don't propagate through a memory
- if (!in->is_Mem() && in_bb(in) && in->bottom_type()->basic_type() == T_INT) {
+ if (!in->is_Mem() && vloop.in_bb(in) && in->bottom_type()->basic_type() == T_INT) {
bool same_type = true;
for (DUIterator_Fast kmax, k = in->fast_outs(kmax); k < kmax; k++) {
Node *use = in->fast_out(k);
- if (!in_bb(use) && use->bottom_type()->basic_type() != bt) {
+ if (!vloop.in_bb(use) && use->bottom_type()->basic_type() != bt) {
same_type = false;
break;
}
@@ -351,6 +245,11 @@ void SuperWord::unrolling_analysis(int &local_loop_unroll_factor) {
}
cl->mark_was_slp();
if (cl->is_main_loop()) {
+#ifndef PRODUCT
+ if (TraceSuperWordLoopUnrollAnalysis) {
+ tty->print_cr("slp analysis: set max unroll to %d", local_loop_unroll_factor);
+ }
+#endif
cl->set_slp_max_unroll(local_loop_unroll_factor);
}
}
@@ -410,8 +309,8 @@ void SuperWord::mark_reductions() {
// Iterate through all phi nodes associated to the loop and search for
// reduction cycles in the basic block.
- for (DUIterator_Fast imax, i = lp()->fast_outs(imax); i < imax; i++) {
- const Node* phi = lp()->fast_out(i);
+ for (DUIterator_Fast imax, i = cl()->fast_outs(imax); i < imax; i++) {
+ const Node* phi = cl()->fast_out(i);
if (!phi->is_Phi()) {
continue;
}
@@ -489,6 +388,44 @@ void SuperWord::mark_reductions() {
}
}
+bool SuperWord::transform_loop() {
+ assert(phase()->C->do_superword(), "SuperWord option should be enabled");
+ assert(cl()->is_main_loop(), "SLP should only work on main loops");
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nSuperWord::transform_loop:");
+ lpt()->dump_head();
+ cl()->dump();
+ }
+#endif
+
+ // Skip any loop that has not been assigned max unroll by analysis
+ if (SuperWordLoopUnrollAnalysis && vloop().cl()->slp_max_unroll() == 0) {
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nSuperWord::transform_loop failed: slp max unroll analysis was not already done");
+ }
+#endif
+ return false;
+ }
+
+ if (!SLP_extract()) {
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nSuperWord::transform_loop failed: SuperWord::SLP_extract did not vectorize");
+ }
+#endif
+ return false;
+ }
+
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nSuperWord::transform_loop: success");
+ }
+#endif
+ return true;
+}
+
//------------------------------SLP_extract---------------------------
// Extract the superword level parallelism
//
@@ -524,28 +461,41 @@ void SuperWord::mark_reductions() {
// extraction of scalar values from vectors.
//
bool SuperWord::SLP_extract() {
+ assert(cl()->is_main_loop(), "SLP should only work on main loops");
+
+ if (SuperWordReductions) {
+ mark_reductions();
+ }
+ // Find memory slices
+ find_memory_slices();
+
+ if (!is_marked_reduction_loop() &&
+ _mem_slice_head.is_empty()) {
#ifndef PRODUCT
- if (_do_vector_loop && TraceSuperWord) {
- tty->print("SuperWord::SLP_extract\n");
- tty->print("input loop\n");
- _lpt->dump_head();
- _lpt->dump();
- for (uint i = 0; i < _lpt->_body.size(); i++) {
- _lpt->_body.at(i)->dump();
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nNo reductions or memory slices found, abort SuperWord.");
+ tty->cr();
}
- }
#endif
-
- CountedLoopNode* cl = lpt()->_head->as_CountedLoop();
- assert(cl->is_main_loop(), "SLP should only work on main loops");
+ return false;
+ }
// Ready the block
if (!construct_bb()) {
- return false; // Exit if no interesting nodes or complex graph.
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("\nSuperWord::construct_bb failed: abort SuperWord");
+ tty->cr();
+ }
+#endif
+ return false;
}
- // build _dg, _disjoint_ptrs
+ // Ensure extra info is allocated.
+ initialize_node_info();
+
+ // build _dg
dependence_graph();
// compute function depth(Node*)
@@ -559,7 +509,7 @@ bool SuperWord::SLP_extract() {
if (_packset.length() == 0) {
#ifndef PRODUCT
- if (TraceSuperWord) {
+ if (is_trace_superword_any()) {
tty->print_cr("\nNo pair packs generated, abort SuperWord.");
tty->cr();
}
@@ -567,15 +517,19 @@ bool SuperWord::SLP_extract() {
return false;
}
- extend_packlist();
+ extend_packset_with_more_pairs_by_following_use_and_def();
- combine_packs();
+ combine_pairs_to_longer_packs();
- filter_packs_for_alignment();
+ split_packs_longer_than_max_vector_size();
+ // Now we only remove packs:
construct_my_pack_map();
-
- filter_packs();
+ filter_packs_for_power_of_2_size();
+ filter_packs_for_mutual_independence();
+ filter_packs_for_alignment();
+ filter_packs_for_implemented();
+ filter_packs_for_profitable();
DEBUG_ONLY(verify_packs();)
@@ -602,9 +556,11 @@ void SuperWord::find_adjacent_refs() {
}
}
}
- if (TraceSuperWord) {
+#ifndef PRODUCT
+ if (is_trace_superword_adjacent_memops()) {
tty->print_cr("\nfind_adjacent_refs found %d memops", memops.size());
}
+#endif
int max_idx;
@@ -624,13 +580,13 @@ void SuperWord::find_adjacent_refs() {
set_align_to_ref(align_to_mem_ref);
}
- VPointer align_to_ref_p(mem_ref, phase(), lpt(), nullptr, false);
+ VPointer align_to_ref_p(mem_ref, vloop());
// Set alignment relative to "align_to_ref" for all related memory operations.
for (int i = memops.size() - 1; i >= 0; i--) {
MemNode* s = memops.at(i)->as_Mem();
if (isomorphic(s, mem_ref) &&
(!_do_vector_loop || same_origin_idx(s, mem_ref))) {
- VPointer p2(s, phase(), lpt(), nullptr, false);
+ VPointer p2(s, vloop());
if (p2.comparable(align_to_ref_p)) {
int align = memory_alignment(s, iv_adjustment);
set_alignment(s, align);
@@ -672,8 +628,8 @@ void SuperWord::find_adjacent_refs() {
"packset empty or we find the alignment reference");
#ifndef PRODUCT
- if (TraceSuperWord) {
- tty->print_cr("\nAfter find_adjacent_refs");
+ if (is_trace_superword_packset()) {
+ tty->print_cr("\nAfter Superword::find_adjacent_refs");
print_packset();
}
#endif
@@ -689,11 +645,11 @@ MemNode* SuperWord::find_align_to_ref(Node_List &memops, int &idx) {
// Count number of comparable memory ops
for (uint i = 0; i < memops.size(); i++) {
MemNode* s1 = memops.at(i)->as_Mem();
- VPointer p1(s1, phase(), lpt(), nullptr, false);
+ VPointer p1(s1, vloop());
for (uint j = i+1; j < memops.size(); j++) {
MemNode* s2 = memops.at(j)->as_Mem();
if (isomorphic(s1, s2)) {
- VPointer p2(s2, phase(), lpt(), nullptr, false);
+ VPointer p2(s2, vloop());
if (p1.comparable(p2)) {
(*cmp_ct.adr_at(i))++;
(*cmp_ct.adr_at(j))++;
@@ -714,7 +670,7 @@ MemNode* SuperWord::find_align_to_ref(Node_List &memops, int &idx) {
if (s->is_Store()) {
int vw = vector_width_in_bytes(s);
assert(vw > 1, "sanity");
- VPointer p(s, phase(), lpt(), nullptr, false);
+ VPointer p(s, vloop());
if ( cmp_ct.at(j) > max_ct ||
(cmp_ct.at(j) == max_ct &&
( vw > max_vw ||
@@ -737,7 +693,7 @@ MemNode* SuperWord::find_align_to_ref(Node_List &memops, int &idx) {
if (s->is_Load()) {
int vw = vector_width_in_bytes(s);
assert(vw > 1, "sanity");
- VPointer p(s, phase(), lpt(), nullptr, false);
+ VPointer p(s, vloop());
if ( cmp_ct.at(j) > max_ct ||
(cmp_ct.at(j) == max_ct &&
( vw > max_vw ||
@@ -755,8 +711,8 @@ MemNode* SuperWord::find_align_to_ref(Node_List &memops, int &idx) {
}
}
-#ifdef ASSERT
- if (TraceSuperWord && Verbose) {
+#ifndef PRODUCT
+ if (is_trace_superword_verbose()) {
tty->print_cr("\nVector memops after find_align_to_ref");
for (uint i = 0; i < memops.size(); i++) {
MemNode* s = memops.at(i)->as_Mem();
@@ -767,9 +723,9 @@ MemNode* SuperWord::find_align_to_ref(Node_List &memops, int &idx) {
idx = max_idx;
if (max_ct > 0) {
-#ifdef ASSERT
- if (TraceSuperWord) {
- tty->print("\nVector align to node: ");
+#ifndef PRODUCT
+ if (is_trace_superword_adjacent_memops()) {
+ tty->print("SuperWord::find_align_to_ref: ");
memops.at(max_idx)->as_Mem()->dump();
}
#endif
@@ -794,13 +750,13 @@ int SuperWord::get_vw_bytes_special(MemNode* s) {
}
}
if (should_combine_adjacent) {
- vw = MIN2(Matcher::superword_max_vector_size(btype)*type2aelembytes(btype), vw * 2);
+ vw = MIN2(Matcher::max_vector_size_auto_vectorization(btype)*type2aelembytes(btype), vw * 2);
}
}
// Check for special case where there is a type conversion between different data size.
int vectsize = max_vector_size_in_def_use_chain(s);
- if (vectsize < Matcher::superword_max_vector_size(btype)) {
+ if (vectsize < Matcher::max_vector_size_auto_vectorization(btype)) {
vw = MIN2(vectsize * type2aelembytes(btype), vw);
}
@@ -810,7 +766,7 @@ int SuperWord::get_vw_bytes_special(MemNode* s) {
//---------------------------get_iv_adjustment---------------------------
// Calculate loop's iv adjustment for this memory ops.
int SuperWord::get_iv_adjustment(MemNode* mem_ref) {
- VPointer align_to_ref_p(mem_ref, phase(), lpt(), nullptr, false);
+ VPointer align_to_ref_p(mem_ref, vloop());
int offset = align_to_ref_p.offset_in_bytes();
int scale = align_to_ref_p.scale_in_bytes();
int elt_size = align_to_ref_p.memory_size();
@@ -830,7 +786,7 @@ int SuperWord::get_iv_adjustment(MemNode* mem_ref) {
}
#ifndef PRODUCT
- if (TraceSuperWord) {
+ if (is_trace_superword_alignment()) {
tty->print("SuperWord::get_iv_adjustment: n = %d, noffset = %d iv_adjust = %d elt_size = %d scale = %d iv_stride = %d vect_size %d: ",
mem_ref->_idx, offset, iv_adjustment, elt_size, scale, iv_stride(), vw);
mem_ref->dump();
@@ -863,14 +819,6 @@ void SuperWord::dependence_graph() {
// Get slice in predecessor order (last is first)
mem_slice_preds(n_tail, n, _nlist);
-#ifndef PRODUCT
- if(TraceSuperWord && Verbose) {
- tty->print_cr("SuperWord::dependence_graph: built a new mem slice");
- for (int j = _nlist.length() - 1; j >= 0 ; j--) {
- _nlist.at(j)->dump();
- }
- }
-#endif
// Make the slice dependent on the root
DepMem* slice = _dg.dep(n);
_dg.make_edge(_dg.root(), slice);
@@ -887,21 +835,15 @@ void SuperWord::dependence_graph() {
if (_dg.dep(s1)->in_cnt() == 0) {
_dg.make_edge(slice, s1);
}
- VPointer p1(s1->as_Mem(), phase(), lpt(), nullptr, false);
+ VPointer p1(s1->as_Mem(), vloop());
bool sink_dependent = true;
for (int k = j - 1; k >= 0; k--) {
Node* s2 = _nlist.at(k);
if (s1->is_Load() && s2->is_Load())
continue;
- VPointer p2(s2->as_Mem(), phase(), lpt(), nullptr, false);
+ VPointer p2(s2->as_Mem(), vloop());
int cmp = p1.cmp(p2);
- if (SuperWordRTDepCheck &&
- p1.base() != p2.base() && p1.valid() && p2.valid()) {
- // Trace disjoint pointers
- OrderedPair pp(p1.base(), p2.base());
- _disjoint_ptrs.append_if_missing(pp);
- }
if (!VPointer::not_equal(cmp)) {
// Possibly same address
_dg.make_edge(s1, s2);
@@ -913,27 +855,49 @@ void SuperWord::dependence_graph() {
}
}
- if (TraceSuperWord) {
+#ifndef PRODUCT
+ if (is_trace_superword_dependence_graph()) {
tty->print_cr("\nDependence graph for slice: %d", n->_idx);
for (int q = 0; q < _nlist.length(); q++) {
_dg.print(_nlist.at(q));
}
tty->cr();
}
+#endif
_nlist.clear();
}
+}
- if (TraceSuperWord) {
- tty->print_cr("\ndisjoint_ptrs: %s", _disjoint_ptrs.length() > 0 ? "" : "NONE");
- for (int r = 0; r < _disjoint_ptrs.length(); r++) {
- _disjoint_ptrs.at(r).print();
- tty->cr();
+void SuperWord::find_memory_slices() {
+ assert(_mem_slice_head.length() == 0, "mem_slice_head is empty");
+ assert(_mem_slice_tail.length() == 0, "mem_slice_tail is empty");
+
+ // Iterate over all memory phis
+ for (DUIterator_Fast imax, i = cl()->fast_outs(imax); i < imax; i++) {
+ PhiNode* phi = cl()->fast_out(i)->isa_Phi();
+ if (phi != nullptr && in_bb(phi) && phi->is_memory_phi()) {
+ Node* phi_tail = phi->in(LoopNode::LoopBackControl);
+ if (phi_tail != phi->in(LoopNode::EntryControl)) {
+ _mem_slice_head.push(phi);
+ _mem_slice_tail.push(phi_tail->as_Mem());
+ }
}
- tty->cr();
}
+ NOT_PRODUCT( if (is_trace_superword_memory_slices()) { print_memory_slices(); } )
+}
+
+#ifndef PRODUCT
+void SuperWord::print_memory_slices() {
+ tty->print_cr("\nSuperWord::print_memory_slices: %s",
+ _mem_slice_head.length() > 0 ? "" : "NONE");
+ for (int m = 0; m < _mem_slice_head.length(); m++) {
+ tty->print("%6d ", m); _mem_slice_head.at(m)->dump();
+ tty->print(" "); _mem_slice_tail.at(m)->dump();
+ }
}
+#endif
//---------------------------mem_slice_preds---------------------------
// Return a memory slice (node list) in predecessor order starting at "start"
@@ -942,16 +906,12 @@ void SuperWord::mem_slice_preds(Node* start, Node* stop, GrowableArray &p
Node* n = start;
Node* prev = nullptr;
while (true) {
- NOT_PRODUCT( if(is_trace_mem_slice()) tty->print_cr("SuperWord::mem_slice_preds: n %d", n->_idx);)
assert(in_bb(n), "must be in block");
for (DUIterator_Fast imax, i = n->fast_outs(imax); i < imax; i++) {
Node* out = n->fast_out(i);
if (out->is_Load()) {
if (in_bb(out)) {
preds.push(out);
- if (TraceSuperWord && Verbose) {
- tty->print_cr("SuperWord::mem_slice_preds: added pred(%d)", out->_idx);
- }
}
} else {
// FIXME
@@ -970,13 +930,20 @@ void SuperWord::mem_slice_preds(Node* start, Node* stop, GrowableArray &p
}//for
if (n == stop) break;
preds.push(n);
- if (TraceSuperWord && Verbose) {
- tty->print_cr("SuperWord::mem_slice_preds: added pred(%d)", n->_idx);
- }
prev = n;
assert(n->is_Mem(), "unexpected node %s", n->Name());
n = n->in(MemNode::Memory);
}
+
+#ifndef PRODUCT
+ if (is_trace_superword_memory_slices()) {
+ tty->print_cr("\nSuperWord::mem_slice_preds:");
+ stop->dump();
+ for (int j = preds.length() - 1; j >= 0 ; j--) {
+ preds.at(j)->dump();
+ }
+ }
+#endif
}
//------------------------------stmts_can_pack---------------------------
@@ -990,8 +957,8 @@ bool SuperWord::stmts_can_pack(Node* s1, Node* s2, int align) {
if(!is_java_primitive(bt1) || !is_java_primitive(bt2))
return false;
BasicType longer_bt = longer_type_for_conversion(s1);
- if (Matcher::superword_max_vector_size(bt1) < 2 ||
- (longer_bt != T_ILLEGAL && Matcher::superword_max_vector_size(longer_bt) < 2)) {
+ if (Matcher::max_vector_size_auto_vectorization(bt1) < 2 ||
+ (longer_bt != T_ILLEGAL && Matcher::max_vector_size_auto_vectorization(longer_bt) < 2)) {
return false; // No vectors for this type
}
@@ -1044,8 +1011,8 @@ bool SuperWord::are_adjacent_refs(Node* s1, Node* s2) {
// Adjacent memory references must have the same base, be comparable
// and have the correct distance between them.
- VPointer p1(s1->as_Mem(), phase(), lpt(), nullptr, false);
- VPointer p2(s2->as_Mem(), phase(), lpt(), nullptr, false);
+ VPointer p1(s1->as_Mem(), vloop());
+ VPointer p2(s2->as_Mem(), vloop());
if (p1.base() != p2.base() || !p1.comparable(p2)) return false;
int diff = p2.offset_in_bytes() - p1.offset_in_bytes();
return diff == data_size(s1);
@@ -1077,54 +1044,71 @@ bool SuperWord::isomorphic(Node* s1, Node* s2) {
//------------------------------independent---------------------------
// Is there no data path from s1 to s2 or s2 to s1?
bool SuperWord::independent(Node* s1, Node* s2) {
- // assert(s1->Opcode() == s2->Opcode(), "check isomorphic first");
int d1 = depth(s1);
int d2 = depth(s2);
- if (d1 == d2) return s1 != s2;
+
+ if (d1 == d2) {
+ // Same depth:
+ // 1) same node -> dependent
+ // 2) different nodes -> same level implies there is no path
+ return s1 != s2;
+ }
+
+ // Traversal starting at the deeper node to find the shallower one.
Node* deep = d1 > d2 ? s1 : s2;
Node* shallow = d1 > d2 ? s2 : s1;
+ int min_d = MIN2(d1, d2); // prune traversal at min_d
- visited_clear();
-
- return independent_path(shallow, deep);
+ ResourceMark rm;
+ Unique_Node_List worklist;
+ worklist.push(deep);
+ for (uint i = 0; i < worklist.size(); i++) {
+ Node* n = worklist.at(i);
+ for (DepPreds preds(n, _dg); !preds.done(); preds.next()) {
+ Node* pred = preds.current();
+ if (in_bb(pred) && depth(pred) >= min_d) {
+ if (pred == shallow) {
+ return false; // found it -> dependent
+ }
+ worklist.push(pred);
+ }
+ }
+ }
+ return true; // not found -> independent
}
-//------------------------------find_dependence---------------------
-// Is any s1 in p dependent on any s2 in p? Yes: return such a s2. No: return nullptr.
+// Are all nodes in nodes list mutually independent?
// We could query independent(s1, s2) for all pairs, but that results
-// in O(p.size * p.size) graph traversals. We can do it all in one BFS!
-// Start the BFS traversal at all nodes from the pack. Traverse DepPreds
-// recursively, for nodes that have at least depth min_d, which is the
-// smallest depth of all nodes from the pack. Once we have traversed all
-// those nodes, and have not found another node from the pack, we know
-// that all nodes in the pack are independent.
-Node* SuperWord::find_dependence(Node_List* p) {
- if (is_marked_reduction(p->at(0))) {
- return nullptr; // ignore reductions
- }
+// in O(size * size) graph traversals. We can do it all in one BFS!
+// Start the BFS traversal at all nodes from the nodes list. Traverse
+// Preds recursively, for nodes that have at least depth min_d, which
+// is the smallest depth of all nodes from the nodes list. Once we have
+// traversed all those nodes, and have not found another node from the
+// nodes list, we know that all nodes in the nodes list are independent.
+bool SuperWord::mutually_independent(const Node_List* nodes) const {
ResourceMark rm;
- Unique_Node_List worklist; // traversal queue
- int min_d = depth(p->at(0));
- visited_clear();
- for (uint k = 0; k < p->size(); k++) {
- Node* n = p->at(k);
+ Unique_Node_List worklist;
+ VectorSet nodes_set;
+ int min_d = depth(nodes->at(0));
+ for (uint k = 0; k < nodes->size(); k++) {
+ Node* n = nodes->at(k);
min_d = MIN2(min_d, depth(n));
- worklist.push(n); // start traversal at all nodes in p
- visited_set(n); // mark node
+ worklist.push(n); // start traversal at all nodes in nodes list
+ nodes_set.set(bb_idx(n));
}
for (uint i = 0; i < worklist.size(); i++) {
Node* n = worklist.at(i);
for (DepPreds preds(n, _dg); !preds.done(); preds.next()) {
Node* pred = preds.current();
if (in_bb(pred) && depth(pred) >= min_d) {
- if (visited_test(pred)) { // marked as in p?
- return pred;
+ if (nodes_set.test(bb_idx(pred))) {
+ return false; // found one -> dependent
}
worklist.push(pred);
}
}
}
- return nullptr;
+ return true; // not found -> independent
}
//--------------------------have_similar_inputs-----------------------
@@ -1172,27 +1156,6 @@ bool SuperWord::reduction(Node* s1, Node* s2) {
return retValue;
}
-//------------------------------independent_path------------------------------
-// Helper for independent
-bool SuperWord::independent_path(Node* shallow, Node* deep, uint dp) {
- if (dp >= 1000) return false; // stop deep recursion
- visited_set(deep);
- int shal_depth = depth(shallow);
- assert(shal_depth <= depth(deep), "must be");
- for (DepPreds preds(deep, _dg); !preds.done(); preds.next()) {
- Node* pred = preds.current();
- if (in_bb(pred) && !visited_test(pred)) {
- if (shallow == pred) {
- return false;
- }
- if (shal_depth < depth(pred) && !independent_path(shallow, pred, dp+1)) {
- return false;
- }
- }
- }
- return true;
-}
-
//------------------------------set_alignment---------------------------
void SuperWord::set_alignment(Node* s1, Node* s2, int align) {
set_alignment(s1, align);
@@ -1210,9 +1173,8 @@ int SuperWord::data_size(Node* s) {
return bsize;
}
-//------------------------------extend_packlist---------------------------
// Extend packset by following use->def and def->use links from pack members.
-void SuperWord::extend_packlist() {
+void SuperWord::extend_packset_with_more_pairs_by_following_use_and_def() {
bool changed;
do {
packset_sort(_packset.length());
@@ -1231,10 +1193,12 @@ void SuperWord::extend_packlist() {
}
}
- if (TraceSuperWord) {
- tty->print_cr("\nAfter extend_packlist");
+#ifndef PRODUCT
+ if (is_trace_superword_packset()) {
+ tty->print_cr("\nAfter Superword::extend_packset_with_more_pairs_by_following_use_and_def");
print_packset();
}
+#endif
}
//------------------------------adjust_alignment_for_type_conversion---------------------------------
@@ -1264,7 +1228,12 @@ bool SuperWord::follow_use_defs(Node_List* p) {
if (s1->is_Load()) return false;
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_use_defs: s1 %d, align %d", s1->_idx, alignment(s1));)
+#ifndef PRODUCT
+ if (is_trace_superword_alignment()) {
+ tty->print_cr("SuperWord::follow_use_defs: s1 %d, align %d",
+ s1->_idx, alignment(s1));
+ }
+#endif
bool changed = false;
int start = s1->is_Store() ? MemNode::ValueIn : 1;
int end = s1->is_Store() ? MemNode::ValueIn+1 : s1->req();
@@ -1283,7 +1252,12 @@ bool SuperWord::follow_use_defs(Node_List* p) {
pair->push(t1);
pair->push(t2);
_packset.append(pair);
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_use_defs: set_alignment(%d, %d, %d)", t1->_idx, t2->_idx, align);)
+#ifndef PRODUCT
+ if (is_trace_superword_alignment()) {
+ tty->print_cr("SuperWord::follow_use_defs: set_alignment(%d, %d, %d)",
+ t1->_idx, t2->_idx, align);
+ }
+#endif
set_alignment(t1, t2, align);
changed = true;
}
@@ -1305,7 +1279,12 @@ bool SuperWord::follow_def_uses(Node_List* p) {
if (s1->is_Store()) return false;
int align = alignment(s1);
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_def_uses: s1 %d, align %d", s1->_idx, align);)
+#ifndef PRODUCT
+ if (is_trace_superword_alignment()) {
+ tty->print_cr("SuperWord::follow_def_uses: s1 %d, align %d",
+ s1->_idx, align);
+ }
+#endif
int savings = -1;
int num_s1_uses = 0;
Node* u1 = nullptr;
@@ -1323,7 +1302,7 @@ bool SuperWord::follow_def_uses(Node_List* p) {
// Only follow non-memory nodes in block - we do not want to resurrect misaligned packs.
continue;
}
- if (t2->Opcode() == Op_AddI && t2 == _lp->as_CountedLoop()->incr()) continue; // don't mess with the iv
+ if (t2->Opcode() == Op_AddI && t2 == cl()->incr()) continue; // don't mess with the iv
if (!opnd_positions_match(s1, t1, s2, t2))
continue;
int adjusted_align = alignment(s1);
@@ -1347,7 +1326,12 @@ bool SuperWord::follow_def_uses(Node_List* p) {
pair->push(u1);
pair->push(u2);
_packset.append(pair);
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("SuperWord::follow_def_uses: set_alignment(%d, %d, %d)", u1->_idx, u2->_idx, align);)
+#ifndef PRODUCT
+ if (is_trace_superword_alignment()) {
+ tty->print_cr("SuperWord::follow_def_uses: set_alignment(%d, %d, %d)",
+ u1->_idx, u2->_idx, align);
+ }
+#endif
set_alignment(u1, u2, align);
changed = true;
}
@@ -1497,12 +1481,13 @@ int SuperWord::adjacent_profit(Node* s1, Node* s2) { return 2; }
int SuperWord::pack_cost(int ct) { return ct; }
int SuperWord::unpack_cost(int ct) { return ct; }
-//------------------------------combine_packs---------------------------
// Combine packs A and B with A.last == B.first into A.first..,A.last,B.second,..B.last
-void SuperWord::combine_packs() {
+void SuperWord::combine_pairs_to_longer_packs() {
#ifdef ASSERT
+ assert(!_packset.is_empty(), "packset not empty");
for (int i = 0; i < _packset.length(); i++) {
assert(_packset.at(i) != nullptr, "no nullptr in packset");
+ assert(_packset.at(i)->size() == 2, "all packs are pairs");
}
#endif
@@ -1528,97 +1513,152 @@ void SuperWord::combine_packs() {
}
}
- // Split packs which have size greater then max vector size.
- for (int i = 0; i < _packset.length(); i++) {
- Node_List* p1 = _packset.at(i);
- if (p1 != nullptr) {
- uint max_vlen = max_vector_size_in_def_use_chain(p1->at(0)); // Max elements in vector
- assert(is_power_of_2(max_vlen), "sanity");
- uint psize = p1->size();
- if (!is_power_of_2(psize)) {
- // We currently only support power-of-2 sizes for vectors.
+ // Remove all nullptr from packset
+ compress_packset();
+
+ assert(!_packset.is_empty(), "must have combined some packs");
+
#ifndef PRODUCT
- if (TraceSuperWord) {
- tty->cr();
- tty->print_cr("WARNING: Removed pack[%d] with size that is not a power of 2:", i);
- print_pack(p1);
- }
+ if (is_trace_superword_packset()) {
+ tty->print_cr("\nAfter Superword::combine_pairs_to_longer_packs");
+ print_packset();
+ }
#endif
- _packset.at_put(i, nullptr);
- continue;
- }
- if (psize > max_vlen) {
- Node_List* pack = new Node_List();
- for (uint j = 0; j < psize; j++) {
- pack->push(p1->at(j));
- if (pack->size() >= max_vlen) {
- assert(is_power_of_2(pack->size()), "sanity");
- _packset.append(pack);
- pack = new Node_List();
- }
- }
- _packset.at_put(i, nullptr);
+}
+
+void SuperWord::split_packs_longer_than_max_vector_size() {
+ assert(!_packset.is_empty(), "packset not empty");
+ DEBUG_ONLY( int old_packset_length = _packset.length(); )
+
+ for (int i = 0; i < _packset.length(); i++) {
+ Node_List* pack = _packset.at(i);
+ assert(pack != nullptr, "no nullptr in packset");
+ uint max_vlen = max_vector_size_in_def_use_chain(pack->at(0));
+ assert(is_power_of_2(max_vlen), "sanity");
+ uint pack_size = pack->size();
+ if (pack_size <= max_vlen) {
+ continue;
+ }
+ // Split off the "upper" nodes into new packs
+ Node_List* new_pack = new Node_List();
+ for (uint j = max_vlen; j < pack_size; j++) {
+ Node* n = pack->at(j);
+ // is new_pack full?
+ if (new_pack->size() >= max_vlen) {
+ assert(is_power_of_2(new_pack->size()), "sanity %d", new_pack->size());
+ _packset.append(new_pack);
+ new_pack = new Node_List();
+ }
+ new_pack->push(n);
+ }
+ // remaining new_pack
+ if (new_pack->size() > 1) {
+ _packset.append(new_pack);
+ } else {
+#ifndef PRODUCT
+ if (is_trace_superword_rejections()) {
+ tty->cr();
+ tty->print_cr("WARNING: Node dropped out of odd size pack:");
+ new_pack->at(0)->dump();
+ print_pack(pack);
}
+#endif
+ }
+ // truncate
+ while (pack->size() > max_vlen) {
+ pack->pop();
}
}
- // We know that the nodes in a pair pack were independent - this gives us independence
- // at distance 1. But now that we may have more than 2 nodes in a pack, we need to check
- // if they are all mutually independent. If there is a dependence we remove the pack.
- // This is better than giving up completely - we can have partial vectorization if some
- // are rejected and others still accepted.
- //
- // Examples with dependence at distance 1 (pack pairs are not created):
- // for (int i ...) { v[i + 1] = v[i] + 5; }
- // for (int i ...) { v[i] = v[i - 1] + 5; }
- //
- // Example with independence at distance 1, but dependence at distance 2 (pack pairs are
- // created and we need to filter them out now):
- // for (int i ...) { v[i + 2] = v[i] + 5; }
- // for (int i ...) { v[i] = v[i - 2] + 5; }
- //
- // Note: dependencies are created when a later load may reference the same memory location
- // as an earlier store. This happens in "read backward" or "store forward" cases. On the
- // other hand, "read forward" or "store backward" cases do not have such dependencies:
- // for (int i ...) { v[i] = v[i + 1] + 5; }
- // for (int i ...) { v[i - 1] = v[i] + 5; }
- for (int i = 0; i < _packset.length(); i++) {
- Node_List* p = _packset.at(i);
- if (p != nullptr) {
- Node* dependence = find_dependence(p);
- if (dependence != nullptr) {
+ assert(old_packset_length <= _packset.length(), "we only increased the number of packs");
+
#ifndef PRODUCT
- if (TraceSuperWord) {
- tty->cr();
- tty->print_cr("WARNING: Found dependency at distance greater than 1.");
- dependence->dump();
- tty->print_cr("In pack[%d]", i);
- print_pack(p);
- }
+ if (is_trace_superword_packset()) {
+ tty->print_cr("\nAfter Superword::split_packs_longer_than_max_vector_size");
+ print_packset();
+ }
#endif
- _packset.at_put(i, nullptr);
+}
+
+template
+void SuperWord::filter_packs(const char* filter_name,
+ const char* error_message,
+ FilterPredicate filter) {
+ int new_packset_length = 0;
+ for (int i = 0; i < _packset.length(); i++) {
+ Node_List* pack = _packset.at(i);
+ assert(pack != nullptr, "no nullptr in packset");
+ if (filter(pack)) {
+ assert(i >= new_packset_length, "only move packs down");
+ _packset.at_put(new_packset_length++, pack);
+ } else {
+ remove_pack_at(i);
+#ifndef PRODUCT
+ if (is_trace_superword_rejections()) {
+ tty->cr();
+ tty->print_cr("WARNING: Removed pack: %s:", error_message);
+ print_pack(pack);
}
+#endif
}
}
- // Remove all nullptr from packset
- compress_packset();
+ assert(_packset.length() >= new_packset_length, "filter only reduces number of packs");
+ _packset.trunc_to(new_packset_length);
#ifndef PRODUCT
- if (TraceSuperWord) {
- tty->print_cr("\nAfter combine_packs");
+ if (is_trace_superword_packset() && filter_name != nullptr) {
+ tty->print_cr("\nAfter %s:", filter_name);
print_packset();
}
#endif
}
+void SuperWord::filter_packs_for_power_of_2_size() {
+ filter_packs("SuperWord::filter_packs_for_power_of_2_size",
+ "size is not a power of 2",
+ [&](const Node_List* pack) {
+ return is_power_of_2(pack->size());
+ });
+}
+
+// We know that the nodes in a pair pack were independent - this gives us independence
+// at distance 1. But now that we may have more than 2 nodes in a pack, we need to check
+// if they are all mutually independent. If there is a dependence we remove the pack.
+// This is better than giving up completely - we can have partial vectorization if some
+// are rejected and others still accepted.
+//
+// Examples with dependence at distance 1 (pack pairs are not created):
+// for (int i ...) { v[i + 1] = v[i] + 5; }
+// for (int i ...) { v[i] = v[i - 1] + 5; }
+//
+// Example with independence at distance 1, but dependence at distance 2 (pack pairs are
+// created and we need to filter them out now):
+// for (int i ...) { v[i + 2] = v[i] + 5; }
+// for (int i ...) { v[i] = v[i - 2] + 5; }
+//
+// Note: dependencies are created when a later load may reference the same memory location
+// as an earlier store. This happens in "read backward" or "store forward" cases. On the
+// other hand, "read forward" or "store backward" cases do not have such dependencies:
+// for (int i ...) { v[i] = v[i + 1] + 5; }
+// for (int i ...) { v[i - 1] = v[i] + 5; }
+void SuperWord::filter_packs_for_mutual_independence() {
+ filter_packs("SuperWord::filter_packs_for_mutual_independence",
+ "found dependency between nodes at distance greater than 1",
+ [&](const Node_List* pack) {
+ // reductions are trivially connected
+ return is_marked_reduction(pack->at(0)) ||
+ mutually_independent(pack);
+ });
+}
+
// Find the set of alignment solutions for load/store pack.
-const AlignmentSolution* SuperWord::pack_alignment_solution(Node_List* pack) {
+const AlignmentSolution* SuperWord::pack_alignment_solution(const Node_List* pack) {
assert(pack != nullptr && (pack->at(0)->is_Load() || pack->at(0)->is_Store()), "only load/store packs");
const MemNode* mem_ref = pack->at(0)->as_Mem();
- VPointer mem_ref_p(mem_ref, phase(), lpt(), nullptr, false);
- const CountedLoopEndNode* pre_end = lp()->pre_loop_end();
+ VPointer mem_ref_p(mem_ref, vloop());
+ const CountedLoopEndNode* pre_end = vloop().pre_loop_end();
assert(pre_end->stride_is_con(), "pre loop stride is constant");
AlignmentSolver solver(pack->at(0)->as_Mem(),
@@ -1646,8 +1686,8 @@ void SuperWord::filter_packs_for_alignment() {
}
#ifndef PRODUCT
- if (TraceSuperWord || is_trace_align_vector()) {
- tty->print_cr("\nfilter_packs_for_alignment:");
+ if (is_trace_superword_info() || is_trace_align_vector()) {
+ tty->print_cr("\nSuperWord::filter_packs_for_alignment:");
}
#endif
@@ -1657,38 +1697,39 @@ void SuperWord::filter_packs_for_alignment() {
AlignmentSolution const* current = new TrivialAlignmentSolution();
int mem_ops_count = 0;
int mem_ops_rejected = 0;
- for (int i = 0; i < _packset.length(); i++) {
- Node_List* p = _packset.at(i);
- if (p != nullptr) {
- if (p->at(0)->is_Load() || p->at(0)->is_Store()) {
- mem_ops_count++;
- // Find solution for pack p, and filter with current solution.
- const AlignmentSolution* s = pack_alignment_solution(p);
- const AlignmentSolution* intersect = current->filter(s);
+
+ filter_packs("SuperWord::filter_packs_for_alignment",
+ "rejected by AlignVector (strict alignment requirement)",
+ [&](const Node_List* pack) {
+ // Only memops need to be aligned.
+ if (!pack->at(0)->is_Load() &&
+ !pack->at(0)->is_Store()) {
+ return true; // accept all non memops
+ }
+
+ mem_ops_count++;
+ const AlignmentSolution* s = pack_alignment_solution(pack);
+ const AlignmentSolution* intersect = current->filter(s);
#ifndef PRODUCT
- if (is_trace_align_vector()) {
- tty->print(" solution for pack: ");
- s->print();
- tty->print(" intersection with current: ");
- intersect->print();
- }
+ if (is_trace_align_vector()) {
+ tty->print(" solution for pack: ");
+ s->print();
+ tty->print(" intersection with current: ");
+ intersect->print();
+ }
#endif
+ if (intersect->is_empty()) {
+ mem_ops_rejected++;
+ return false; // reject because of empty solution
+ }
- if (intersect->is_empty()) {
- // Solution failed or is not compatible, remove pack i.
- _packset.at_put(i, nullptr);
- mem_ops_rejected++;
- } else {
- // Solution is compatible.
- current = intersect;
- }
- }
- }
- }
+ current = intersect;
+ return true; // accept because of non-empty solution
+ });
#ifndef PRODUCT
- if (TraceSuperWord || is_trace_align_vector()) {
+ if (is_trace_superword_info() || is_trace_align_vector()) {
tty->print("\n final solution: ");
current->print();
tty->print_cr(" rejected mem_ops packs: %d of %d", mem_ops_rejected, mem_ops_count);
@@ -1702,9 +1743,6 @@ void SuperWord::filter_packs_for_alignment() {
// -> must change pre-limit to achieve alignment
set_align_to_ref(current->as_constrained()->mem_ref());
}
-
- // Remove all nullptr from packset
- compress_packset();
}
// Compress packset, such that it has no nullptr entries
@@ -1722,7 +1760,7 @@ void SuperWord::compress_packset() {
//-----------------------------construct_my_pack_map--------------------------
// Construct the map from nodes to packs. Only valid after the
-// point where a node is only in one pack (after combine_packs).
+// point where a node is only in one pack (after combine_pairs_to_longer_packs).
void SuperWord::construct_my_pack_map() {
for (int i = 0; i < _packset.length(); i++) {
Node_List* p = _packset.at(i);
@@ -1741,23 +1779,22 @@ void SuperWord::construct_my_pack_map() {
}
}
-//------------------------------filter_packs---------------------------
-// Remove packs that are not implemented or not profitable.
-void SuperWord::filter_packs() {
- // Remove packs that are not implemented
- for (int i = _packset.length() - 1; i >= 0; i--) {
- Node_List* pk = _packset.at(i);
- bool impl = implemented(pk);
- if (!impl) {
-#ifndef PRODUCT
- if ((TraceSuperWord && Verbose) || _vector_loop_debug) {
- tty->print_cr("Unimplemented");
- pk->at(0)->dump();
- }
-#endif
- remove_pack_at(i);
- }
- Node *n = pk->at(0);
+// Remove packs that are not implemented
+void SuperWord::filter_packs_for_implemented() {
+ filter_packs("SuperWord::filter_packs_for_implemented",
+ "Unimplemented",
+ [&](const Node_List* pack) {
+ return implemented(pack);
+ });
+}
+
+// Remove packs that are not profitable.
+void SuperWord::filter_packs_for_profitable() {
+ // Count the number of reductions vs other vector ops, for the
+ // reduction profitability heuristic.
+ for (int i = 0; i < _packset.length(); i++) {
+ Node_List* pack = _packset.at(i);
+ Node* n = pack->at(0);
if (is_marked_reduction(n)) {
_num_reductions++;
} else {
@@ -1766,28 +1803,22 @@ void SuperWord::filter_packs() {
}
// Remove packs that are not profitable
- bool changed;
- do {
- changed = false;
- for (int i = _packset.length() - 1; i >= 0; i--) {
- Node_List* pk = _packset.at(i);
- bool prof = profitable(pk);
- if (!prof) {
-#ifndef PRODUCT
- if ((TraceSuperWord && Verbose) || _vector_loop_debug) {
- tty->print_cr("Unprofitable");
- pk->at(0)->dump();
- }
-#endif
- remove_pack_at(i);
- changed = true;
- }
+ while (true) {
+ int old_packset_length = _packset.length();
+ filter_packs(nullptr, // don't dump each time
+ "size is not a power of 2",
+ [&](const Node_List* pack) {
+ return profitable(pack);
+ });
+ // Repeat until stable
+ if (old_packset_length == _packset.length()) {
+ break;
}
- } while (changed);
+ }
#ifndef PRODUCT
- if (TraceSuperWord) {
- tty->print_cr("\nAfter filter_packs");
+ if (is_trace_superword_packset()) {
+ tty->print_cr("\nAfter Superword::filter_packs_for_profitable");
print_packset();
tty->cr();
}
@@ -1796,7 +1827,7 @@ void SuperWord::filter_packs() {
//------------------------------implemented---------------------------
// Can code be generated for pack p?
-bool SuperWord::implemented(Node_List* p) {
+bool SuperWord::implemented(const Node_List* p) {
bool retValue = false;
Node* p0 = p->at(0);
if (p0 != nullptr) {
@@ -1856,7 +1887,7 @@ bool SuperWord::requires_long_to_int_conversion(int opc) {
//------------------------------same_inputs--------------------------
// For pack p, are all idx operands the same?
-bool SuperWord::same_inputs(Node_List* p, int idx) {
+bool SuperWord::same_inputs(const Node_List* p, int idx) {
Node* p0 = p->at(0);
uint vlen = p->size();
Node* p0_def = p0->in(idx);
@@ -1872,7 +1903,7 @@ bool SuperWord::same_inputs(Node_List* p, int idx) {
//------------------------------profitable---------------------------
// For pack p, are all operands and all uses (with in the block) vector?
-bool SuperWord::profitable(Node_List* p) {
+bool SuperWord::profitable(const Node_List* p) {
Node* p0 = p->at(0);
uint start, end;
VectorNode::vector_operands(p0, &start, &end);
@@ -1924,8 +1955,8 @@ bool SuperWord::profitable(Node_List* p) {
// Reductions should only have a Phi use at the loop head or a non-phi use
// outside of the loop if it is the last element of the pack (e.g. SafePoint).
if (is_marked_reduction(def) &&
- ((use->is_Phi() && use->in(0) == _lpt->_head) ||
- (!_lpt->is_member(_phase->get_loop(_phase->ctrl_or_self(use))) && i == p->size()-1))) {
+ ((use->is_Phi() && use->in(0) == lpt()->_head) ||
+ (!lpt()->is_member(phase()->get_loop(phase()->ctrl_or_self(use))) && i == p->size()-1))) {
continue;
}
if (!is_vector_use(use, k)) {
@@ -1974,24 +2005,16 @@ void SuperWord::verify_packs() {
// Verify independence at pack level.
for (int i = 0; i < _packset.length(); i++) {
Node_List* p = _packset.at(i);
- Node* dependence = find_dependence(p);
- if (dependence != nullptr) {
- tty->print_cr("Other nodes in pack have dependence on:");
- dependence->dump();
- tty->print_cr("The following nodes are not independent:");
- for (uint k = 0; k < p->size(); k++) {
- Node* n = p->at(k);
- if (!independent(n, dependence)) {
- n->dump();
- }
- }
- tty->print_cr("They are all from pack[%d]", i);
+ if (!is_marked_reduction(p->at(0)) &&
+ !mutually_independent(p)) {
+ tty->print_cr("FAILURE: nodes not mutually independent in pack[%d]", i);
print_pack(p);
+ assert(false, "pack nodes not mutually independent");
}
- assert(dependence == nullptr, "all nodes in pack must be mutually independent");
}
// Verify all nodes in packset have my_pack set correctly.
+ ResourceMark rm;
Unique_Node_List processed;
for (int i = 0; i < _packset.length(); i++) {
Node_List* p = _packset.at(i);
@@ -2269,24 +2292,26 @@ void SuperWord::schedule() {
// introduced a cycle. The SuperWord paper mentions the need for this
// in "3.7 Scheduling".
if (!graph.schedule_success()) {
- if (TraceSuperWord) {
+#ifndef PRODUCT
+ if (is_trace_superword_rejections()) {
tty->print_cr("SuperWord::schedule found cycle in PacksetGraph:");
graph.print(true, false);
tty->print_cr("removing all packs from packset.");
}
+#endif
_packset.clear();
return;
}
#ifndef PRODUCT
- if (TraceSuperWord) {
+ if (is_trace_superword_info()) {
tty->print_cr("SuperWord::schedule: memops_schedule:");
memops_schedule.dump();
}
#endif
CountedLoopNode* cl = lpt()->_head->as_CountedLoop();
- _phase->C->print_method(PHASE_SUPERWORD1_BEFORE_SCHEDULE, 4, cl);
+ phase()->C->print_method(PHASE_SUPERWORD1_BEFORE_SCHEDULE, 4, cl);
// (4) Use the memops_schedule to re-order the memops in all slices.
schedule_reorder_memops(memops_schedule);
@@ -2296,7 +2321,7 @@ void SuperWord::schedule() {
// Reorder the memory graph for all slices in parallel. We walk over the schedule once,
// and track the current memory state of each slice.
void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
- int max_slices = _phase->C->num_alias_types();
+ int max_slices = phase()->C->num_alias_types();
// When iterating over the memops_schedule, we keep track of the current memory state,
// which is the Phi or a store in the loop.
GrowableArray current_state_in_slice(max_slices, max_slices, nullptr);
@@ -2308,7 +2333,7 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
for (int i = 0; i < _mem_slice_head.length(); i++) {
Node* phi = _mem_slice_head.at(i);
assert(phi->is_Phi(), "must be phi");
- int alias_idx = _phase->C->get_alias_index(phi->adr_type());
+ int alias_idx = phase()->C->get_alias_index(phi->adr_type());
current_state_in_slice.at_put(alias_idx, phi);
// If we have a memory phi, we have a last store in the loop, find it over backedge.
@@ -2321,7 +2346,7 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
for (uint i = 0; i < memops_schedule.size(); i++) {
MemNode* n = memops_schedule.at(i)->as_Mem();
assert(n->is_Load() || n->is_Store(), "only loads or stores");
- int alias_idx = _phase->C->get_alias_index(n->adr_type());
+ int alias_idx = phase()->C->get_alias_index(n->adr_type());
Node* current_state = current_state_in_slice.at(alias_idx);
if (current_state == nullptr) {
// If there are only loads in a slice, we never update the memory
@@ -2330,7 +2355,7 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
assert(n->is_Load() && !in_bb(n->in(MemNode::Memory)),
"only loads can have memory state from outside loop");
} else {
- _igvn.replace_input_of(n, MemNode::Memory, current_state);
+ igvn().replace_input_of(n, MemNode::Memory, current_state);
if (n->is_Store()) {
current_state_in_slice.at_put(alias_idx, n);
}
@@ -2343,12 +2368,12 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
Node_List uses_after_loop;
for (int i = 0; i < _mem_slice_head.length(); i++) {
Node* phi = _mem_slice_head.at(i);
- int alias_idx = _phase->C->get_alias_index(phi->adr_type());
+ int alias_idx = phase()->C->get_alias_index(phi->adr_type());
Node* current_state = current_state_in_slice.at(alias_idx);
assert(current_state != nullptr, "slice is mapped");
assert(current_state != phi, "did some work in between");
assert(current_state->is_Store(), "sanity");
- _igvn.replace_input_of(phi, 2, current_state);
+ igvn().replace_input_of(phi, 2, current_state);
// Replace uses of old last store with current_state (new last store)
// Do it in two loops: first find all the uses, and change the graph
@@ -2367,7 +2392,7 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
for (uint j = 0; j < use->req(); j++) {
Node* def = use->in(j);
if (def == last_store) {
- _igvn.replace_input_of(use, j, current_state);
+ igvn().replace_input_of(use, j, current_state);
}
}
}
@@ -2384,7 +2409,7 @@ void SuperWord::schedule_reorder_memops(Node_List &memops_schedule) {
bool SuperWord::output() {
CountedLoopNode *cl = lpt()->_head->as_CountedLoop();
assert(cl->is_main_loop(), "SLP should only work on main loops");
- Compile* C = _phase->C;
+ Compile* C = phase()->C;
if (_packset.length() == 0) {
return false;
}
@@ -2395,14 +2420,11 @@ bool SuperWord::output() {
lpt()->dump_head();
}
#endif
- _phase->C->print_method(PHASE_SUPERWORD2_BEFORE_OUTPUT, 4, cl);
+ phase()->C->print_method(PHASE_SUPERWORD2_BEFORE_OUTPUT, 4, cl);
adjust_pre_loop_limit_to_align_main_loop_vectors();
- // Insert extract (unpack) operations for scalar uses
- for (int i = 0; i < _packset.length(); i++) {
- insert_extracts(_packset.at(i));
- }
+ DEBUG_ONLY(verify_no_extract());
uint max_vlen_in_bytes = 0;
uint max_vlen = 0;
@@ -2418,7 +2440,6 @@ bool SuperWord::output() {
uint vlen = p->size();
uint vlen_in_bytes = 0;
Node* vn = nullptr;
- NOT_PRODUCT(if(is_trace_cmov()) {tty->print_cr("VPointer::output: %d executed first, %d executed last in pack", first->_idx, n->_idx); print_pack(p);})
int opc = n->Opcode();
if (n->is_Load()) {
Node* ctl = n->in(MemNode::Control);
@@ -2427,7 +2448,7 @@ bool SuperWord::output() {
// Walk up the memory chain, and ignore any StoreVector that provably
// does not have any memory dependency.
while (mem->is_StoreVector()) {
- VPointer p_store(mem->as_Mem(), phase(), lpt(), nullptr, false);
+ VPointer p_store(mem->as_Mem(), vloop());
if (p_store.overlap_possible_with_any_in(p)) {
break;
} else {
@@ -2561,13 +2582,13 @@ bool SuperWord::output() {
}
// VectorMaskCmp
- ConINode* bol_test_node = _igvn.intcon((int)bol_test);
+ ConINode* bol_test_node = igvn().intcon((int)bol_test);
BasicType bt = velt_basic_type(cmp);
const TypeVect* vt = TypeVect::make(bt, vlen);
VectorNode* mask = new VectorMaskCmpNode(bol_test, cmp_in1, cmp_in2, bol_test_node, vt);
- _igvn.register_new_node_with_optimizer(mask);
- _phase->set_ctrl(mask, _phase->get_ctrl(p->at(0)));
- _igvn._worklist.push(mask);
+ igvn().register_new_node_with_optimizer(mask);
+ phase()->set_ctrl(mask, phase()->get_ctrl(p->at(0)));
+ igvn()._worklist.push(mask);
// VectorBlend
vn = new VectorBlendNode(blend_in1, blend_in2, mask);
@@ -2640,8 +2661,8 @@ bool SuperWord::output() {
assert(n->req() == 2, "only one input expected");
Node* in = vector_opd(p, 1);
Node* longval = VectorNode::make(opc, in, nullptr, vlen, T_LONG);
- _igvn.register_new_node_with_optimizer(longval);
- _phase->set_ctrl(longval, _phase->get_ctrl(first));
+ igvn().register_new_node_with_optimizer(longval);
+ phase()->set_ctrl(longval, phase()->get_ctrl(first));
vn = VectorCastNode::make(Op_VectorCastL2X, longval, T_INT, vlen);
vlen_in_bytes = vn->as_Vector()->length_in_bytes();
} else if (VectorNode::is_convert_opcode(opc)) {
@@ -2682,13 +2703,13 @@ bool SuperWord::output() {
#endif
_block.at_put(i, vn);
- _igvn.register_new_node_with_optimizer(vn);
- _phase->set_ctrl(vn, _phase->get_ctrl(first));
+ igvn().register_new_node_with_optimizer(vn);
+ phase()->set_ctrl(vn, phase()->get_ctrl(first));
for (uint j = 0; j < p->size(); j++) {
Node* pm = p->at(j);
- _igvn.replace_node(pm, vn);
+ igvn().replace_node(pm, vn);
}
- _igvn._worklist.push(vn);
+ igvn()._worklist.push(vn);
if (vlen > max_vlen) {
max_vlen = vlen;
@@ -2711,9 +2732,11 @@ bool SuperWord::output() {
if (cl->has_passed_slp()) {
uint slp_max_unroll_factor = cl->slp_max_unroll();
if (slp_max_unroll_factor == max_vlen) {
+#ifndef PRODUCT
if (TraceSuperWordLoopUnrollAnalysis) {
tty->print_cr("vector loop(unroll=%d, len=%d)\n", max_vlen, max_vlen_in_bytes*BitsPerByte);
}
+#endif
// For atomic unrolled loops which are vector mapped, instigate more unrolling
cl->set_notpassed_slp();
// if vector resources are limited, do not allow additional unrolling
@@ -2725,7 +2748,7 @@ bool SuperWord::output() {
}
}
- _phase->C->print_method(PHASE_SUPERWORD3_AFTER_OUTPUT, 4, cl);
+ phase()->C->print_method(PHASE_SUPERWORD3_AFTER_OUTPUT, 4, cl);
return true;
}
@@ -2748,10 +2771,10 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
BasicType iv_bt = is_subword_type(p0_bt) ? p0_bt : T_INT;
assert(VectorNode::is_populate_index_supported(iv_bt), "Should support");
const TypeVect* vt = TypeVect::make(iv_bt, vlen);
- Node* vn = new PopulateIndexNode(iv(), _igvn.intcon(1), vt);
+ Node* vn = new PopulateIndexNode(iv(), igvn().intcon(1), vt);
VectorNode::trace_new_vector(vn, "SuperWord");
- _igvn.register_new_node_with_optimizer(vn);
- _phase->set_ctrl(vn, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(vn);
+ phase()->set_ctrl(vn, phase()->get_ctrl(opd));
return vn;
}
@@ -2772,15 +2795,15 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
juint shift = t->get_con();
if (shift > mask) { // Unsigned cmp
cnt = ConNode::make(TypeInt::make(shift & mask));
- _igvn.register_new_node_with_optimizer(cnt);
+ igvn().register_new_node_with_optimizer(cnt);
}
} else {
if (t == nullptr || t->_lo < 0 || t->_hi > (int)mask) {
cnt = ConNode::make(TypeInt::make(mask));
- _igvn.register_new_node_with_optimizer(cnt);
+ igvn().register_new_node_with_optimizer(cnt);
cnt = new AndINode(opd, cnt);
- _igvn.register_new_node_with_optimizer(cnt);
- _phase->set_ctrl(cnt, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(cnt);
+ phase()->set_ctrl(cnt, phase()->get_ctrl(opd));
}
if (!opd->bottom_type()->isa_int()) {
assert(false, "int type only");
@@ -2789,8 +2812,8 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
}
// Move shift count into vector register.
cnt = VectorNode::shift_count(p0->Opcode(), cnt, vlen, velt_basic_type(p0));
- _igvn.register_new_node_with_optimizer(cnt);
- _phase->set_ctrl(cnt, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(cnt);
+ phase()->set_ctrl(cnt, phase()->get_ctrl(opd));
return cnt;
}
if (opd->is_StoreVector()) {
@@ -2808,8 +2831,8 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
if (p0->bottom_type()->isa_long()) {
p0_t = TypeLong::LONG;
conv = new ConvI2LNode(opd);
- _igvn.register_new_node_with_optimizer(conv);
- _phase->set_ctrl(conv, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(conv);
+ phase()->set_ctrl(conv, phase()->get_ctrl(opd));
}
vn = VectorNode::scalar2vector(conv, vlen, p0_t);
} else {
@@ -2817,8 +2840,8 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
vn = VectorNode::scalar2vector(opd, vlen, p0_t);
}
- _igvn.register_new_node_with_optimizer(vn);
- _phase->set_ctrl(vn, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(vn);
+ phase()->set_ctrl(vn, phase()->get_ctrl(opd));
VectorNode::trace_new_vector(vn, "SuperWord");
return vn;
}
@@ -2847,61 +2870,41 @@ Node* SuperWord::vector_opd(Node_List* p, int opd_idx) {
pk->add_opd(in2);
}
}
- _igvn.register_new_node_with_optimizer(pk);
- _phase->set_ctrl(pk, _phase->get_ctrl(opd));
+ igvn().register_new_node_with_optimizer(pk);
+ phase()->set_ctrl(pk, phase()->get_ctrl(opd));
VectorNode::trace_new_vector(pk, "SuperWord");
return pk;
}
-//------------------------------insert_extracts---------------------------
-// If a use of pack p is not a vector use, then replace the
-// use with an extract operation.
-void SuperWord::insert_extracts(Node_List* p) {
- if (p->at(0)->is_Store()) return;
- assert(_n_idx_list.is_empty(), "empty (node,index) list");
+#ifdef ASSERT
+// We check that every packset (name it p_def) only has vector uses (p_use),
+// which are proper vector uses of def.
+void SuperWord::verify_no_extract() {
+ for (int i = 0; i < _packset.length(); i++) {
+ Node_List* p_def = _packset.at(i);
- // Inspect each use of each pack member. For each use that is
- // not a vector use, replace the use with an extract operation.
+ // A vector store has no uses
+ if (p_def->at(0)->is_Store()) { continue; }
- for (uint i = 0; i < p->size(); i++) {
- Node* def = p->at(i);
- for (DUIterator_Fast jmax, j = def->fast_outs(jmax); j < jmax; j++) {
- Node* use = def->fast_out(j);
- for (uint k = 0; k < use->req(); k++) {
- Node* n = use->in(k);
- if (def == n) {
- Node_List* u_pk = my_pack(use);
- if ((u_pk == nullptr || use->is_CMove()) && !is_vector_use(use, k)) {
- _n_idx_list.push(use, k);
+ // for every def in p_def, and every use:
+ for (uint i = 0; i < p_def->size(); i++) {
+ Node* def = p_def->at(i);
+ for (DUIterator_Fast jmax, j = def->fast_outs(jmax); j < jmax; j++) {
+ Node* use = def->fast_out(j);
+ // find every use->def edge:
+ for (uint k = 0; k < use->req(); k++) {
+ Node* maybe_def = use->in(k);
+ if (def == maybe_def) {
+ Node_List* p_use = my_pack(use);
+ if (is_marked_reduction(def)) { continue; }
+ assert(p_use != nullptr && is_vector_use(use, k), "all uses must be vector uses");
}
}
}
}
}
-
- while (_n_idx_list.is_nonempty()) {
- Node* use = _n_idx_list.node();
- int idx = _n_idx_list.index();
- _n_idx_list.pop();
- Node* def = use->in(idx);
-
- if (is_marked_reduction(def)) continue;
-
- // Insert extract operation
- _igvn.hash_delete(def);
- int def_pos = alignment(def) / data_size(def);
-
- ConINode* def_pos_con = _igvn.intcon(def_pos)->as_ConI();
- Node* ex = ExtractNode::make(def, def_pos_con, velt_basic_type(def));
- _igvn.register_new_node_with_optimizer(ex);
- _phase->set_ctrl(ex, _phase->get_ctrl(def));
- _igvn.replace_input_of(use, idx, ex);
- _igvn._worklist.push(def);
-
- bb_insert_after(ex, bb_idx(def));
- set_velt_type(ex, velt_type(def));
- }
}
+#endif
//------------------------------is_vector_use---------------------------
// Is use->in(u_idx) a vector use?
@@ -2983,174 +2986,110 @@ bool SuperWord::is_vector_use(Node* use, int u_idx) {
//------------------------------construct_bb---------------------------
// Construct reverse postorder list of block members
bool SuperWord::construct_bb() {
- Node* entry = bb();
-
- assert(_stk.length() == 0, "stk is empty");
assert(_block.length() == 0, "block is empty");
- assert(_data_entry.length() == 0, "data_entry is empty");
- assert(_mem_slice_head.length() == 0, "mem_slice_head is empty");
- assert(_mem_slice_tail.length() == 0, "mem_slice_tail is empty");
- // Find non-control nodes with no inputs from within block,
- // create a temporary map from node _idx to bb_idx for use
- // by the visited and post_visited sets,
- // and count number of nodes in block.
- int bb_ct = 0;
+ // First pass over loop body:
+ // (1) Check that there are no unwanted nodes (LoadStore, MergeMem, data Proj).
+ // (2) Count number of nodes, and create a temporary map (_idx -> bb_idx).
+ // (3) Verify that all non-ctrl nodes have an input inside the loop.
+ int block_count = 0;
for (uint i = 0; i < lpt()->_body.size(); i++) {
- Node *n = lpt()->_body.at(i);
+ Node* n = lpt()->_body.at(i);
set_bb_idx(n, i); // Create a temporary map
if (in_bb(n)) {
+ block_count++;
+
if (n->is_LoadStore() || n->is_MergeMem() ||
(n->is_Proj() && !n->as_Proj()->is_CFG())) {
// Bailout if the loop has LoadStore, MergeMem or data Proj
// nodes. Superword optimization does not work with them.
+#ifndef PRODUCT
+ if (is_trace_superword_any()) {
+ tty->print_cr("SuperWord::construct_bb: fails because of unhandled node:");
+ n->dump();
+ }
+#endif
return false;
}
- bb_ct++;
+
+#ifdef ASSERT
if (!n->is_CFG()) {
bool found = false;
for (uint j = 0; j < n->req(); j++) {
Node* def = n->in(j);
- if (def && in_bb(def)) {
+ if (def != nullptr && in_bb(def)) {
found = true;
break;
}
}
- if (!found) {
- assert(n != entry, "can't be entry");
- _data_entry.push(n);
- }
- }
- }
- }
-
- // Find memory slices (head and tail)
- for (DUIterator_Fast imax, i = lp()->fast_outs(imax); i < imax; i++) {
- Node *n = lp()->fast_out(i);
- if (in_bb(n) && n->is_memory_phi()) {
- Node* n_tail = n->in(LoopNode::LoopBackControl);
- if (n_tail != n->in(LoopNode::EntryControl)) {
- if (!n_tail->is_Mem()) {
- assert(n_tail->is_Mem(), "unexpected node for memory slice: %s", n_tail->Name());
- return false; // Bailout
- }
- _mem_slice_head.push(n);
- _mem_slice_tail.push(n_tail);
+ assert(found, "every non-cfg node must have an input that is also inside the loop");
}
+#endif
}
}
- // Create an RPO list of nodes in block
-
- visited_clear();
- post_visited_clear();
+ // Create a reverse-post-order list of nodes in block
+ ResourceMark rm;
+ GrowableArray stack;
+ VectorSet visited;
+ VectorSet post_visited;
- // Push all non-control nodes with no inputs from within block, then control entry
- for (int j = 0; j < _data_entry.length(); j++) {
- Node* n = _data_entry.at(j);
- visited_set(n);
- _stk.push(n);
- }
- visited_set(entry);
- _stk.push(entry);
+ visited.set(bb_idx(cl()));
+ stack.push(cl());
// Do a depth first walk over out edges
- int rpo_idx = bb_ct - 1;
- int size;
- int reduction_uses = 0;
- while ((size = _stk.length()) > 0) {
- Node* n = _stk.top(); // Leave node on stack
- if (!visited_test_set(n)) {
+ int rpo_idx = block_count - 1;
+ while (!stack.is_empty()) {
+ Node* n = stack.top(); // Leave node on stack
+ if (!visited.test_set(bb_idx(n))) {
// forward arc in graph
- } else if (!post_visited_test(n)) {
+ } else if (!post_visited.test(bb_idx(n))) {
// cross or back arc
+ const int old_length = stack.length();
for (DUIterator_Fast imax, i = n->fast_outs(imax); i < imax; i++) {
- Node *use = n->fast_out(i);
- if (in_bb(use) && !visited_test(use) &&
+ Node* use = n->fast_out(i);
+ if (in_bb(use) && !visited.test(bb_idx(use)) &&
// Don't go around backedge
- (!use->is_Phi() || n == entry)) {
- if (is_marked_reduction(use)) {
- // First see if we can map the reduction on the given system we are on, then
- // make a data entry operation for each reduction we see.
- BasicType bt = use->bottom_type()->basic_type();
- if (ReductionNode::implemented(use->Opcode(), Matcher::superword_max_vector_size(bt), bt)) {
- reduction_uses++;
- }
- }
- _stk.push(use);
+ (!use->is_Phi() || n == cl())) {
+ stack.push(use);
}
}
- if (_stk.length() == size) {
+ if (stack.length() == old_length) {
// There were no additional uses, post visit node now
- _stk.pop(); // Remove node from stack
- assert(rpo_idx >= 0, "");
+ stack.pop(); // Remove node from stack
+ assert(rpo_idx >= 0, "must still have idx to pass out");
_block.at_put_grow(rpo_idx, n);
rpo_idx--;
- post_visited_set(n);
- assert(rpo_idx >= 0 || _stk.is_empty(), "");
+ post_visited.set(bb_idx(n));
+ assert(rpo_idx >= 0 || stack.is_empty(), "still have idx left or are finished");
}
} else {
- _stk.pop(); // Remove post-visited node from stack
+ stack.pop(); // Remove post-visited node from stack
}
- }//while
+ }
- int ii_current = -1;
- unsigned int load_idx = (unsigned int)-1;
// Create real map of block indices for nodes
for (int j = 0; j < _block.length(); j++) {
Node* n = _block.at(j);
set_bb_idx(n, j);
- }//for
-
- // Ensure extra info is allocated.
- initialize_bb();
+ }
#ifndef PRODUCT
- if (TraceSuperWord) {
+ if (is_trace_superword_info()) {
print_bb();
- tty->print_cr("\ndata entry nodes: %s", _data_entry.length() > 0 ? "" : "NONE");
- for (int m = 0; m < _data_entry.length(); m++) {
- tty->print("%3d ", m);
- _data_entry.at(m)->dump();
- }
- tty->print_cr("\nmemory slices: %s", _mem_slice_head.length() > 0 ? "" : "NONE");
- for (int m = 0; m < _mem_slice_head.length(); m++) {
- tty->print("%3d ", m); _mem_slice_head.at(m)->dump();
- tty->print(" "); _mem_slice_tail.at(m)->dump();
- }
}
#endif
- assert(rpo_idx == -1 && bb_ct == _block.length(), "all block members found");
- return (_mem_slice_head.length() > 0) || (reduction_uses > 0) || (_data_entry.length() > 0);
+
+ assert(rpo_idx == -1 && block_count == _block.length(), "all block members found");
+ return true;
}
-//------------------------------initialize_bb---------------------------
// Initialize per node info
-void SuperWord::initialize_bb() {
+void SuperWord::initialize_node_info() {
Node* last = _block.at(_block.length() - 1);
grow_node_info(bb_idx(last));
}
-//------------------------------bb_insert_after---------------------------
-// Insert n into block after pos
-void SuperWord::bb_insert_after(Node* n, int pos) {
- int n_pos = pos + 1;
- // Make room
- for (int i = _block.length() - 1; i >= n_pos; i--) {
- _block.at_put_grow(i+1, _block.at(i));
- }
- for (int j = _node_info.length() - 1; j >= n_pos; j--) {
- _node_info.at_put_grow(j+1, _node_info.at(j));
- }
- // Set value
- _block.at_put_grow(n_pos, n);
- _node_info.at_put_grow(n_pos, SWNodeInfo::initial);
- // Adjust map from node->_idx to _block index
- for (int i = n_pos; i < _block.length(); i++) {
- set_bb_idx(_block.at(i), i);
- }
-}
-
//------------------------------compute_max_depth---------------------------
// Compute max depth for expressions from beginning of block
// Use to prune search paths during test for independence.
@@ -3179,9 +3118,11 @@ void SuperWord::compute_max_depth() {
ct++;
} while (again);
- if (TraceSuperWord && Verbose) {
+#ifndef PRODUCT
+ if (is_trace_superword_dependence_graph()) {
tty->print_cr("compute_max_depth iterated: %d times", ct);
}
+#endif
}
BasicType SuperWord::longer_type_for_conversion(Node* n) {
@@ -3227,10 +3168,10 @@ int SuperWord::max_vector_size_in_def_use_chain(Node* n) {
vt = (newt == T_ILLEGAL) ? vt : newt;
}
- int max = Matcher::superword_max_vector_size(vt);
+ int max = Matcher::max_vector_size_auto_vectorization(vt);
// If now there is no vectors for the longest type, the nodes with the longest
// type in the def-use chain are not packed in SuperWord::stmts_can_pack.
- return max < 2 ? Matcher::superword_max_vector_size(bt) : max;
+ return max < 2 ? Matcher::max_vector_size_auto_vectorization(bt) : max;
}
//-------------------------compute_vector_element_type-----------------------
@@ -3241,9 +3182,11 @@ int SuperWord::max_vector_size_in_def_use_chain(Node* n) {
// Normally the type of the add is integer, but for packed character
// operations the type of the add needs to be char.
void SuperWord::compute_vector_element_type() {
- if (TraceSuperWord && Verbose) {
+#ifndef PRODUCT
+ if (is_trace_superword_vector_element_type()) {
tty->print_cr("\ncompute_velt_type:");
}
+#endif
// Initial type
for (int i = 0; i < _block.length(); i++) {
@@ -3319,7 +3262,7 @@ void SuperWord::compute_vector_element_type() {
}
}
#ifndef PRODUCT
- if (TraceSuperWord && Verbose) {
+ if (is_trace_superword_vector_element_type()) {
for (int i = 0; i < _block.length(); i++) {
Node* n = _block.at(i);
velt_type(n)->dump();
@@ -3334,18 +3277,18 @@ void SuperWord::compute_vector_element_type() {
// Alignment within a vector memory reference
int SuperWord::memory_alignment(MemNode* s, int iv_adjust) {
#ifndef PRODUCT
- if ((TraceSuperWord && Verbose) || is_trace_alignment()) {
+ if (is_trace_superword_alignment()) {
tty->print("SuperWord::memory_alignment within a vector memory reference for %d: ", s->_idx); s->dump();
}
#endif
- VPointer p(s, phase(), lpt(), nullptr, false);
+ VPointer p(s, vloop());
if (!p.valid()) {
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("VPointer::memory_alignment: VPointer p invalid, return bottom_align");)
+ NOT_PRODUCT(if(is_trace_superword_alignment()) tty->print_cr("SuperWord::memory_alignment: VPointer p invalid, return bottom_align");)
return bottom_align;
}
int vw = get_vw_bytes_special(s);
if (vw < 2) {
- NOT_PRODUCT(if(is_trace_alignment()) tty->print_cr("VPointer::memory_alignment: vector_width_in_bytes < 2, return bottom_align");)
+ NOT_PRODUCT(if(is_trace_superword_alignment()) tty->print_cr("SuperWord::memory_alignment: vector_width_in_bytes < 2, return bottom_align");)
return bottom_align; // No vectors for this type
}
int offset = p.offset_in_bytes();
@@ -3353,8 +3296,8 @@ int SuperWord::memory_alignment(MemNode* s, int iv_adjust) {
int off_rem = offset % vw;
int off_mod = off_rem >= 0 ? off_rem : off_rem + vw;
#ifndef PRODUCT
- if ((TraceSuperWord && Verbose) || is_trace_alignment()) {
- tty->print_cr("VPointer::memory_alignment: off_rem = %d, off_mod = %d (offset = %d)", off_rem, off_mod, offset);
+ if (is_trace_superword_alignment()) {
+ tty->print_cr("SuperWord::memory_alignment: off_rem = %d, off_mod = %d (offset = %d)", off_rem, off_mod, offset);
}
#endif
return off_mod;
@@ -3379,7 +3322,7 @@ const Type* SuperWord::container_type(Node* n) {
}
return Type::get_const_basic_type(bt);
}
- const Type* t = _igvn.type(n);
+ const Type* t = igvn().type(n);
if (t->basic_type() == T_INT) {
// A narrow type of arithmetic operations will be determined by
// propagating the type of memory operations.
@@ -3399,7 +3342,7 @@ bool SuperWord::same_velt_type(Node* n1, Node* n2) {
}
bool SuperWord::same_memory_slice(MemNode* best_align_to_mem_ref, MemNode* mem_ref) const {
- return _phase->C->get_alias_index(mem_ref->adr_type()) == _phase->C->get_alias_index(best_align_to_mem_ref->adr_type());
+ return phase()->C->get_alias_index(mem_ref->adr_type()) == phase()->C->get_alias_index(best_align_to_mem_ref->adr_type());
}
//------------------------------in_packset---------------------------
@@ -3423,7 +3366,7 @@ void SuperWord::remove_pack_at(int pos) {
Node* s = p->at(i);
set_my_pack(s, nullptr);
}
- _packset.remove_at(pos);
+ _packset.at_put(pos, nullptr);
}
void SuperWord::packset_sort(int n) {
@@ -3479,22 +3422,22 @@ LoadNode::ControlDependency SuperWord::control_dependency(Node_List* p) {
void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
const MemNode* align_to_ref = _align_to_ref;
assert(align_to_ref != nullptr, "align_to_ref must be set");
- assert(lp()->is_main_loop(), "can only do alignment for main loop");
+ assert(cl()->is_main_loop(), "can only do alignment for main loop");
// The opaque node for the limit, where we adjust the input
- Opaque1Node* pre_opaq = lp()->pre_loop_end()->limit()->as_Opaque1();
+ Opaque1Node* pre_opaq = vloop().pre_loop_end()->limit()->as_Opaque1();
// Current pre-loop limit.
Node* old_limit = pre_opaq->in(1);
// Where we put new limit calculations.
- Node* pre_ctrl = lp()->pre_loop_head()->in(LoopNode::EntryControl);
+ Node* pre_ctrl = vloop().pre_loop_head()->in(LoopNode::EntryControl);
// Ensure the original loop limit is available from the pre-loop Opaque1 node.
Node* orig_limit = pre_opaq->original_loop_limit();
- assert(orig_limit != nullptr && _igvn.type(orig_limit) != Type::TOP, "");
+ assert(orig_limit != nullptr && igvn().type(orig_limit) != Type::TOP, "");
- VPointer align_to_ref_p(align_to_ref, phase(), lpt(), nullptr, false);
+ VPointer align_to_ref_p(align_to_ref, vloop());
assert(align_to_ref_p.valid(), "sanity");
// For the main-loop, we want the address of align_to_ref to be memory aligned
@@ -3688,17 +3631,17 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
const bool is_sub = scale * stride > 0;
// 1.1: offset
- Node* xboi = _igvn.intcon(is_sub ? -offset : offset);
+ Node* xboi = igvn().intcon(is_sub ? -offset : offset);
TRACE_ALIGN_VECTOR_NODE(xboi);
// 1.2: invar (if it exists)
if (invar != nullptr) {
- if (_igvn.type(invar)->isa_long()) {
+ if (igvn().type(invar)->isa_long()) {
// Computations are done % (vector width/element size) so it's
// safe to simply convert invar to an int and loose the upper 32
// bit half.
invar = new ConvL2INode(invar);
- _igvn.register_new_node_with_optimizer(invar);
+ igvn().register_new_node_with_optimizer(invar);
TRACE_ALIGN_VECTOR_NODE(invar);
}
if (is_sub) {
@@ -3706,8 +3649,8 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
} else {
xboi = new AddINode(xboi, invar);
}
- _igvn.register_new_node_with_optimizer(xboi);
- _phase->set_ctrl(xboi, pre_ctrl);
+ igvn().register_new_node_with_optimizer(xboi);
+ phase()->set_ctrl(xboi, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(xboi);
}
@@ -3717,11 +3660,11 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
// When the base() is top, we have no alignment guarantee at all.
// Hence, we must now take the base into account for the calculation.
Node* xbase = new CastP2XNode(nullptr, base);
- _igvn.register_new_node_with_optimizer(xbase);
+ igvn().register_new_node_with_optimizer(xbase);
TRACE_ALIGN_VECTOR_NODE(xbase);
#ifdef _LP64
xbase = new ConvL2INode(xbase);
- _igvn.register_new_node_with_optimizer(xbase);
+ igvn().register_new_node_with_optimizer(xbase);
TRACE_ALIGN_VECTOR_NODE(xbase);
#endif
if (is_sub) {
@@ -3729,18 +3672,18 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
} else {
xboi = new AddINode(xboi, xbase);
}
- _igvn.register_new_node_with_optimizer(xboi);
- _phase->set_ctrl(xboi, pre_ctrl);
+ igvn().register_new_node_with_optimizer(xboi);
+ phase()->set_ctrl(xboi, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(xboi);
}
// 2: Compute (14):
// XBOI = xboi / abs(scale)
// The division is executed as shift
- Node* log2_abs_scale = _igvn.intcon(exact_log2(abs(scale)));
+ Node* log2_abs_scale = igvn().intcon(exact_log2(abs(scale)));
Node* XBOI = new URShiftINode(xboi, log2_abs_scale);
- _igvn.register_new_node_with_optimizer(XBOI);
- _phase->set_ctrl(XBOI, pre_ctrl);
+ igvn().register_new_node_with_optimizer(XBOI);
+ phase()->set_ctrl(XBOI, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(log2_abs_scale);
TRACE_ALIGN_VECTOR_NODE(XBOI);
@@ -3754,8 +3697,8 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
} else {
XBOI_OP_old_limit = new AddINode(XBOI, old_limit);
}
- _igvn.register_new_node_with_optimizer(XBOI_OP_old_limit);
- _phase->set_ctrl(XBOI_OP_old_limit, pre_ctrl);
+ igvn().register_new_node_with_optimizer(XBOI_OP_old_limit);
+ phase()->set_ctrl(XBOI_OP_old_limit, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(XBOI_OP_old_limit);
// 3.2: Compute:
@@ -3764,10 +3707,10 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
// = XBOI_OP_old_limit AND (AW - 1)
// Since AW is a power of 2, the modulo operation can be replaced with
// a bitmask operation.
- Node* mask_AW = _igvn.intcon(AW-1);
+ Node* mask_AW = igvn().intcon(AW-1);
Node* adjust_pre_iter = new AndINode(XBOI_OP_old_limit, mask_AW);
- _igvn.register_new_node_with_optimizer(adjust_pre_iter);
- _phase->set_ctrl(adjust_pre_iter, pre_ctrl);
+ igvn().register_new_node_with_optimizer(adjust_pre_iter);
+ phase()->set_ctrl(adjust_pre_iter, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(mask_AW);
TRACE_ALIGN_VECTOR_NODE(adjust_pre_iter);
@@ -3780,8 +3723,8 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
} else {
new_limit = new AddINode(old_limit, adjust_pre_iter);
}
- _igvn.register_new_node_with_optimizer(new_limit);
- _phase->set_ctrl(new_limit, pre_ctrl);
+ igvn().register_new_node_with_optimizer(new_limit);
+ phase()->set_ctrl(new_limit, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(new_limit);
// 5: Compute (15a, b):
@@ -3789,29 +3732,12 @@ void SuperWord::adjust_pre_loop_limit_to_align_main_loop_vectors() {
Node* constrained_limit =
(stride > 0) ? (Node*) new MinINode(new_limit, orig_limit)
: (Node*) new MaxINode(new_limit, orig_limit);
- _igvn.register_new_node_with_optimizer(constrained_limit);
- _phase->set_ctrl(constrained_limit, pre_ctrl);
+ igvn().register_new_node_with_optimizer(constrained_limit);
+ phase()->set_ctrl(constrained_limit, pre_ctrl);
TRACE_ALIGN_VECTOR_NODE(constrained_limit);
// 6: Hack the pre-loop limit
- _igvn.replace_input_of(pre_opaq, 1, constrained_limit);
-}
-
-//------------------------------init---------------------------
-void SuperWord::init() {
- _dg.init();
- _packset.clear();
- _disjoint_ptrs.clear();
- _block.clear();
- _data_entry.clear();
- _mem_slice_head.clear();
- _mem_slice_tail.clear();
- _node_info.clear();
- _align_to_ref = nullptr;
- _race_possible = 0;
- _early_return = false;
- _num_work_vecs = 0;
- _num_reductions = 0;
+ igvn().replace_input_of(pre_opaq, 1, constrained_limit);
}
//------------------------------print_packset---------------------------
@@ -3859,10 +3785,6 @@ void SuperWord::print_stmt(Node* s) {
#endif
}
-// ========================= OrderedPair =====================
-
-const OrderedPair OrderedPair::initial;
-
// ========================= SWNodeInfo =====================
const SWNodeInfo SWNodeInfo::initial;
diff --git a/src/hotspot/share/opto/superword.hpp b/src/hotspot/share/opto/superword.hpp
index db7101b26cb..8813284e351 100644
--- a/src/hotspot/share/opto/superword.hpp
+++ b/src/hotspot/share/opto/superword.hpp
@@ -57,7 +57,6 @@
// second statement is considered the right element.
class VPointer;
-class OrderedPair;
// ========================= Dependence Graph =====================
@@ -140,8 +139,6 @@ class DepGraph {
DepEdge* make_edge(DepMem* pred, Node* succ) { return make_edge(pred, dep(succ)); }
DepEdge* make_edge(Node* pred, DepMem* succ) { return make_edge(dep(pred), succ); }
- void init() { _map.clear(); } // initialize
-
void print(Node* n) { dep(n)->print(); }
void print(DepMem* d) { d->print(); }
};
@@ -198,123 +195,134 @@ class SWNodeInfo {
static const SWNodeInfo initial;
};
-class SuperWord;
-
-// JVMCI: OrderedPair is moved up to deal with compilation issues on Windows
-//------------------------------OrderedPair---------------------------
-// Ordered pair of Node*.
-class OrderedPair {
- protected:
- Node* _p1;
- Node* _p2;
- public:
- OrderedPair() : _p1(nullptr), _p2(nullptr) {}
- OrderedPair(Node* p1, Node* p2) {
- if (p1->_idx < p2->_idx) {
- _p1 = p1; _p2 = p2;
- } else {
- _p1 = p2; _p2 = p1;
- }
- }
-
- bool operator==(const OrderedPair &rhs) {
- return _p1 == rhs._p1 && _p2 == rhs._p2;
- }
- void print() { tty->print(" (%d, %d)", _p1->_idx, _p2->_idx); }
-
- static const OrderedPair initial;
-};
-
// -----------------------------SuperWord---------------------------------
// Transforms scalar operations into packed (superword) operations.
class SuperWord : public ResourceObj {
- friend class VPointer;
- friend class CMoveKit;
private:
- PhaseIdealLoop* _phase;
- Arena* _arena;
- PhaseIterGVN &_igvn;
+ const VLoop& _vloop;
+
+ // Arena for small data structures. Large data structures are allocated in
+ // VSharedData, and reused over many AutoVectorizations.
+ Arena _arena;
enum consts { top_align = -1, bottom_align = -666 };
GrowableArray _packset; // Packs for the current block
- GrowableArray _bb_idx; // Map from Node _idx to index within block
+ GrowableArray &_bb_idx; // Map from Node _idx to index within block
GrowableArray _block; // Nodes in current block
- GrowableArray _data_entry; // Nodes with all inputs from outside
- GrowableArray _mem_slice_head; // Memory slice head nodes
- GrowableArray _mem_slice_tail; // Memory slice tail nodes
+ GrowableArray _mem_slice_head; // Memory slice head nodes
+ GrowableArray _mem_slice_tail; // Memory slice tail nodes
GrowableArray _node_info; // Info needed per node
CloneMap& _clone_map; // map of nodes created in cloning
MemNode const* _align_to_ref; // Memory reference that pre-loop will align to
- GrowableArray _disjoint_ptrs; // runtime disambiguated pointer pairs
-
DepGraph _dg; // Dependence graph
// Scratch pads
- VectorSet _visited; // Visited set
- VectorSet _post_visited; // Post-visited set
- Node_Stack _n_idx_list; // List of (node,index) pairs
GrowableArray _nlist; // List of nodes
- GrowableArray _stk; // Stack of nodes
public:
- SuperWord(PhaseIdealLoop* phase);
-
- bool transform_loop(IdealLoopTree* lpt, bool do_optimization);
+ SuperWord(const VLoop &vloop, VSharedData &vshared);
- void unrolling_analysis(int &local_loop_unroll_factor);
+ // Attempt to run the SuperWord algorithm on the loop. Return true if we succeed.
+ bool transform_loop();
- // Accessors for VPointer
- PhaseIdealLoop* phase() const { return _phase; }
- IdealLoopTree* lpt() const { return _lpt; }
- PhiNode* iv() const { return _iv; }
+ // Decide if loop can eventually be vectorized, and what unrolling factor is required.
+ static void unrolling_analysis(const VLoop &vloop, int &local_loop_unroll_factor);
- bool early_return() const { return _early_return; }
+ // VLoop Accessors
+ const VLoop& vloop() const { return _vloop; }
+ PhaseIdealLoop* phase() const { return vloop().phase(); }
+ PhaseIterGVN& igvn() const { return vloop().phase()->igvn(); }
+ IdealLoopTree* lpt() const { return vloop().lpt(); }
+ CountedLoopNode* cl() const { return vloop().cl(); }
+ PhiNode* iv() const { return vloop().iv(); }
+ int iv_stride() const { return cl()->stride_con(); }
+ bool in_bb(const Node* n) const { return vloop().in_bb(n); }
#ifndef PRODUCT
- bool is_debug() { return _vector_loop_debug > 0; }
- bool is_trace_alignment() { return (_vector_loop_debug & 2) > 0; }
- bool is_trace_mem_slice() { return (_vector_loop_debug & 4) > 0; }
- bool is_trace_loop() { return (_vector_loop_debug & 8) > 0; }
- bool is_trace_adjacent() { return (_vector_loop_debug & 16) > 0; }
- bool is_trace_cmov() { return (_vector_loop_debug & 32) > 0; }
- bool is_trace_align_vector() { return (_vector_loop_debug & 128) > 0; }
+ // TraceAutoVectorization and TraceSuperWord
+ bool is_trace_superword_vector_element_type() const {
+ // Too verbose for TraceSuperWord
+ return vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_TYPES);
+ }
+
+ bool is_trace_superword_alignment() const {
+ // Too verbose for TraceSuperWord
+ return vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_ALIGNMENT);
+ }
+
+ bool is_trace_superword_memory_slices() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_MEMORY_SLICES);
+ }
+
+ bool is_trace_superword_dependence_graph() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_DEPENDENCE_GRAPH);
+ }
+
+ bool is_trace_superword_adjacent_memops() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_ADJACENT_MEMOPS);
+ }
+
+ bool is_trace_superword_rejections() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_REJECTIONS);
+ }
+
+ bool is_trace_superword_packset() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_PACKSET);
+ }
+
+ bool is_trace_superword_info() const {
+ return TraceSuperWord ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_INFO);
+ }
+
+ bool is_trace_superword_verbose() const {
+ // Too verbose for TraceSuperWord
+ return vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_VERBOSE);
+ }
+
+ bool is_trace_superword_any() const {
+ return TraceSuperWord ||
+ is_trace_align_vector() ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_TYPES) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_ALIGNMENT) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_MEMORY_SLICES) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_DEPENDENCE_GRAPH) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_ADJACENT_MEMOPS) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_REJECTIONS) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_PACKSET) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_INFO) ||
+ vloop().vtrace().is_trace(TraceAutoVectorizationTag::SW_VERBOSE);
+ }
+
+ bool is_trace_align_vector() const {
+ return vloop().vtrace().is_trace(TraceAutoVectorizationTag::ALIGN_VECTOR) ||
+ is_trace_superword_verbose();
+ }
#endif
+
bool do_vector_loop() { return _do_vector_loop; }
const GrowableArray& packset() const { return _packset; }
const GrowableArray& block() const { return _block; }
const DepGraph& dg() const { return _dg; }
private:
- IdealLoopTree* _lpt; // Current loop tree node
- CountedLoopNode* _lp; // Current CountedLoopNode
VectorSet _loop_reductions; // Reduction nodes in the current loop
- Node* _bb; // Current basic block
- PhiNode* _iv; // Induction var
bool _race_possible; // In cases where SDMU is true
- bool _early_return; // True if we do not initialize
bool _do_vector_loop; // whether to do vectorization/simd style
int _num_work_vecs; // Number of non memory vector operations
int _num_reductions; // Number of reduction expressions applied
-#ifndef PRODUCT
- uintx _vector_loop_debug; // provide more printing in debug mode
-#endif
// Accessors
- Arena* arena() { return _arena; }
-
- Node* bb() { return _bb; }
- void set_bb(Node* bb) { _bb = bb; }
- void set_lpt(IdealLoopTree* lpt) { _lpt = lpt; }
- CountedLoopNode* lp() const { return _lp; }
- void set_lp(CountedLoopNode* lp) {
- _lp = lp;
- _iv = lp->as_CountedLoop()->phi()->as_Phi();
- }
- int iv_stride() const { return lp()->stride_con(); }
+ Arena* arena() { return &_arena; }
int vector_width(const Node* n) const {
BasicType bt = velt_basic_type(n);
@@ -328,24 +336,12 @@ class SuperWord : public ResourceObj {
const MemNode* align_to_ref() const { return _align_to_ref; }
void set_align_to_ref(const MemNode* m) { _align_to_ref = m; }
- const Node* ctrl(const Node* n) const { return _phase->has_ctrl(n) ? _phase->get_ctrl(n) : n; }
-
// block accessors
public:
- bool in_bb(const Node* n) const { return n != nullptr && n->outcnt() > 0 && ctrl(n) == _bb; }
int bb_idx(const Node* n) const { assert(in_bb(n), "must be"); return _bb_idx.at(n->_idx); }
private:
void set_bb_idx(Node* n, int i) { _bb_idx.at_put_grow(n->_idx, i); }
- // visited set accessors
- void visited_clear() { _visited.clear(); }
- void visited_set(Node* n) { return _visited.set(bb_idx(n)); }
- int visited_test(Node* n) { return _visited.test(bb_idx(n)); }
- int visited_test_set(Node* n) { return _visited.test_set(bb_idx(n)); }
- void post_visited_clear() { _post_visited.clear(); }
- void post_visited_set(Node* n) { return _post_visited.set(bb_idx(n)); }
- int post_visited_test(Node* n) { return _post_visited.test(bb_idx(n)); }
-
// Ensure node_info contains element "i"
void grow_node_info(int i) { if (i >= _node_info.length()) _node_info.at_put_grow(i, SWNodeInfo::initial); }
@@ -357,7 +353,7 @@ class SuperWord : public ResourceObj {
void set_alignment(Node* n, int a) { int i = bb_idx(n); grow_node_info(i); _node_info.adr_at(i)->_alignment = a; }
// Max expression (DAG) depth from beginning of the block for each node
- int depth(Node* n) { return _node_info.adr_at(bb_idx(n))->_depth; }
+ int depth(Node* n) const { return _node_info.adr_at(bb_idx(n))->_depth; }
void set_depth(Node* n, int d) { int i = bb_idx(n); grow_node_info(i); _node_info.adr_at(i)->_depth = d; }
// vector element type
@@ -375,7 +371,7 @@ class SuperWord : public ResourceObj {
// is pack good for converting into one vector node replacing bunches of Cmp, Bool, CMov nodes.
static bool requires_long_to_int_conversion(int opc);
// For pack p, are all idx operands the same?
- bool same_inputs(Node_List* p, int idx);
+ bool same_inputs(const Node_List* p, int idx);
// CloneMap utilities
bool same_origin_idx(Node* a, Node* b) const;
bool same_generation(Node* a, Node* b) const;
@@ -448,8 +444,13 @@ class SuperWord : public ResourceObj {
int get_iv_adjustment(MemNode* mem);
// Construct dependency graph.
void dependence_graph();
+
+ // Analyze the memory slices
+ void find_memory_slices();
+ NOT_PRODUCT( void print_memory_slices(); )
// Return a memory slice (node list) in predecessor order starting at "start"
void mem_slice_preds(Node* start, Node* stop, GrowableArray &preds);
+
// Can s1 and s2 be in a pack with s1 immediately preceding s2 and s1 aligned at "align"
bool stmts_can_pack(Node* s1, Node* s2, int align);
// Does s exist in a pack at position pos?
@@ -460,19 +461,17 @@ class SuperWord : public ResourceObj {
bool isomorphic(Node* s1, Node* s2);
// Is there no data path from s1 to s2 or s2 to s1?
bool independent(Node* s1, Node* s2);
- // Is any s1 in p dependent on any s2 in p? Yes: return such a s2. No: return nullptr.
- Node* find_dependence(Node_List* p);
+ // Are all nodes in nodes list mutually independent?
+ bool mutually_independent(const Node_List* nodes) const;
// For a node pair (s1, s2) which is isomorphic and independent,
// do s1 and s2 have similar input edges?
bool have_similar_inputs(Node* s1, Node* s2);
// Is there a data path between s1 and s2 and both are reductions?
bool reduction(Node* s1, Node* s2);
- // Helper for independent
- bool independent_path(Node* shallow, Node* deep, uint dp=0);
void set_alignment(Node* s1, Node* s2, int align);
int data_size(Node* s);
// Extend packset by following use->def and def->use links from pack members.
- void extend_packlist();
+ void extend_packset_with_more_pairs_by_following_use_and_def();
int adjust_alignment_for_type_conversion(Node* s, Node* t, int align);
// Extend the packset by visiting operand definitions of nodes in pack p
bool follow_use_defs(Node_List* p);
@@ -485,18 +484,31 @@ class SuperWord : public ResourceObj {
int adjacent_profit(Node* s1, Node* s2);
int pack_cost(int ct);
int unpack_cost(int ct);
+
// Combine packs A and B with A.last == B.first into A.first..,A.last,B.second,..B.last
- void combine_packs();
+ void combine_pairs_to_longer_packs();
+
+ void split_packs_longer_than_max_vector_size();
+
+ // Filter out packs with various filter predicates
+ template
+ void filter_packs(const char* filter_name,
+ const char* error_message,
+ FilterPredicate filter);
+ void filter_packs_for_power_of_2_size();
+ void filter_packs_for_mutual_independence();
// Ensure all packs are aligned, if AlignVector is on.
void filter_packs_for_alignment();
// Find the set of alignment solutions for load/store pack.
- const AlignmentSolution* pack_alignment_solution(Node_List* pack);
+ const AlignmentSolution* pack_alignment_solution(const Node_List* pack);
// Compress packset, such that it has no nullptr entries.
void compress_packset();
// Construct the map from nodes to packs.
void construct_my_pack_map();
- // Remove packs that are not implemented or not profitable.
- void filter_packs();
+ // Remove packs that are not implemented.
+ void filter_packs_for_implemented();
+ // Remove packs that are not profitable.
+ void filter_packs_for_profitable();
// Verify that for every pack, all nodes are mutually independent.
// Also verify that packset and my_pack are consistent.
DEBUG_ONLY(void verify_packs();)
@@ -510,19 +522,17 @@ class SuperWord : public ResourceObj {
// Create a vector operand for the nodes in pack p for operand: in(opd_idx)
Node* vector_opd(Node_List* p, int opd_idx);
// Can code be generated for pack p?
- bool implemented(Node_List* p);
+ bool implemented(const Node_List* p);
// For pack p, are all operands and all uses (with in the block) vector?
- bool profitable(Node_List* p);
- // If a use of pack p is not a vector use, then replace the use with an extract operation.
- void insert_extracts(Node_List* p);
+ bool profitable(const Node_List* p);
+ // Verify that all uses of packs are also packs, i.e. we do not need extract operations.
+ DEBUG_ONLY(void verify_no_extract();)
// Is use->in(u_idx) a vector use?
bool is_vector_use(Node* use, int u_idx);
// Construct reverse postorder list of block members
bool construct_bb();
// Initialize per node info
- void initialize_bb();
- // Insert n into block after pos
- void bb_insert_after(Node* n, int pos);
+ void initialize_node_info();
// Compute max depth for expressions from beginning of block
void compute_max_depth();
// Return the longer type for vectorizable type-conversion node or illegal type for other nodes.
@@ -544,7 +554,6 @@ class SuperWord : public ResourceObj {
void adjust_pre_loop_limit_to_align_main_loop_vectors();
// Is the use of d1 in u1 at the same operand position as d2 in u2?
bool opnd_positions_match(Node* d1, Node* u1, Node* d2, Node* u2);
- void init();
// print methods
void print_packset();
diff --git a/src/hotspot/share/opto/traceAutoVectorizationTag.hpp b/src/hotspot/share/opto/traceAutoVectorizationTag.hpp
new file mode 100644
index 00000000000..79157aca309
--- /dev/null
+++ b/src/hotspot/share/opto/traceAutoVectorizationTag.hpp
@@ -0,0 +1,163 @@
+/*
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ *
+ */
+
+#ifndef SHARE_OPTO_TRACEAUTOVECTORIZATIONTAG_HPP
+#define SHARE_OPTO_TRACEAUTOVECTORIZATIONTAG_HPP
+
+#include "utilities/bitMap.inline.hpp"
+#include "utilities/stringUtils.hpp"
+
+#define COMPILER_TRACE_AUTO_VECTORIZATION_TAG(flags) \
+ flags(POINTER_ANALYSIS, "Trace VPointer") \
+ flags(PRECONDITIONS, "Trace VLoop::check_preconditions") \
+ flags(SW_TYPES, "Trace SuperWord::compute_vector_element_type") \
+ flags(SW_ALIGNMENT, "Trace SuperWord alignment analysis") \
+ flags(SW_MEMORY_SLICES, "Trace SuperWord memory slices") \
+ flags(SW_DEPENDENCE_GRAPH, "Trace SuperWord::dependence_graph") \
+ flags(SW_ADJACENT_MEMOPS, "Trace SuperWord::find_adjacent_refs") \
+ flags(SW_REJECTIONS, "Trace SuperWord rejections (non vectorizations)") \
+ flags(SW_PACKSET, "Trace SuperWord packset at different stages") \
+ flags(SW_INFO, "Trace SuperWord info (equivalent to TraceSuperWord)") \
+ flags(SW_VERBOSE, "Trace SuperWord verbose (all SW tags enabled)") \
+ flags(ALIGN_VECTOR, "Trace AlignVector") \
+ flags(ALL, "Trace everything (very verbose)")
+
+#define table_entry(name, description) name,
+enum TraceAutoVectorizationTag {
+ COMPILER_TRACE_AUTO_VECTORIZATION_TAG(table_entry)
+ TRACE_AUTO_VECTORIZATION_TAG_NUM,
+ TRACE_AUTO_VECTORIZATION_TAG_NONE
+};
+#undef table_entry
+
+static const char* tag_descriptions[] = {
+#define array_of_labels(name, description) description,
+ COMPILER_TRACE_AUTO_VECTORIZATION_TAG(array_of_labels)
+#undef array_of_labels
+};
+
+static const char* tag_names[] = {
+#define array_of_labels(name, description) #name,
+ COMPILER_TRACE_AUTO_VECTORIZATION_TAG(array_of_labels)
+#undef array_of_labels
+};
+
+static TraceAutoVectorizationTag find_tag(const char* str) {
+ for (int i = 0; i < TRACE_AUTO_VECTORIZATION_TAG_NUM; i++) {
+ if (strcmp(tag_names[i], str) == 0) {
+ return (TraceAutoVectorizationTag)i;
+ }
+ }
+ return TRACE_AUTO_VECTORIZATION_TAG_NONE;
+}
+
+class TraceAutoVectorizationTagValidator {
+ private:
+ CHeapBitMap _tags;
+ bool _valid;
+ char* _bad;
+ bool _is_print_usage;
+
+ public:
+ TraceAutoVectorizationTagValidator(ccstrlist option, bool is_print_usage) :
+ _tags(TRACE_AUTO_VECTORIZATION_TAG_NUM, mtCompiler),
+ _valid(true),
+ _bad(nullptr),
+ _is_print_usage(is_print_usage)
+ {
+ for (StringUtils::CommaSeparatedStringIterator iter(option); *iter != nullptr && _valid; ++iter) {
+ char const* tag_name = *iter;
+ if (strcmp("help", tag_name) == 0) {
+ if (_is_print_usage) {
+ print_help();
+ }
+ continue;
+ }
+ bool set_bit = true;
+ // Check for "TAG" or "-TAG"
+ if (strncmp("-", tag_name, strlen("-")) == 0) {
+ tag_name++;
+ set_bit = false;
+ }
+ TraceAutoVectorizationTag tag = find_tag(tag_name);
+ if (TRACE_AUTO_VECTORIZATION_TAG_NONE == tag) {
+ // cap len to a value we know is enough for all tags
+ const size_t len = MIN2(strlen(*iter), 63) + 1;
+ _bad = NEW_C_HEAP_ARRAY(char, len, mtCompiler);
+ // strncpy always writes len characters. If the source string is
+ // shorter, the function fills the remaining bytes with nulls.
+ strncpy(_bad, *iter, len);
+ _valid = false;
+ } else if (ALL == tag) {
+ _tags.set_range(0, TRACE_AUTO_VECTORIZATION_TAG_NUM);
+ } else if (SW_VERBOSE == tag) {
+ _tags.at_put(SW_TYPES, set_bit);
+ _tags.at_put(SW_ALIGNMENT, set_bit);
+ _tags.at_put(SW_MEMORY_SLICES, set_bit);
+ _tags.at_put(SW_DEPENDENCE_GRAPH, set_bit);
+ _tags.at_put(SW_ADJACENT_MEMOPS, set_bit);
+ _tags.at_put(SW_REJECTIONS, set_bit);
+ _tags.at_put(SW_PACKSET, set_bit);
+ _tags.at_put(SW_INFO, set_bit);
+ _tags.at_put(SW_VERBOSE, set_bit);
+ } else if (SW_INFO == tag) {
+ _tags.at_put(SW_MEMORY_SLICES, set_bit);
+ _tags.at_put(SW_DEPENDENCE_GRAPH, set_bit);
+ _tags.at_put(SW_ADJACENT_MEMOPS, set_bit);
+ _tags.at_put(SW_REJECTIONS, set_bit);
+ _tags.at_put(SW_PACKSET, set_bit);
+ _tags.at_put(SW_INFO, set_bit);
+ } else {
+ assert(tag < TRACE_AUTO_VECTORIZATION_TAG_NUM, "out of bounds");
+ _tags.at_put(tag, set_bit);
+ }
+ }
+ }
+
+ ~TraceAutoVectorizationTagValidator() {
+ if (_bad != nullptr) {
+ FREE_C_HEAP_ARRAY(char, _bad);
+ }
+ }
+
+ bool is_valid() const { return _valid; }
+ const char* what() const { return _bad; }
+ const CHeapBitMap& tags() const {
+ assert(is_valid(), "only read tags when valid");
+ return _tags;
+ }
+
+ static void print_help() {
+ tty->cr();
+ tty->print_cr("Usage for CompileCommand TraceAutoVectorization:");
+ tty->print_cr(" -XX:CompileCommand=TraceAutoVectorization,,");
+ tty->print_cr(" %-22s %s", "tags", "descriptions");
+ for (int i = 0; i < TRACE_AUTO_VECTORIZATION_TAG_NUM; i++) {
+ tty->print_cr(" %-22s %s", tag_names[i], tag_descriptions[i]);
+ }
+ tty->cr();
+ }
+};
+
+#endif // SHARE_OPTO_TRACEAUTOVECTORIZATIONTAG_HPP
diff --git a/src/hotspot/share/opto/vector.cpp b/src/hotspot/share/opto/vector.cpp
index 641e7ebcde8..1b4d64e20b1 100644
--- a/src/hotspot/share/opto/vector.cpp
+++ b/src/hotspot/share/opto/vector.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -276,7 +276,7 @@ void PhaseVector::scalarize_vbox_node(VectorBoxNode* vec_box) {
SafePointNode* sfpt = safepoints.pop()->as_SafePoint();
uint first_ind = (sfpt->req() - sfpt->jvms()->scloff());
- Node* sobj = new SafePointScalarObjectNode(vec_box->box_type(), vec_box, first_ind, n_fields);
+ Node* sobj = new SafePointScalarObjectNode(vec_box->box_type(), vec_box, first_ind, sfpt->jvms()->depth(), n_fields);
sobj->init_req(0, C->root());
sfpt->add_req(vec_value);
diff --git a/src/hotspot/share/opto/vectorization.cpp b/src/hotspot/share/opto/vectorization.cpp
index 4794140ab0e..a8374cbcd20 100644
--- a/src/hotspot/share/opto/vectorization.cpp
+++ b/src/hotspot/share/opto/vectorization.cpp
@@ -31,22 +31,103 @@
#include "opto/rootnode.hpp"
#include "opto/vectorization.hpp"
+bool VLoop::check_preconditions() {
+#ifndef PRODUCT
+ if (is_trace_preconditions()) {
+ tty->print_cr("\nVLoop::check_preconditions");
+ lpt()->dump_head();
+ lpt()->head()->dump();
+ }
+#endif
+
+ const char* return_state = check_preconditions_helper();
+ assert(return_state != nullptr, "must have return state");
+ if (return_state == VLoop::SUCCESS) {
+ return true; // success
+ }
+
+#ifndef PRODUCT
+ if (is_trace_preconditions()) {
+ tty->print_cr("VLoop::check_preconditions: failed: %s", return_state);
+ }
+#endif
+ return false; // failure
+}
+
+const char* VLoop::check_preconditions_helper() {
+ // Only accept vector width that is power of 2
+ int vector_width = Matcher::vector_width_in_bytes(T_BYTE);
+ if (vector_width < 2 || !is_power_of_2(vector_width)) {
+ return VLoop::FAILURE_VECTOR_WIDTH;
+ }
+
+ // Only accept valid counted loops (int)
+ if (!_lpt->_head->as_Loop()->is_valid_counted_loop(T_INT)) {
+ return VLoop::FAILURE_VALID_COUNTED_LOOP;
+ }
+ _cl = _lpt->_head->as_CountedLoop();
+ _iv = _cl->phi()->as_Phi();
+
+ if (_cl->is_vectorized_loop()) {
+ return VLoop::FAILURE_ALREADY_VECTORIZED;
+ }
+
+ if (_cl->is_unroll_only()) {
+ return VLoop::FAILURE_UNROLL_ONLY;
+ }
+
+ // Check for control flow in the body
+ _cl_exit = _cl->loopexit();
+ bool has_cfg = _cl_exit->in(0) != _cl;
+ if (has_cfg && !is_allow_cfg()) {
+#ifndef PRODUCT
+ if (is_trace_preconditions()) {
+ tty->print_cr("VLoop::check_preconditions: fails because of control flow.");
+ tty->print(" cl_exit %d", _cl_exit->_idx); _cl_exit->dump();
+ tty->print(" cl_exit->in(0) %d", _cl_exit->in(0)->_idx); _cl_exit->in(0)->dump();
+ tty->print(" lpt->_head %d", _cl->_idx); _cl->dump();
+ _lpt->dump_head();
+ }
+#endif
+ return VLoop::FAILURE_CONTROL_FLOW;
+ }
+
+ // Make sure the are no extra control users of the loop backedge
+ if (_cl->back_control()->outcnt() != 1) {
+ return VLoop::FAILURE_BACKEDGE;
+ }
+
+ // To align vector memory accesses in the main-loop, we will have to adjust
+ // the pre-loop limit.
+ if (_cl->is_main_loop()) {
+ CountedLoopEndNode* pre_end = _cl->find_pre_loop_end();
+ if (pre_end == nullptr) {
+ return VLoop::FAILURE_PRE_LOOP_LIMIT;
+ }
+ Node* pre_opaq1 = pre_end->limit();
+ if (pre_opaq1->Opcode() != Op_Opaque1) {
+ return VLoop::FAILURE_PRE_LOOP_LIMIT;
+ }
+ _pre_loop_end = pre_end;
+ }
+
+ return VLoop::SUCCESS;
+}
+
#ifndef PRODUCT
int VPointer::Tracer::_depth = 0;
#endif
-VPointer::VPointer(const MemNode* mem,
- PhaseIdealLoop* phase, IdealLoopTree* lpt,
+VPointer::VPointer(const MemNode* mem, const VLoop& vloop,
Node_Stack* nstack, bool analyze_only) :
- _mem(mem), _phase(phase), _lpt(lpt),
- _iv(lpt->_head->as_CountedLoop()->phi()->as_Phi()),
+ _mem(mem), _vloop(vloop),
_base(nullptr), _adr(nullptr), _scale(0), _offset(0), _invar(nullptr),
#ifdef ASSERT
_debug_invar(nullptr), _debug_negate_invar(false), _debug_invar_scale(nullptr),
#endif
_nstack(nstack), _analyze_only(analyze_only), _stack_idx(0)
#ifndef PRODUCT
- , _tracer((phase->C->directive()->VectorizeDebugOption & 2) > 0)
+ , _tracer(vloop.is_trace_pointer_analysis())
#endif
{
NOT_PRODUCT(_tracer.ctor_1(mem);)
@@ -109,7 +190,7 @@ VPointer::VPointer(const MemNode* mem,
// Following is used to create a temporary object during
// the pattern match of an address expression.
VPointer::VPointer(VPointer* p) :
- _mem(p->_mem), _phase(p->_phase), _lpt(p->_lpt), _iv(p->_iv),
+ _mem(p->_mem), _vloop(p->_vloop),
_base(nullptr), _adr(nullptr), _scale(0), _offset(0), _invar(nullptr),
#ifdef ASSERT
_debug_invar(nullptr), _debug_negate_invar(false), _debug_invar_scale(nullptr),
@@ -153,7 +234,7 @@ bool VPointer::invariant(Node* n) const {
// main loop (Illegal invariant happens when n_c is a CastII node that
// prevents data nodes to flow above the main loop).
Node* n_c = phase()->get_ctrl(n);
- return phase()->is_dominator(n_c, cl->pre_loop_head());
+ return phase()->is_dominator(n_c, vloop().pre_loop_head());
}
}
return is_not_member;
@@ -1191,7 +1272,7 @@ AlignmentSolution* AlignmentSolver::solve() const {
}
#ifdef ASSERT
-void print_con_or_idx(const Node* n) {
+static void print_con_or_idx(const Node* n) {
if (n == nullptr) {
tty->print("(0)");
} else if (n->is_ConI()) {
diff --git a/src/hotspot/share/opto/vectorization.hpp b/src/hotspot/share/opto/vectorization.hpp
index 8e63b40d5ac..7aff58db4bb 100644
--- a/src/hotspot/share/opto/vectorization.hpp
+++ b/src/hotspot/share/opto/vectorization.hpp
@@ -27,19 +27,152 @@
#include "opto/node.hpp"
#include "opto/loopnode.hpp"
+#include "opto/traceAutoVectorizationTag.hpp"
// Code in this file and the vectorization.cpp contains shared logics and
// utilities for C2's loop auto-vectorization.
+#ifndef PRODUCT
+// Access to TraceAutoVectorization tags
+class VTrace : public StackObj {
+private:
+ const CHeapBitMap &_trace_tags;
+
+public:
+ VTrace() : _trace_tags(Compile::current()->directive()->trace_auto_vectorization_tags()) {}
+ NONCOPYABLE(VTrace);
+
+ bool is_trace(TraceAutoVectorizationTag tag) const {
+ return _trace_tags.at(tag);
+ }
+};
+#endif
+
+// Basic loop structure accessors and vectorization preconditions checking
+class VLoop : public StackObj {
+private:
+ PhaseIdealLoop* const _phase;
+ IdealLoopTree* const _lpt;
+ const bool _allow_cfg;
+ CountedLoopNode* _cl;
+ Node* _cl_exit;
+ PhiNode* _iv;
+ CountedLoopEndNode* _pre_loop_end; // cache access to pre-loop for main loops only
+
+ NOT_PRODUCT(VTrace _vtrace;)
+
+ static constexpr char const* SUCCESS = "success";
+ static constexpr char const* FAILURE_ALREADY_VECTORIZED = "loop already vectorized";
+ static constexpr char const* FAILURE_UNROLL_ONLY = "loop only wants to be unrolled";
+ static constexpr char const* FAILURE_VECTOR_WIDTH = "vector_width must be power of 2";
+ static constexpr char const* FAILURE_VALID_COUNTED_LOOP = "must be valid counted loop (int)";
+ static constexpr char const* FAILURE_CONTROL_FLOW = "control flow in loop not allowed";
+ static constexpr char const* FAILURE_BACKEDGE = "nodes on backedge not allowed";
+ static constexpr char const* FAILURE_PRE_LOOP_LIMIT = "main-loop must be able to adjust pre-loop-limit (not found)";
+
+public:
+ VLoop(IdealLoopTree* lpt, bool allow_cfg) :
+ _phase (lpt->_phase),
+ _lpt (lpt),
+ _allow_cfg (allow_cfg),
+ _cl (nullptr),
+ _cl_exit (nullptr),
+ _iv (nullptr) {}
+ NONCOPYABLE(VLoop);
+
+ IdealLoopTree* lpt() const { return _lpt; };
+ PhaseIdealLoop* phase() const { return _phase; }
+ CountedLoopNode* cl() const { return _cl; };
+ Node* cl_exit() const { return _cl_exit; };
+ PhiNode* iv() const { return _iv; };
+ int iv_stride() const { return cl()->stride_con(); };
+ bool is_allow_cfg() const { return _allow_cfg; }
+
+ CountedLoopEndNode* pre_loop_end() const {
+ assert(cl()->is_main_loop(), "only main loop can reference pre-loop");
+ assert(_pre_loop_end != nullptr, "must have found it");
+ return _pre_loop_end;
+ };
+
+ CountedLoopNode* pre_loop_head() const {
+ CountedLoopNode* head = pre_loop_end()->loopnode();
+ assert(head != nullptr, "must find head");
+ return head;
+ };
+
+ // Estimate maximum size for data structures, to avoid repeated reallocation
+ int estimated_body_length() const { return lpt()->_body.size(); };
+ int estimated_node_count() const { return (int)(1.10 * phase()->C->unique()); };
+
+#ifndef PRODUCT
+ const VTrace& vtrace() const { return _vtrace; }
+
+ bool is_trace_preconditions() const {
+ return vtrace().is_trace(TraceAutoVectorizationTag::PRECONDITIONS);
+ }
+
+ bool is_trace_pointer_analysis() const {
+ return vtrace().is_trace(TraceAutoVectorizationTag::POINTER_ANALYSIS);
+ }
+#endif
+
+ // Is the node in the basic block of the loop?
+ // We only accept any nodes which have the loop head as their ctrl.
+ bool in_bb(const Node* n) const {
+ const Node* ctrl = _phase->has_ctrl(n) ? _phase->get_ctrl(n) : n;
+ return n != nullptr && n->outcnt() > 0 && ctrl == _cl;
+ }
+
+ // Check if the loop passes some basic preconditions for vectorization.
+ // Return indicates if analysis succeeded.
+ bool check_preconditions();
+
+private:
+ const char* check_preconditions_helper();
+};
+
+// Optimization to keep allocation of large arrays in AutoVectorization low.
+// We allocate the arrays once, and reuse them for multiple loops that we
+// AutoVectorize, clearing them before every new use.
+class VSharedData : public StackObj {
+private:
+ // Arena, used to allocate all arrays from.
+ Arena _arena;
+
+ // An array that maps node->_idx to a much smaller idx, which is at most the
+ // size of a loop body. This allow us to have smaller arrays for other data
+ // structures, since we are using smaller indices.
+ GrowableArray _node_idx_to_loop_body_idx;
+
+public:
+ VSharedData() :
+ _arena(mtCompiler),
+ _node_idx_to_loop_body_idx(&_arena, estimated_node_count(), 0, 0)
+ {
+ }
+
+ GrowableArray& node_idx_to_loop_body_idx() {
+ return _node_idx_to_loop_body_idx;
+ }
+
+ // Must be cleared before each AutoVectorization use
+ void clear() {
+ _node_idx_to_loop_body_idx.clear();
+ }
+
+private:
+ static int estimated_node_count() {
+ return (int)(1.10 * Compile::current()->unique());
+ }
+};
+
// A vectorization pointer (VPointer) has information about an address for
// dependence checking and vector alignment. It's usually bound to a memory
// operation in a counted loop for vectorizable analysis.
class VPointer : public ArenaObj {
protected:
const MemNode* _mem; // My memory reference node
- PhaseIdealLoop* _phase; // PhaseIdealLoop handle
- IdealLoopTree* _lpt; // Current IdealLoopTree
- PhiNode* _iv; // The loop induction variable
+ const VLoop& _vloop;
Node* _base; // null if unsafe nonheap reference
Node* _adr; // address pointer
@@ -57,9 +190,10 @@ class VPointer : public ArenaObj {
bool _analyze_only; // Used in loop unrolling only for vpointer trace
uint _stack_idx; // Used in loop unrolling only for vpointer trace
- PhaseIdealLoop* phase() const { return _phase; }
- IdealLoopTree* lpt() const { return _lpt; }
- PhiNode* iv() const { return _iv; }
+ const VLoop& vloop() const { return _vloop; }
+ PhaseIdealLoop* phase() const { return vloop().phase(); }
+ IdealLoopTree* lpt() const { return vloop().lpt(); }
+ PhiNode* iv() const { return vloop().iv(); }
bool is_loop_member(Node* n) const;
bool invariant(Node* n) const;
@@ -80,13 +214,19 @@ class VPointer : public ArenaObj {
NotComparable = (Less | Greater | Equal)
};
- VPointer(const MemNode* mem,
- PhaseIdealLoop* phase, IdealLoopTree* lpt,
+ VPointer(const MemNode* mem, const VLoop& vloop) :
+ VPointer(mem, vloop, nullptr, false) {}
+ VPointer(const MemNode* mem, const VLoop& vloop, Node_Stack* nstack) :
+ VPointer(mem, vloop, nstack, true) {}
+ private:
+ VPointer(const MemNode* mem, const VLoop& vloop,
Node_Stack* nstack, bool analyze_only);
// Following is used to create a temporary object during
// the pattern match of an address expression.
VPointer(VPointer* p);
+ NONCOPYABLE(VPointer);
+ public:
bool valid() const { return _adr != nullptr; }
bool has_iv() const { return _scale != 0; }
@@ -126,7 +266,7 @@ class VPointer : public ArenaObj {
bool overlap_possible_with_any_in(Node_List* p) {
for (uint k = 0; k < p->size(); k++) {
MemNode* mem = p->at(k)->as_Mem();
- VPointer p_mem(mem, phase(), lpt(), nullptr, false);
+ VPointer p_mem(mem, vloop());
// Only if we know that we have Less or Greater can we
// be sure that there can never be an overlap between
// the two memory regions.
diff --git a/src/hotspot/share/opto/vectornode.cpp b/src/hotspot/share/opto/vectornode.cpp
index b07aa387712..f5fdae79e32 100644
--- a/src/hotspot/share/opto/vectornode.cpp
+++ b/src/hotspot/share/opto/vectornode.cpp
@@ -389,7 +389,7 @@ int VectorNode::scalar_opcode(int sopc, BasicType bt) {
// Limits on vector size (number of elements) for auto-vectorization.
bool VectorNode::vector_size_supported_superword(const BasicType bt, int size) {
- return Matcher::superword_max_vector_size(bt) >= size &&
+ return Matcher::max_vector_size_auto_vectorization(bt) >= size &&
Matcher::min_vector_size(bt) <= size;
}
@@ -409,7 +409,7 @@ bool VectorNode::implemented(int opc, uint vlen, BasicType bt) {
if (VectorNode::is_vector_integral_negate(vopc)) {
return is_vector_integral_negate_supported(vopc, vlen, bt, false);
}
- return vopc > 0 && Matcher::match_rule_supported_superword(vopc, vlen, bt);
+ return vopc > 0 && Matcher::match_rule_supported_auto_vectorization(vopc, vlen, bt);
}
return false;
}
@@ -1434,7 +1434,7 @@ bool VectorCastNode::implemented(int opc, uint vlen, BasicType src_type, BasicTy
(vlen > 1) && is_power_of_2(vlen) &&
VectorNode::vector_size_supported_superword(dst_type, vlen)) {
int vopc = VectorCastNode::opcode(opc, src_type);
- return vopc > 0 && Matcher::match_rule_supported_superword(vopc, vlen, dst_type);
+ return vopc > 0 && Matcher::match_rule_supported_auto_vectorization(vopc, vlen, dst_type);
}
return false;
}
@@ -1528,7 +1528,7 @@ bool ReductionNode::implemented(int opc, uint vlen, BasicType bt) {
(vlen > 1) && is_power_of_2(vlen) &&
VectorNode::vector_size_supported_superword(bt, vlen)) {
int vopc = ReductionNode::opcode(opc, bt);
- return vopc != opc && Matcher::match_rule_supported_superword(vopc, vlen, bt);
+ return vopc != opc && Matcher::match_rule_supported_auto_vectorization(vopc, vlen, bt);
}
return false;
}
diff --git a/src/hotspot/share/prims/jvm.cpp b/src/hotspot/share/prims/jvm.cpp
index 7487a1fc00d..4b5bf091ad0 100644
--- a/src/hotspot/share/prims/jvm.cpp
+++ b/src/hotspot/share/prims/jvm.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -3991,7 +3991,7 @@ JVM_ENTRY(void, JVM_VirtualThreadUnmount(JNIEnv* env, jobject vthread, jboolean
JVM_END
// Always update the temporary VTMS transition bit.
-JVM_ENTRY(void, JVM_VirtualThreadHideFrames(JNIEnv* env, jobject vthread, jboolean hide))
+JVM_ENTRY(void, JVM_VirtualThreadHideFrames(JNIEnv* env, jclass clazz, jboolean hide))
#if INCLUDE_JVMTI
if (!DoJVMTIVirtualThreadTransitions) {
assert(!JvmtiExport::can_support_virtual_threads(), "sanity check");
@@ -4005,7 +4005,7 @@ JVM_END
// Notification from VirtualThread about disabling JVMTI Suspend in a sync critical section.
// Needed to avoid deadlocks with JVMTI suspend mechanism.
-JVM_ENTRY(void, JVM_VirtualThreadDisableSuspend(JNIEnv* env, jobject vthread, jboolean enter))
+JVM_ENTRY(void, JVM_VirtualThreadDisableSuspend(JNIEnv* env, jclass clazz, jboolean enter))
#if INCLUDE_JVMTI
if (!DoJVMTIVirtualThreadTransitions) {
assert(!JvmtiExport::can_support_virtual_threads(), "sanity check");
diff --git a/src/hotspot/share/prims/jvmtiExport.cpp b/src/hotspot/share/prims/jvmtiExport.cpp
index eee5b2a70fa..0a436b2778e 100644
--- a/src/hotspot/share/prims/jvmtiExport.cpp
+++ b/src/hotspot/share/prims/jvmtiExport.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -2437,7 +2437,7 @@ void JvmtiExport::post_native_method_bind(Method* method, address* function_ptr)
}
// Returns a record containing inlining information for the given nmethod
-jvmtiCompiledMethodLoadInlineRecord* create_inline_record(nmethod* nm) {
+static jvmtiCompiledMethodLoadInlineRecord* create_inline_record(nmethod* nm) {
jint numstackframes = 0;
jvmtiCompiledMethodLoadInlineRecord* record = (jvmtiCompiledMethodLoadInlineRecord*)NEW_RESOURCE_OBJ(jvmtiCompiledMethodLoadInlineRecord);
record->header.kind = JVMTI_CMLR_INLINE_INFO;
diff --git a/src/hotspot/share/prims/jvmtiExport.hpp b/src/hotspot/share/prims/jvmtiExport.hpp
index 263104f74f8..36df28f2716 100644
--- a/src/hotspot/share/prims/jvmtiExport.hpp
+++ b/src/hotspot/share/prims/jvmtiExport.hpp
@@ -149,7 +149,15 @@ class JvmtiExport : public AllStatic {
JVMTI_ONLY(_can_access_local_variables = (on != 0);)
}
inline static void set_can_hotswap_or_post_breakpoint(bool on) {
- JVMTI_ONLY(_can_hotswap_or_post_breakpoint = (on != 0);)
+#if INCLUDE_JVMTI
+ // Check that _can_hotswap_or_post_breakpoint is not reset once it
+ // was set to true. When _can_hotswap_or_post_breakpoint is set to true
+ // _all_dependencies_are_recorded is also set to true and never
+ // reset so we have to ensure that evol dependencies are always
+ // recorded from that point on.
+ assert(!_can_hotswap_or_post_breakpoint || on, "sanity check");
+ _can_hotswap_or_post_breakpoint = (on != 0);
+#endif
}
inline static void set_can_walk_any_space(bool on) {
JVMTI_ONLY(_can_walk_any_space = (on != 0);)
diff --git a/src/hotspot/share/prims/jvmtiRedefineClasses.cpp b/src/hotspot/share/prims/jvmtiRedefineClasses.cpp
index c03c4b48538..bcfb361c89f 100644
--- a/src/hotspot/share/prims/jvmtiRedefineClasses.cpp
+++ b/src/hotspot/share/prims/jvmtiRedefineClasses.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -48,6 +48,7 @@
#include "oops/fieldStreams.inline.hpp"
#include "oops/klass.inline.hpp"
#include "oops/klassVtable.hpp"
+#include "oops/method.hpp"
#include "oops/oop.inline.hpp"
#include "oops/recordComponent.hpp"
#include "prims/jvmtiImpl.hpp"
@@ -64,6 +65,7 @@
#include "utilities/bitMap.inline.hpp"
#include "utilities/checkedCast.hpp"
#include "utilities/events.hpp"
+#include "utilities/macros.hpp"
Array* VM_RedefineClasses::_old_methods = nullptr;
Array* VM_RedefineClasses::_new_methods = nullptr;
@@ -1171,6 +1173,7 @@ jvmtiError VM_RedefineClasses::compare_and_normalize_class_versions(
}
}
}
+ JFR_ONLY(k_new_method->copy_trace_flags(*k_old_method->trace_flags_addr());)
log_trace(redefine, class, normalize)
("Method matched: new: %s [%d] == old: %s [%d]",
k_new_method->name_and_sig_as_C_string(), ni, k_old_method->name_and_sig_as_C_string(), oi);
@@ -1823,6 +1826,12 @@ jvmtiError VM_RedefineClasses::merge_cp_and_rewrite(
return JVMTI_ERROR_INTERNAL;
}
+ // ensure merged constant pool size does not overflow u2
+ if (merge_cp_length > 0xFFFF) {
+ log_warning(redefine, class, constantpool)("Merged constant pool overflow: %d entries", merge_cp_length);
+ return JVMTI_ERROR_INTERNAL;
+ }
+
// Set dynamic constants attribute from the original CP.
if (old_cp->has_dynamic_constant()) {
scratch_cp->set_has_dynamic_constant();
@@ -4060,21 +4069,22 @@ void VM_RedefineClasses::transfer_old_native_function_registrations(InstanceKlas
// Deoptimize all compiled code that depends on the classes redefined.
//
// If the can_redefine_classes capability is obtained in the onload
-// phase then the compiler has recorded all dependencies from startup.
-// In that case we need only deoptimize and throw away all compiled code
-// that depends on the class.
+// phase or 'AlwaysRecordEvolDependencies' is true, then the compiler has
+// recorded all dependencies from startup. In that case we need only
+// deoptimize and throw away all compiled code that depends on the class.
//
-// If can_redefine_classes is obtained sometime after the onload
-// phase then the dependency information may be incomplete. In that case
-// the first call to RedefineClasses causes all compiled code to be
-// thrown away. As can_redefine_classes has been obtained then
-// all future compilations will record dependencies so second and
-// subsequent calls to RedefineClasses need only throw away code
-// that depends on the class.
+// If can_redefine_classes is obtained sometime after the onload phase
+// (and 'AlwaysRecordEvolDependencies' is false) then the dependency
+// information may be incomplete. In that case the first call to
+// RedefineClasses causes all compiled code to be thrown away. As
+// can_redefine_classes has been obtained then all future compilations will
+// record dependencies so second and subsequent calls to RedefineClasses
+// need only throw away code that depends on the class.
//
void VM_RedefineClasses::flush_dependent_code() {
assert(SafepointSynchronize::is_at_safepoint(), "sanity check");
+ assert(JvmtiExport::all_dependencies_are_recorded() || !AlwaysRecordEvolDependencies, "sanity check");
DeoptimizationScope deopt_scope;
diff --git a/src/hotspot/share/prims/jvmtiTagMap.cpp b/src/hotspot/share/prims/jvmtiTagMap.cpp
index 77bbbbfa77c..bc91c105073 100644
--- a/src/hotspot/share/prims/jvmtiTagMap.cpp
+++ b/src/hotspot/share/prims/jvmtiTagMap.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2003, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -397,6 +397,9 @@ class ClassFieldMap: public CHeapObj {
// constructor
ClassFieldMap();
+ // calculates number of fields in all interfaces
+ static int interfaces_field_count(InstanceKlass* ik);
+
// add a field
void add(int index, char type, int offset);
@@ -424,6 +427,16 @@ ClassFieldMap::~ClassFieldMap() {
delete _fields;
}
+int ClassFieldMap::interfaces_field_count(InstanceKlass* ik) {
+ const Array* interfaces = ik->transitive_interfaces();
+ int count = 0;
+ for (int i = 0; i < interfaces->length(); i++) {
+ FilteredJavaFieldStream fld(interfaces->at(i));
+ count += fld.field_count();
+ }
+ return count;
+}
+
void ClassFieldMap::add(int index, char type, int offset) {
ClassFieldDescriptor* field = new ClassFieldDescriptor(index, type, offset);
_fields->append(field);
@@ -431,48 +444,59 @@ void ClassFieldMap::add(int index, char type, int offset) {
// Returns a heap allocated ClassFieldMap to describe the static fields
// of the given class.
-//
ClassFieldMap* ClassFieldMap::create_map_of_static_fields(Klass* k) {
InstanceKlass* ik = InstanceKlass::cast(k);
// create the field map
ClassFieldMap* field_map = new ClassFieldMap();
- FilteredFieldStream f(ik, false, false);
- int max_field_index = f.field_count()-1;
+ // Static fields of interfaces and superclasses are reported as references from the interfaces/superclasses.
+ // Need to calculate start index of this class fields: number of fields in all interfaces and superclasses.
+ int index = interfaces_field_count(ik);
+ for (InstanceKlass* super_klass = ik->java_super(); super_klass != nullptr; super_klass = super_klass->java_super()) {
+ FilteredJavaFieldStream super_fld(super_klass);
+ index += super_fld.field_count();
+ }
- int index = 0;
- for (FilteredFieldStream fld(ik, true, true); !fld.eos(); fld.next(), index++) {
+ for (FilteredJavaFieldStream fld(ik); !fld.done(); fld.next(), index++) {
// ignore instance fields
if (!fld.access_flags().is_static()) {
continue;
}
- field_map->add(max_field_index - index, fld.signature()->char_at(0), fld.offset());
+ field_map->add(index, fld.signature()->char_at(0), fld.offset());
}
+
return field_map;
}
// Returns a heap allocated ClassFieldMap to describe the instance fields
// of the given class. All instance fields are included (this means public
-// and private fields declared in superclasses and superinterfaces too).
-//
+// and private fields declared in superclasses too).
ClassFieldMap* ClassFieldMap::create_map_of_instance_fields(oop obj) {
InstanceKlass* ik = InstanceKlass::cast(obj->klass());
// create the field map
ClassFieldMap* field_map = new ClassFieldMap();
- FilteredFieldStream f(ik, false, false);
-
- int max_field_index = f.field_count()-1;
+ // fields of the superclasses are reported first, so need to know total field number to calculate field indices
+ int total_field_number = interfaces_field_count(ik);
+ for (InstanceKlass* klass = ik; klass != nullptr; klass = klass->java_super()) {
+ FilteredJavaFieldStream fld(klass);
+ total_field_number += fld.field_count();
+ }
- int index = 0;
- for (FilteredFieldStream fld(ik, false, false); !fld.eos(); fld.next(), index++) {
- // ignore static fields
- if (fld.access_flags().is_static()) {
- continue;
+ for (InstanceKlass* klass = ik; klass != nullptr; klass = klass->java_super()) {
+ FilteredJavaFieldStream fld(klass);
+ int start_index = total_field_number - fld.field_count();
+ for (int index = 0; !fld.done(); fld.next(), index++) {
+ // ignore static fields
+ if (fld.access_flags().is_static()) {
+ continue;
+ }
+ field_map->add(start_index + index, fld.signature()->char_at(0), fld.offset());
}
- field_map->add(max_field_index - index, fld.signature()->char_at(0), fld.offset());
+ // update total_field_number for superclass (decrease by the field count in the current class)
+ total_field_number = start_index;
}
return field_map;
diff --git a/src/hotspot/share/prims/methodHandles.cpp b/src/hotspot/share/prims/methodHandles.cpp
index e1f1977b67e..3ffe4853b8c 100644
--- a/src/hotspot/share/prims/methodHandles.cpp
+++ b/src/hotspot/share/prims/methodHandles.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2008, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2008, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -55,7 +55,6 @@
#include "runtime/jniHandles.inline.hpp"
#include "runtime/timerTrace.hpp"
#include "runtime/reflection.hpp"
-#include "runtime/reflectionUtils.hpp"
#include "runtime/safepointVerifiers.hpp"
#include "runtime/signature.hpp"
#include "runtime/stubRoutines.hpp"
diff --git a/src/hotspot/share/prims/resolvedMethodTable.cpp b/src/hotspot/share/prims/resolvedMethodTable.cpp
index 4ff09c6581b..0dd96357506 100644
--- a/src/hotspot/share/prims/resolvedMethodTable.cpp
+++ b/src/hotspot/share/prims/resolvedMethodTable.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2017, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2017, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -53,7 +53,7 @@ static const size_t GROW_HINT = 32;
static const size_t ResolvedMethodTableSizeLog = 10;
-unsigned int method_hash(const Method* method) {
+static unsigned int method_hash(const Method* method) {
unsigned int hash = method->method_holder()->class_loader_data()->identity_hash();
hash = (hash * 31) ^ method->klass_name()->identity_hash();
hash = (hash * 31) ^ method->name()->identity_hash();
diff --git a/src/hotspot/share/prims/whitebox.cpp b/src/hotspot/share/prims/whitebox.cpp
index d70c775937f..dbb9566cd22 100644
--- a/src/hotspot/share/prims/whitebox.cpp
+++ b/src/hotspot/share/prims/whitebox.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2012, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -63,6 +63,7 @@
#include "oops/constantPool.inline.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
+#include "oops/methodData.inline.hpp"
#include "oops/objArrayKlass.hpp"
#include "oops/objArrayOop.inline.hpp"
#include "oops/oop.inline.hpp"
@@ -84,6 +85,7 @@
#include "runtime/javaCalls.hpp"
#include "runtime/javaThread.inline.hpp"
#include "runtime/jniHandles.inline.hpp"
+#include "runtime/lockStack.hpp"
#include "runtime/os.hpp"
#include "runtime/stackFrameStream.inline.hpp"
#include "runtime/synchronizer.hpp"
@@ -1226,7 +1228,6 @@ WB_ENTRY(void, WB_ClearMethodState(JNIEnv* env, jobject o, jobject method))
for (int i = 0; i < arg_count; i++) {
mdo->set_arg_modified(i, 0);
}
- MutexLocker mu(THREAD, mdo->extra_data_lock());
mdo->clean_method_data(/*always_clean*/true);
}
@@ -1847,6 +1848,14 @@ WB_ENTRY(jboolean, WB_IsMonitorInflated(JNIEnv* env, jobject wb, jobject obj))
return (jboolean) obj_oop->mark().has_monitor();
WB_END
+WB_ENTRY(jint, WB_getLockStackCapacity(JNIEnv* env))
+ return (jint) LockStack::CAPACITY;
+WB_END
+
+WB_ENTRY(jboolean, WB_supportsRecursiveLightweightLocking(JNIEnv* env))
+ return (jboolean) VM_Version::supports_recursive_lightweight_locking();
+WB_END
+
WB_ENTRY(jboolean, WB_DeflateIdleMonitors(JNIEnv* env, jobject wb))
log_info(monitorinflation)("WhiteBox initiated DeflateIdleMonitors");
return ObjectSynchronizer::request_deflate_idle_monitors_from_wb();
@@ -1890,7 +1899,7 @@ WB_END
WB_ENTRY(jint, WB_getFieldCPIndex(JNIEnv* env, jobject wb, jclass klass, jint index))
InstanceKlass* ik = InstanceKlass::cast(java_lang_Class::as_Klass(JNIHandles::resolve(klass)));
ConstantPool* cp = ik->constants();
- if (cp->cache() == NULL) {
+ if (cp->cache() == nullptr) {
return -1;
}
return cp->resolved_field_entry_at(index)->constant_pool_index();
@@ -1908,7 +1917,7 @@ WB_END
WB_ENTRY(jint, WB_getMethodCPIndex(JNIEnv* env, jobject wb, jclass klass, jint index))
InstanceKlass* ik = InstanceKlass::cast(java_lang_Class::as_Klass(JNIHandles::resolve(klass)));
ConstantPool* cp = ik->constants();
- if (cp->cache() == NULL) {
+ if (cp->cache() == nullptr) {
return -1;
}
return cp->resolved_method_entry_at(index)->constant_pool_index();
@@ -2829,6 +2838,8 @@ static JNINativeMethod methods[] = {
(void*)&WB_AddModuleExportsToAll },
{CC"deflateIdleMonitors", CC"()Z", (void*)&WB_DeflateIdleMonitors },
{CC"isMonitorInflated0", CC"(Ljava/lang/Object;)Z", (void*)&WB_IsMonitorInflated },
+ {CC"getLockStackCapacity", CC"()I", (void*)&WB_getLockStackCapacity },
+ {CC"supportsRecursiveLightweightLocking", CC"()Z", (void*)&WB_supportsRecursiveLightweightLocking },
{CC"forceSafepoint", CC"()V", (void*)&WB_ForceSafepoint },
{CC"forceClassLoaderStatsSafepoint", CC"()V", (void*)&WB_ForceClassLoaderStatsSafepoint },
{CC"getConstantPool0", CC"(Ljava/lang/Class;)J", (void*)&WB_GetConstantPool },
diff --git a/src/hotspot/share/runtime/abstract_vm_version.cpp b/src/hotspot/share/runtime/abstract_vm_version.cpp
index f1f8888653e..9fada531d0b 100644
--- a/src/hotspot/share/runtime/abstract_vm_version.cpp
+++ b/src/hotspot/share/runtime/abstract_vm_version.cpp
@@ -34,6 +34,7 @@ const char* Abstract_VM_Version::_s_internal_vm_info_string = Abstract_VM_Versio
uint64_t Abstract_VM_Version::_features = 0;
const char* Abstract_VM_Version::_features_string = "";
+uint64_t Abstract_VM_Version::_cpu_features = 0;
#ifndef SUPPORTS_NATIVE_CX8
bool Abstract_VM_Version::_supports_cx8 = false;
diff --git a/src/hotspot/share/runtime/abstract_vm_version.hpp b/src/hotspot/share/runtime/abstract_vm_version.hpp
index d8ffca8de81..6fca13ece85 100644
--- a/src/hotspot/share/runtime/abstract_vm_version.hpp
+++ b/src/hotspot/share/runtime/abstract_vm_version.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -54,10 +54,13 @@ class Abstract_VM_Version: AllStatic {
static const char* _s_vm_release;
static const char* _s_internal_vm_info_string;
- // CPU feature flags.
+ // CPU feature flags, can be affected by VM settings.
static uint64_t _features;
static const char* _features_string;
+ // Original CPU feature flags, not affected by VM settings.
+ static uint64_t _cpu_features;
+
// These are set by machine-dependent initializations
#ifndef SUPPORTS_NATIVE_CX8
static bool _supports_cx8;
@@ -184,6 +187,9 @@ class Abstract_VM_Version: AllStatic {
// Does platform support stack watermark barriers for concurrent stack processing?
constexpr static bool supports_stack_watermark_barrier() { return false; }
+ // Is recursive lightweight locking implemented for this platform?
+ constexpr static bool supports_recursive_lightweight_locking() { return false; }
+
// Does platform support float16 instructions?
static bool supports_float16() { return false; }
diff --git a/src/hotspot/share/runtime/arguments.cpp b/src/hotspot/share/runtime/arguments.cpp
index 522443398a0..ee687d5e553 100644
--- a/src/hotspot/share/runtime/arguments.cpp
+++ b/src/hotspot/share/runtime/arguments.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -494,10 +494,6 @@ void Arguments::init_version_specific_system_properties() {
static SpecialFlag const special_jvm_flags[] = {
// -------------- Deprecated Flags --------------
// --- Non-alias flags - sorted by obsolete_in then expired_in:
- { "MaxGCMinorPauseMillis", JDK_Version::jdk(8), JDK_Version::undefined(), JDK_Version::undefined() },
- { "MaxRAMFraction", JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() },
- { "MinRAMFraction", JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() },
- { "InitialRAMFraction", JDK_Version::jdk(10), JDK_Version::undefined(), JDK_Version::undefined() },
{ "AllowRedefinitionToAddDeleteMethods", JDK_Version::jdk(13), JDK_Version::undefined(), JDK_Version::undefined() },
{ "FlightRecorder", JDK_Version::jdk(13), JDK_Version::undefined(), JDK_Version::undefined() },
{ "DumpSharedSpaces", JDK_Version::jdk(18), JDK_Version::jdk(19), JDK_Version::undefined() },
@@ -507,9 +503,7 @@ static SpecialFlag const special_jvm_flags[] = {
{ "RegisterFinalizersAtInit", JDK_Version::jdk(22), JDK_Version::jdk(23), JDK_Version::jdk(24) },
// --- Deprecated alias flags (see also aliased_jvm_flags) - sorted by obsolete_in then expired_in:
- { "DefaultMaxRAMFraction", JDK_Version::jdk(8), JDK_Version::undefined(), JDK_Version::undefined() },
{ "CreateMinidumpOnCrash", JDK_Version::jdk(9), JDK_Version::undefined(), JDK_Version::undefined() },
- { "TLABStats", JDK_Version::jdk(12), JDK_Version::undefined(), JDK_Version::undefined() },
// -------------- Obsolete Flags - sorted by expired_in --------------
@@ -526,6 +520,12 @@ static SpecialFlag const special_jvm_flags[] = {
{ "MetaspaceReclaimPolicy", JDK_Version::undefined(), JDK_Version::jdk(21), JDK_Version::undefined() },
{ "AdaptiveSizePolicyCollectionCostMargin", JDK_Version::undefined(), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "MaxGCMinorPauseMillis", JDK_Version::jdk(8), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "MaxRAMFraction", JDK_Version::jdk(10), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "MinRAMFraction", JDK_Version::jdk(10), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "InitialRAMFraction", JDK_Version::jdk(10), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "DefaultMaxRAMFraction", JDK_Version::jdk(8), JDK_Version::jdk(23), JDK_Version::jdk(24) },
+ { "TLABStats", JDK_Version::jdk(12), JDK_Version::jdk(23), JDK_Version::jdk(24) },
#ifdef ASSERT
{ "DummyObsoleteTestFlag", JDK_Version::undefined(), JDK_Version::jdk(18), JDK_Version::undefined() },
#endif
@@ -551,7 +551,6 @@ typedef struct {
} AliasedFlag;
static AliasedFlag const aliased_jvm_flags[] = {
- { "DefaultMaxRAMFraction", "MaxRAMFraction" },
{ "CreateMinidumpOnCrash", "CreateCoredumpOnCrash" },
{ nullptr, nullptr}
};
@@ -1120,18 +1119,7 @@ bool Arguments::process_argument(const char* arg,
if (found_flag != nullptr) {
char locked_message_buf[BUFLEN];
JVMFlag::MsgType msg_type = found_flag->get_locked_message(locked_message_buf, BUFLEN);
- if (strlen(locked_message_buf) == 0) {
- if (found_flag->is_bool() && !has_plus_minus) {
- jio_fprintf(defaultStream::error_stream(),
- "Missing +/- setting for VM option '%s'\n", argname);
- } else if (!found_flag->is_bool() && has_plus_minus) {
- jio_fprintf(defaultStream::error_stream(),
- "Unexpected +/- setting in VM option '%s'\n", argname);
- } else {
- jio_fprintf(defaultStream::error_stream(),
- "Improperly specified VM option '%s'\n", argname);
- }
- } else {
+ if (strlen(locked_message_buf) != 0) {
#ifdef PRODUCT
bool mismatched = ((msg_type == JVMFlag::NOTPRODUCT_FLAG_BUT_PRODUCT_BUILD) ||
(msg_type == JVMFlag::DEVELOPER_FLAG_BUT_PRODUCT_BUILD));
@@ -1141,6 +1129,16 @@ bool Arguments::process_argument(const char* arg,
#endif
jio_fprintf(defaultStream::error_stream(), "%s", locked_message_buf);
}
+ if (found_flag->is_bool() && !has_plus_minus) {
+ jio_fprintf(defaultStream::error_stream(),
+ "Missing +/- setting for VM option '%s'\n", argname);
+ } else if (!found_flag->is_bool() && has_plus_minus) {
+ jio_fprintf(defaultStream::error_stream(),
+ "Unexpected +/- setting in VM option '%s'\n", argname);
+ } else {
+ jio_fprintf(defaultStream::error_stream(),
+ "Improperly specified VM option '%s'\n", argname);
+ }
} else {
if (ignore_unrecognized) {
return true;
@@ -1366,7 +1364,7 @@ void Arguments::no_shared_spaces(const char* message) {
}
}
-void set_object_alignment() {
+static void set_object_alignment() {
// Object alignment.
assert(is_power_of_2(ObjectAlignmentInBytes), "ObjectAlignmentInBytes must be power of 2");
MinObjAlignmentInBytes = ObjectAlignmentInBytes;
@@ -1479,11 +1477,8 @@ void Arguments::set_heap_size() {
// available os physical memory, not our MaxRAM limit,
// unless MaxRAM is also specified.
bool override_coop_limit = (!FLAG_IS_DEFAULT(MaxRAMPercentage) ||
- !FLAG_IS_DEFAULT(MaxRAMFraction) ||
!FLAG_IS_DEFAULT(MinRAMPercentage) ||
- !FLAG_IS_DEFAULT(MinRAMFraction) ||
!FLAG_IS_DEFAULT(InitialRAMPercentage) ||
- !FLAG_IS_DEFAULT(InitialRAMFraction) ||
!FLAG_IS_DEFAULT(MaxRAM));
if (override_coop_limit) {
if (FLAG_IS_DEFAULT(MaxRAM)) {
@@ -1497,20 +1492,6 @@ void Arguments::set_heap_size() {
: (julong)MaxRAM;
}
-
- // Convert deprecated flags
- if (FLAG_IS_DEFAULT(MaxRAMPercentage) &&
- !FLAG_IS_DEFAULT(MaxRAMFraction))
- MaxRAMPercentage = 100.0 / (double)MaxRAMFraction;
-
- if (FLAG_IS_DEFAULT(MinRAMPercentage) &&
- !FLAG_IS_DEFAULT(MinRAMFraction))
- MinRAMPercentage = 100.0 / (double)MinRAMFraction;
-
- if (FLAG_IS_DEFAULT(InitialRAMPercentage) &&
- !FLAG_IS_DEFAULT(InitialRAMFraction))
- InitialRAMPercentage = 100.0 / (double)InitialRAMFraction;
-
// If the maximum heap size has not been set with -Xmx,
// then set it as fraction of the size of physical memory,
// respecting the maximum and minimum sizes of the heap.
@@ -2028,11 +2009,12 @@ jint Arguments::parse_vm_init_args(const JavaVMInitArgs *vm_options_args,
return JNI_OK;
}
+#if !INCLUDE_JVMTI
// Checks if name in command-line argument -agent{lib,path}:name[=options]
// represents a valid JDWP agent. is_path==true denotes that we
// are dealing with -agentpath (case where name is a path), otherwise with
// -agentlib
-bool valid_jdwp_agent(char *name, bool is_path) {
+static bool valid_jdwp_agent(char *name, bool is_path) {
char *_name;
const char *_jdwp = "jdwp";
size_t _len_jdwp, _len_prefix;
@@ -2072,6 +2054,7 @@ bool valid_jdwp_agent(char *name, bool is_path) {
return false;
}
+#endif
int Arguments::process_patch_mod_option(const char* patch_mod_tail, bool* patch_mod_javabase) {
// --patch-module==()*
diff --git a/src/hotspot/share/runtime/continuationFreezeThaw.cpp b/src/hotspot/share/runtime/continuationFreezeThaw.cpp
index 26c915852e5..c9ddfa900dd 100644
--- a/src/hotspot/share/runtime/continuationFreezeThaw.cpp
+++ b/src/hotspot/share/runtime/continuationFreezeThaw.cpp
@@ -2181,7 +2181,7 @@ void ThawBase::clear_bitmap_bits(address start, address end) {
log_develop_trace(continuations)("clearing bitmap for " INTPTR_FORMAT " - " INTPTR_FORMAT, p2i(start), p2i(effective_end));
stackChunkOop chunk = _cont.tail();
chunk->bitmap().clear_range(chunk->bit_index_for(start), chunk->bit_index_for(effective_end));
- assert(chunk->bitmap().count_one_bits(chunk->bit_index_for(effective_end), chunk->bit_index_for(end)) == 0, "bits should not be set");
+ assert(effective_end == end || !chunk->bitmap().at(chunk->bit_index_for(effective_end)), "bit should not be set");
}
NOINLINE void ThawBase::recurse_thaw_interpreted_frame(const frame& hf, frame& caller, int num_frames) {
diff --git a/src/hotspot/share/runtime/deoptimization.cpp b/src/hotspot/share/runtime/deoptimization.cpp
index a5f2e09afaa..e8f443ed1e0 100644
--- a/src/hotspot/share/runtime/deoptimization.cpp
+++ b/src/hotspot/share/runtime/deoptimization.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -391,7 +391,8 @@ static void restore_eliminated_locks(JavaThread* thread, GrowableArraylength(); i++) {
+ // Start locking from outermost/oldest frame
+ for (int i = (chunk->length() - 1); i >= 0; i--) {
compiledVFrame* cvf = chunk->at(i);
assert (cvf->scope() != nullptr,"expect only compiled java frames");
GrowableArray* monitors = cvf->monitors();
@@ -1447,7 +1448,7 @@ class ReassignedField {
}
};
-int compare(ReassignedField* left, ReassignedField* right) {
+static int compare(ReassignedField* left, ReassignedField* right) {
return left->_offset - right->_offset;
}
@@ -1603,6 +1604,10 @@ void Deoptimization::reassign_fields(frame* fr, RegisterMap* reg_map, GrowableAr
reassign_object_array_elements(fr, reg_map, sv, (objArrayOop) obj());
}
}
+ // These objects may escape when we return to Interpreter after deoptimization.
+ // We need barrier so that stores that initialize these objects can't be reordered
+ // with subsequent stores that make these objects accessible by other threads.
+ OrderAccess::storestore();
}
@@ -1641,13 +1646,13 @@ bool Deoptimization::relock_objects(JavaThread* thread, GrowableArrayowner()->is_locked(), "object must be locked now");
- ObjectMonitor* mon = ObjectSynchronizer::inflate(deoptee_thread, obj(), ObjectSynchronizer::inflate_cause_vm_internal);
+ ObjectMonitor* mon = ObjectSynchronizer::inflate_for(deoptee_thread, obj(), ObjectSynchronizer::inflate_cause_vm_internal);
assert(mon->owner() == deoptee_thread, "must be");
} else {
BasicLock* lock = mon_info->lock();
- ObjectSynchronizer::enter(obj, lock, deoptee_thread);
+ ObjectSynchronizer::enter_for(obj, lock, deoptee_thread);
assert(mon_info->owner()->is_locked(), "object must be locked now");
}
}
@@ -1720,7 +1725,8 @@ void Deoptimization::pop_frames_failed_reallocs(JavaThread* thread, vframeArray*
for (int i = 0; i < array->frames(); i++) {
MonitorChunk* monitors = array->element(i)->monitors();
if (monitors != nullptr) {
- for (int j = 0; j < monitors->number_of_monitors(); j++) {
+ // Unlock in reverse order starting from most nested monitor.
+ for (int j = (monitors->number_of_monitors() - 1); j >= 0; j--) {
BasicObjectLock* src = monitors->at(j);
if (src->obj() != nullptr) {
ObjectSynchronizer::exit(src->obj(), src->lock(), thread);
@@ -1797,6 +1803,9 @@ address Deoptimization::deoptimize_for_missing_exception_handler(CompiledMethod*
ScopeDesc* imm_scope = cvf->scope();
MethodData* imm_mdo = get_method_data(thread, methodHandle(thread, imm_scope->method()), true);
if (imm_mdo != nullptr) {
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(imm_mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
ProfileData* pdata = imm_mdo->allocate_bci_to_data(imm_scope->bci(), nullptr);
if (pdata != nullptr && pdata->is_BitData()) {
BitData* bit_data = (BitData*) pdata;
@@ -2102,7 +2111,14 @@ JRT_ENTRY(void, Deoptimization::uncommon_trap_inner(JavaThread* current, jint tr
// Print a bunch of diagnostics, if requested.
if (TraceDeoptimization || LogCompilation || is_receiver_constraint_failure) {
ResourceMark rm;
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ // We must do this already now, since we cannot acquire this lock while
+ // holding the tty lock (lock ordering by rank).
+ MutexLocker ml(trap_mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
ttyLocker ttyl;
+
char buf[100];
if (xtty != nullptr) {
xtty->begin_head("uncommon_trap thread='" UINTX_FORMAT "' %s",
@@ -2137,6 +2153,9 @@ JRT_ENTRY(void, Deoptimization::uncommon_trap_inner(JavaThread* current, jint tr
int dcnt = trap_mdo->trap_count(reason);
if (dcnt != 0)
xtty->print(" count='%d'", dcnt);
+
+ // We need to lock to read the ProfileData. But to keep the locks ordered, we need to
+ // lock extra_data_lock before the tty lock.
ProfileData* pdata = trap_mdo->bci_to_data(trap_bci);
int dos = (pdata == nullptr)? 0: pdata->trap_state();
if (dos != 0) {
@@ -2312,12 +2331,18 @@ JRT_ENTRY(void, Deoptimization::uncommon_trap_inner(JavaThread* current, jint tr
// to use the MDO to detect hot deoptimization points and control
// aggressive optimization.
bool inc_recompile_count = false;
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ ConditionalMutexLocker ml((trap_mdo != nullptr) ? trap_mdo->extra_data_lock() : nullptr,
+ (trap_mdo != nullptr),
+ Mutex::_no_safepoint_check_flag);
ProfileData* pdata = nullptr;
if (ProfileTraps && CompilerConfig::is_c2_or_jvmci_compiler_enabled() && update_trap_state && trap_mdo != nullptr) {
assert(trap_mdo == get_method_data(current, profiled_method, false), "sanity");
uint this_trap_count = 0;
bool maybe_prior_trap = false;
bool maybe_prior_recompile = false;
+
pdata = query_update_method_data(trap_mdo, trap_bci, reason, true,
#if INCLUDE_JVMCI
nm->is_compiled_by_jvmci() && nm->is_osr_method(),
@@ -2473,6 +2498,8 @@ Deoptimization::query_update_method_data(MethodData* trap_mdo,
uint& ret_this_trap_count,
bool& ret_maybe_prior_trap,
bool& ret_maybe_prior_recompile) {
+ trap_mdo->check_extra_data_locked();
+
bool maybe_prior_trap = false;
bool maybe_prior_recompile = false;
uint this_trap_count = 0;
@@ -2555,6 +2582,10 @@ Deoptimization::update_method_data_from_interpreter(MethodData* trap_mdo, int tr
assert(!reason_is_speculate(reason), "reason speculate only used by compiler");
// JVMCI uses the total counts to determine if deoptimizations are happening too frequently -> do not adjust total counts
bool update_total_counts = true JVMCI_ONLY( && !UseJVMCICompiler);
+
+ // Lock to read ProfileData, and ensure lock is not broken by a safepoint
+ MutexLocker ml(trap_mdo->extra_data_lock(), Mutex::_no_safepoint_check_flag);
+
query_update_method_data(trap_mdo, trap_bci,
(DeoptReason)reason,
update_total_counts,
diff --git a/src/hotspot/share/runtime/flags/jvmFlag.cpp b/src/hotspot/share/runtime/flags/jvmFlag.cpp
index eb41f2ccb4b..005cd84f214 100644
--- a/src/hotspot/share/runtime/flags/jvmFlag.cpp
+++ b/src/hotspot/share/runtime/flags/jvmFlag.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -133,7 +133,7 @@ JVMFlag::MsgType JVMFlag::get_locked_message(char* buf, int buflen) const {
// Fills current line up to requested position.
// Should the current position already be past the requested position,
// one separator blank is enforced.
-void fill_to_pos(outputStream* st, unsigned int req_pos) {
+static void fill_to_pos(outputStream* st, unsigned int req_pos) {
if ((unsigned int)st->position() < req_pos) {
st->fill_to(req_pos); // need to fill with blanks to reach req_pos
} else {
@@ -275,7 +275,6 @@ void JVMFlag::print_on(outputStream* st, bool withComments, bool printRanges) co
//
// Sample output:
// intx MinPassesBeforeFlush [ 0 ... 9223372036854775807 ] {diagnostic} {default}
- // uintx MinRAMFraction [ 1 ... 18446744073709551615 ] {product} {default}
// double MinRAMPercentage [ 0.000 ... 100.000 ] {product} {default}
// uintx MinSurvivorRatio [ 3 ... 18446744073709551615 ] {product} {default}
// size_t MinTLABSize [ 1 ... 9223372036854775807 ] {product} {default}
diff --git a/src/hotspot/share/runtime/globals.hpp b/src/hotspot/share/runtime/globals.hpp
index b5e16c7ec25..323cfb421f9 100644
--- a/src/hotspot/share/runtime/globals.hpp
+++ b/src/hotspot/share/runtime/globals.hpp
@@ -932,9 +932,6 @@ const int ObjectAlignmentInBytes = 8;
notproduct(bool, TraceInvocationCounterOverflow, false, \
"Trace method invocation counter overflow") \
\
- develop(bool, TraceInlineCacheClearing, false, \
- "Trace clearing of inline caches in nmethods") \
- \
develop(bool, VerifyDependencies, trueInDebug, \
"Exercise and verify the compilation dependency mechanism") \
\
@@ -953,12 +950,6 @@ const int ObjectAlignmentInBytes = 8;
develop(bool, TraceOopMapRewrites, false, \
"Trace rewriting of methods during oop map generation") \
\
- develop(bool, TraceICBuffer, false, \
- "Trace usage of IC buffer") \
- \
- develop(bool, TraceCompiledIC, false, \
- "Trace changes of compiled IC") \
- \
develop(bool, FLSVerifyDictionary, false, \
"Do lots of (expensive) FLS dictionary verification") \
\
@@ -2062,6 +2053,11 @@ const int ObjectAlignmentInBytes = 8;
\
product(bool, ProfileExceptionHandlers, true, \
"Profile exception handlers") \
+ \
+ product(bool, AlwaysRecordEvolDependencies, true, EXPERIMENTAL, \
+ "Unconditionally record nmethod dependencies on class " \
+ "rewriting/transformation independently of the JVMTI " \
+ "can_{retransform/redefine}_classes capabilities.") \
// end of RUNTIME_FLAGS
diff --git a/src/hotspot/share/runtime/init.cpp b/src/hotspot/share/runtime/init.cpp
index 3343521af62..d37ae99b418 100644
--- a/src/hotspot/share/runtime/init.cpp
+++ b/src/hotspot/share/runtime/init.cpp
@@ -25,7 +25,6 @@
#include "precompiled.hpp"
#include "classfile/stringTable.hpp"
#include "classfile/symbolTable.hpp"
-#include "code/icBuffer.hpp"
#include "compiler/compiler_globals.hpp"
#include "gc/shared/collectedHeap.hpp"
#include "gc/shared/gcHeapSummary.hpp"
@@ -83,7 +82,6 @@ void jni_handles_init();
void vmStructs_init() NOT_DEBUG_RETURN;
void vtableStubs_init();
-void InlineCacheBuffer_init();
bool compilerOracle_init();
bool compileBroker_init();
void dependencyContext_init();
@@ -115,6 +113,12 @@ void vm_init_globals() {
jint init_globals() {
management_init();
JvmtiExport::initialize_oop_storage();
+#if INCLUDE_JVMTI
+ if (AlwaysRecordEvolDependencies) {
+ JvmtiExport::set_can_hotswap_or_post_breakpoint(true);
+ JvmtiExport::set_all_dependencies_are_recorded(true);
+ }
+#endif
bytecodes_init();
classLoader_init1();
compilationPolicy_init();
@@ -157,7 +161,6 @@ jint init_globals2() {
#endif // INCLUDE_VM_STRUCTS
vtableStubs_init();
- InlineCacheBuffer_init();
if (!compilerOracle_init()) {
return JNI_EINVAL;
}
diff --git a/src/hotspot/share/runtime/java.cpp b/src/hotspot/share/runtime/java.cpp
index 78ea902adf4..da1b4e00dff 100644
--- a/src/hotspot/share/runtime/java.cpp
+++ b/src/hotspot/share/runtime/java.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -109,7 +109,7 @@
GrowableArray* collected_profiled_methods;
-int compare_methods(Method** a, Method** b) {
+static int compare_methods(Method** a, Method** b) {
// compiled_invocation_count() returns int64_t, forcing the entire expression
// to be evaluated as int64_t. Overflow is not an issue.
int64_t diff = (((*b)->invocation_count() + (*b)->compiled_invocation_count())
@@ -117,7 +117,7 @@ int compare_methods(Method** a, Method** b) {
return (diff < 0) ? -1 : (diff > 0) ? 1 : 0;
}
-void collect_profiled_methods(Method* m) {
+static void collect_profiled_methods(Method* m) {
Thread* thread = Thread::current();
methodHandle mh(thread, m);
if ((m->method_data() != nullptr) &&
@@ -126,7 +126,7 @@ void collect_profiled_methods(Method* m) {
}
}
-void print_method_profiling_data() {
+static void print_method_profiling_data() {
if ((ProfileInterpreter COMPILER1_PRESENT(|| C1UpdateMethodData)) &&
(PrintMethodData || CompilerOracle::should_print_methods())) {
ResourceMark rm;
@@ -139,17 +139,23 @@ void print_method_profiling_data() {
if (count > 0) {
for (int index = 0; index < count; index++) {
Method* m = collected_profiled_methods->at(index);
- ttyLocker ttyl;
- tty->print_cr("------------------------------------------------------------------------");
- m->print_invocation_count();
- tty->print_cr(" mdo size: %d bytes", m->method_data()->size_in_bytes());
- tty->cr();
+
+ // Instead of taking tty lock, we collect all lines into a string stream
+ // and then print them all at once.
+ ResourceMark rm2;
+ stringStream ss;
+
+ ss.print_cr("------------------------------------------------------------------------");
+ m->print_invocation_count(&ss);
+ ss.print_cr(" mdo size: %d bytes", m->method_data()->size_in_bytes());
+ ss.cr();
// Dump data on parameters if any
if (m->method_data() != nullptr && m->method_data()->parameters_type_data() != nullptr) {
- tty->fill_to(2);
- m->method_data()->parameters_type_data()->print_data_on(tty);
+ ss.fill_to(2);
+ m->method_data()->parameters_type_data()->print_data_on(&ss);
}
- m->print_codes();
+ m->print_codes_on(&ss);
+ tty->print("%s", ss.as_string()); // print all at once
total_size += m->method_data()->size_in_bytes();
}
tty->print_cr("------------------------------------------------------------------------");
@@ -165,7 +171,7 @@ void print_method_profiling_data() {
GrowableArray* collected_invoked_methods;
-void collect_invoked_methods(Method* m) {
+static void collect_invoked_methods(Method* m) {
if (m->invocation_count() + m->compiled_invocation_count() >= 1) {
collected_invoked_methods->push(m);
}
@@ -175,7 +181,7 @@ void collect_invoked_methods(Method* m) {
// Invocation count accumulators should be unsigned long to shift the
// overflow border. Longer-running workloads tend to create invocation
// counts which already overflow 32-bit counters for individual methods.
-void print_method_invocation_histogram() {
+static void print_method_invocation_histogram() {
ResourceMark rm;
collected_invoked_methods = new GrowableArray(1024);
SystemDictionary::methods_do(collect_invoked_methods);
@@ -201,7 +207,7 @@ void print_method_invocation_histogram() {
Method* m = collected_invoked_methods->at(index);
uint64_t iic = (uint64_t)m->invocation_count();
uint64_t cic = (uint64_t)m->compiled_invocation_count();
- if ((iic + cic) >= (uint64_t)MethodHistogramCutoff) m->print_invocation_count();
+ if ((iic + cic) >= (uint64_t)MethodHistogramCutoff) m->print_invocation_count(tty);
int_total += iic;
comp_total += cic;
if (m->is_final()) final_total += iic + cic;
@@ -229,7 +235,7 @@ void print_method_invocation_histogram() {
SharedRuntime::print_call_statistics(comp_total);
}
-void print_bytecode_count() {
+static void print_bytecode_count() {
if (CountBytecodes || TraceBytecodes || StopInterpreterAt) {
tty->print_cr("[BytecodeCounter::counter_value = %d]", BytecodeCounter::counter_value());
}
@@ -237,8 +243,8 @@ void print_bytecode_count() {
#else
-void print_method_invocation_histogram() {}
-void print_bytecode_count() {}
+static void print_method_invocation_histogram() {}
+static void print_bytecode_count() {}
#endif // PRODUCT
@@ -592,7 +598,7 @@ void vm_direct_exit(int code, const char* message) {
vm_direct_exit(code);
}
-void vm_perform_shutdown_actions() {
+static void vm_perform_shutdown_actions() {
if (is_init_completed()) {
Thread* thread = Thread::current_or_null();
if (thread != nullptr && thread->is_Java_thread()) {
@@ -627,7 +633,7 @@ void vm_abort(bool dump_core) {
ShouldNotReachHere();
}
-void vm_notify_during_cds_dumping(const char* error, const char* message) {
+static void vm_notify_during_cds_dumping(const char* error, const char* message) {
if (error != nullptr) {
tty->print_cr("Error occurred during CDS dumping");
tty->print("%s", error);
@@ -647,7 +653,7 @@ void vm_exit_during_cds_dumping(const char* error, const char* message) {
vm_abort(false);
}
-void vm_notify_during_shutdown(const char* error, const char* message) {
+static void vm_notify_during_shutdown(const char* error, const char* message) {
if (error != nullptr) {
tty->print_cr("Error occurred during initialization of VM");
tty->print("%s", error);
diff --git a/src/hotspot/share/runtime/javaThread.cpp b/src/hotspot/share/runtime/javaThread.cpp
index a789988e379..0ec1236785b 100644
--- a/src/hotspot/share/runtime/javaThread.cpp
+++ b/src/hotspot/share/runtime/javaThread.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2021, Azul Systems, Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -194,7 +194,7 @@ oop JavaThread::scopedValueCache() const {
}
void JavaThread::set_scopedValueCache(oop p) {
- if (_scopedValueCache.ptr_raw() != nullptr) { // i.e. if the OopHandle has been allocated
+ if (!_scopedValueCache.is_empty()) { // i.e. if the OopHandle has been allocated
_scopedValueCache.replace(p);
} else {
assert(p == nullptr, "not yet initialized");
@@ -263,7 +263,7 @@ void JavaThread::allocate_threadObj(Handle thread_group, const char* thread_name
jlong* JavaThread::_jvmci_old_thread_counters;
-bool jvmci_counters_include(JavaThread* thread) {
+static bool jvmci_counters_include(JavaThread* thread) {
return !JVMCICountersExcludeCompiler || !thread->is_Compiler_thread();
}
@@ -282,7 +282,7 @@ void JavaThread::collect_counters(jlong* array, int length) {
}
// Attempt to enlarge the array for per thread counters.
-jlong* resize_counters_array(jlong* old_counters, int current_size, int new_size) {
+static jlong* resize_counters_array(jlong* old_counters, int current_size, int new_size) {
jlong* new_counters = NEW_C_HEAP_ARRAY_RETURN_NULL(jlong, new_size, mtJVMCI);
if (new_counters == nullptr) {
return nullptr;
@@ -745,6 +745,7 @@ static void ensure_join(JavaThread* thread) {
// Clear the native thread instance - this makes isAlive return false and allows the join()
// to complete once we've done the notify_all below. Needs a release() to obey Java Memory Model
// requirements.
+ assert(java_lang_Thread::thread(threadObj()) == thread, "must be alive");
java_lang_Thread::release_set_thread(threadObj(), nullptr);
lock.notify_all(thread);
// Ignore pending exception, since we are exiting anyway
@@ -1455,7 +1456,7 @@ void JavaThread::metadata_do(MetadataClosure* f) {
}
// Printing
-const char* _get_thread_state_name(JavaThreadState _thread_state) {
+static const char* _get_thread_state_name(JavaThreadState _thread_state) {
switch (_thread_state) {
case _thread_uninitialized: return "_thread_uninitialized";
case _thread_new: return "_thread_new";
@@ -2155,6 +2156,8 @@ void JavaThread::start_internal_daemon(JavaThread* current, JavaThread* target,
// on a ThreadsList. We don't want to wait for the release when the
// Theads_lock is dropped when the 'mu' destructor is run since the
// JavaThread* is already visible to JVM/TI via the ThreadsList.
+
+ assert(java_lang_Thread::thread(thread_oop()) == nullptr, "must not be alive");
java_lang_Thread::release_set_thread(thread_oop(), target); // isAlive == true now
Thread::start(target);
}
diff --git a/src/hotspot/share/runtime/javaThread.hpp b/src/hotspot/share/runtime/javaThread.hpp
index fb1355b852a..51e3cc31a52 100644
--- a/src/hotspot/share/runtime/javaThread.hpp
+++ b/src/hotspot/share/runtime/javaThread.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2021, Azul Systems, Inc. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
@@ -258,6 +258,7 @@ class JavaThread: public Thread {
public:
void inc_no_safepoint_count() { _no_safepoint_count++; }
void dec_no_safepoint_count() { _no_safepoint_count--; }
+ bool is_in_no_safepoint_scope() { return _no_safepoint_count > 0; }
#endif // ASSERT
public:
// These functions check conditions before possibly going to a safepoint.
diff --git a/src/hotspot/share/runtime/lockStack.cpp b/src/hotspot/share/runtime/lockStack.cpp
index b4a3bf1e8e6..d7dcbdda7e9 100644
--- a/src/hotspot/share/runtime/lockStack.cpp
+++ b/src/hotspot/share/runtime/lockStack.cpp
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2022, Red Hat, Inc. All rights reserved.
* Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -25,20 +26,30 @@
#include "precompiled.hpp"
#include "memory/allocation.hpp"
+#include "runtime/globals.hpp"
#include "runtime/lockStack.inline.hpp"
#include "runtime/safepoint.hpp"
#include "runtime/stackWatermark.hpp"
#include "runtime/stackWatermarkSet.inline.hpp"
#include "runtime/thread.hpp"
#include "utilities/copy.hpp"
+#include "utilities/debug.hpp"
+#include "utilities/globalDefinitions.hpp"
#include "utilities/ostream.hpp"
+#include
+
const int LockStack::lock_stack_offset = in_bytes(JavaThread::lock_stack_offset());
const int LockStack::lock_stack_top_offset = in_bytes(JavaThread::lock_stack_top_offset());
const int LockStack::lock_stack_base_offset = in_bytes(JavaThread::lock_stack_base_offset());
LockStack::LockStack(JavaThread* jt) :
_top(lock_stack_base_offset), _base() {
+ // Make sure the layout of the object is compatible with the emitted code's assumptions.
+ STATIC_ASSERT(sizeof(_bad_oop_sentinel) == oopSize);
+ STATIC_ASSERT(sizeof(_base[0]) == oopSize);
+ STATIC_ASSERT(std::is_standard_layout::value);
+ STATIC_ASSERT(offsetof(LockStack, _bad_oop_sentinel) == offsetof(LockStack, _base) - oopSize);
#ifdef ASSERT
for (int i = 0; i < CAPACITY; i++) {
_base[i] = nullptr;
@@ -62,11 +73,21 @@ uint32_t LockStack::end_offset() {
void LockStack::verify(const char* msg) const {
assert(LockingMode == LM_LIGHTWEIGHT, "never use lock-stack when light weight locking is disabled");
assert((_top <= end_offset()), "lockstack overflow: _top %d end_offset %d", _top, end_offset());
- assert((_top >= start_offset()), "lockstack underflow: _top %d end_offset %d", _top, start_offset());
+ assert((_top >= start_offset()), "lockstack underflow: _top %d start_offset %d", _top, start_offset());
if (SafepointSynchronize::is_at_safepoint() || (Thread::current()->is_Java_thread() && is_owning_thread())) {
int top = to_index(_top);
for (int i = 0; i < top; i++) {
assert(_base[i] != nullptr, "no zapped before top");
+ if (VM_Version::supports_recursive_lightweight_locking()) {
+ oop o = _base[i];
+ for (; i < top - 1; i++) {
+ // Consecutive entries may be the same
+ if (_base[i + 1] != o) {
+ break;
+ }
+ }
+ }
+
for (int j = i + 1; j < top; j++) {
assert(_base[i] != _base[j], "entries must be unique: %s", msg);
}
diff --git a/src/hotspot/share/runtime/lockStack.hpp b/src/hotspot/share/runtime/lockStack.hpp
index d5216793aea..17b0a1ca836 100644
--- a/src/hotspot/share/runtime/lockStack.hpp
+++ b/src/hotspot/share/runtime/lockStack.hpp
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2022, Red Hat, Inc. All rights reserved.
* Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -35,10 +36,12 @@ class OopClosure;
class outputStream;
class LockStack {
+ friend class LockStackTest;
friend class VMStructs;
JVMCI_ONLY(friend class JVMCIVMStructs;)
-private:
+public:
static const int CAPACITY = 8;
+private:
// TODO: It would be very useful if JavaThread::lock_stack_offset() and friends were constexpr,
// but this is currently not the case because we're using offset_of() which is non-constexpr,
@@ -51,6 +54,9 @@ class LockStack {
// We do this instead of a simple index into the array because this allows for
// efficient addressing in generated code.
uint32_t _top;
+ // The _bad_oop_sentinel acts as a sentinel value to elide underflow checks in generated code.
+ // The correct layout is statically asserted in the constructor.
+ const uintptr_t _bad_oop_sentinel = badOopVal;
oop _base[CAPACITY];
// Get the owning thread of this lock-stack.
@@ -75,14 +81,35 @@ class LockStack {
static uint32_t start_offset();
static uint32_t end_offset();
- // Return true if we have room to push onto this lock-stack, false otherwise.
- inline bool can_push() const;
+ // Returns true if the lock-stack is full. False otherwise.
+ inline bool is_full() const;
// Pushes an oop on this lock-stack.
inline void push(oop o);
+ // Get the oldest oop from this lock-stack.
+ // Precondition: This lock-stack must not be empty.
+ inline oop bottom() const;
+
+ // Is the lock-stack empty.
+ inline bool is_empty() const;
+
+ // Check if object is recursive.
+ // Precondition: This lock-stack must contain the oop.
+ inline bool is_recursive(oop o) const;
+
+ // Try recursive enter.
+ // Precondition: This lock-stack must not be full.
+ inline bool try_recursive_enter(oop o);
+
+ // Try recursive exit.
+ // Precondition: This lock-stack must contain the oop.
+ inline bool try_recursive_exit(oop o);
+
// Removes an oop from an arbitrary location of this lock-stack.
- inline void remove(oop o);
+ // Precondition: This lock-stack must contain the oop.
+ // Returns the number of oops removed.
+ inline size_t remove(oop o);
// Tests whether the oop is on this lock-stack.
inline bool contains(oop o) const;
diff --git a/src/hotspot/share/runtime/lockStack.inline.hpp b/src/hotspot/share/runtime/lockStack.inline.hpp
index be63c51d8e5..7a9874a9291 100644
--- a/src/hotspot/share/runtime/lockStack.inline.hpp
+++ b/src/hotspot/share/runtime/lockStack.inline.hpp
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2022, Red Hat, Inc. All rights reserved.
* Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+ * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -26,14 +27,20 @@
#ifndef SHARE_RUNTIME_LOCKSTACK_INLINE_HPP
#define SHARE_RUNTIME_LOCKSTACK_INLINE_HPP
+#include "runtime/lockStack.hpp"
+
#include "memory/iterator.hpp"
#include "runtime/javaThread.hpp"
-#include "runtime/lockStack.hpp"
#include "runtime/safepoint.hpp"
#include "runtime/stackWatermark.hpp"
#include "runtime/stackWatermarkSet.inline.hpp"
+#include "utilities/align.hpp"
+#include "utilities/globalDefinitions.hpp"
inline int LockStack::to_index(uint32_t offset) {
+ assert(is_aligned(offset, oopSize), "Bad alignment: %u", offset);
+ assert((offset <= end_offset()), "lockstack overflow: offset %d end_offset %d", offset, end_offset());
+ assert((offset >= start_offset()), "lockstack underflow: offset %d start_offset %d", offset, start_offset());
return (offset - lock_stack_base_offset) / oopSize;
}
@@ -42,8 +49,8 @@ JavaThread* LockStack::get_thread() const {
return reinterpret_cast(addr - lock_stack_offset);
}
-inline bool LockStack::can_push() const {
- return to_index(_top) < CAPACITY;
+inline bool LockStack::is_full() const {
+ return to_index(_top) == CAPACITY;
}
inline bool LockStack::is_owning_thread() const {
@@ -61,32 +68,132 @@ inline void LockStack::push(oop o) {
verify("pre-push");
assert(oopDesc::is_oop(o), "must be");
assert(!contains(o), "entries must be unique");
- assert(can_push(), "must have room");
+ assert(!is_full(), "must have room");
assert(_base[to_index(_top)] == nullptr, "expect zapped entry");
_base[to_index(_top)] = o;
_top += oopSize;
verify("post-push");
}
-inline void LockStack::remove(oop o) {
+inline oop LockStack::bottom() const {
+ assert(to_index(_top) > 0, "must contain an oop");
+ return _base[0];
+}
+
+inline bool LockStack::is_empty() const {
+ return to_index(_top) == 0;
+}
+
+inline bool LockStack::is_recursive(oop o) const {
+ if (!VM_Version::supports_recursive_lightweight_locking()) {
+ return false;
+ }
+ verify("pre-is_recursive");
+
+ // This will succeed iff there is a consecutive run of oops on the
+ // lock-stack with a length of at least 2.
+
+ assert(contains(o), "at least one entry must exist");
+ int end = to_index(_top);
+ // Start iterating from the top because the runtime code is more
+ // interested in the balanced locking case when the top oop on the
+ // lock-stack matches o. This will cause the for loop to break out
+ // in the first loop iteration if it is non-recursive.
+ for (int i = end - 1; i > 0; i--) {
+ if (_base[i - 1] == o && _base[i] == o) {
+ verify("post-is_recursive");
+ return true;
+ }
+ if (_base[i] == o) {
+ // o can only occur in one consecutive run on the lock-stack.
+ // Only one of the two oops checked matched o, so this run
+ // must be of length 1 and thus not be recursive. Stop the search.
+ break;
+ }
+ }
+
+ verify("post-is_recursive");
+ return false;
+}
+
+inline bool LockStack::try_recursive_enter(oop o) {
+ if (!VM_Version::supports_recursive_lightweight_locking()) {
+ return false;
+ }
+ verify("pre-try_recursive_enter");
+
+ // This will succeed iff the top oop on the stack matches o.
+ // When successful o will be pushed to the lock-stack creating
+ // a consecutive run at least 2 oops that matches o on top of
+ // the lock-stack.
+
+ assert(!is_full(), "precond");
+
+ int end = to_index(_top);
+ if (end == 0 || _base[end - 1] != o) {
+ // Topmost oop does not match o.
+ verify("post-try_recursive_enter");
+ return false;
+ }
+
+ _base[end] = o;
+ _top += oopSize;
+ verify("post-try_recursive_enter");
+ return true;
+}
+
+inline bool LockStack::try_recursive_exit(oop o) {
+ if (!VM_Version::supports_recursive_lightweight_locking()) {
+ return false;
+ }
+ verify("pre-try_recursive_exit");
+
+ // This will succeed iff the top two oops on the stack matches o.
+ // When successful the top oop will be popped of the lock-stack.
+ // When unsuccessful the lock may still be recursive, in which
+ // case the locking is unbalanced. This case is handled externally.
+
+ assert(contains(o), "entries must exist");
+
+ int end = to_index(_top);
+ if (end <= 1 || _base[end - 1] != o || _base[end - 2] != o) {
+ // The two topmost oops do not match o.
+ verify("post-try_recursive_exit");
+ return false;
+ }
+
+ _top -= oopSize;
+ DEBUG_ONLY(_base[to_index(_top)] = nullptr;)
+ verify("post-try_recursive_exit");
+ return true;
+}
+
+inline size_t LockStack::remove(oop o) {
verify("pre-remove");
assert(contains(o), "entry must be present: " PTR_FORMAT, p2i(o));
+
int end = to_index(_top);
+ int inserted = 0;
for (int i = 0; i < end; i++) {
- if (_base[i] == o) {
- int last = end - 1;
- for (; i < last; i++) {
- _base[i] = _base[i + 1];
+ if (_base[i] != o) {
+ if (inserted != i) {
+ _base[inserted] = _base[i];
}
- _top -= oopSize;
-#ifdef ASSERT
- _base[to_index(_top)] = nullptr;
-#endif
- break;
+ inserted++;
}
}
- assert(!contains(o), "entries must be unique: " PTR_FORMAT, p2i(o));
+
+#ifdef ASSERT
+ for (int i = inserted; i < end; i++) {
+ _base[i] = nullptr;
+ }
+#endif
+
+ uint32_t removed = end - inserted;
+ _top -= removed * oopSize;
+ assert(!contains(o), "entry must have been removed: " PTR_FORMAT, p2i(o));
verify("post-remove");
+ return removed;
}
inline bool LockStack::contains(oop o) const {
diff --git a/src/hotspot/share/runtime/objectMonitor.cpp b/src/hotspot/share/runtime/objectMonitor.cpp
index a8e2e571c5b..42832cc3c2a 100644
--- a/src/hotspot/share/runtime/objectMonitor.cpp
+++ b/src/hotspot/share/runtime/objectMonitor.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -39,6 +39,7 @@
#include "prims/jvmtiDeferredUpdates.hpp"
#include "prims/jvmtiExport.hpp"
#include "runtime/atomic.hpp"
+#include "runtime/globals.hpp"
#include "runtime/handles.inline.hpp"
#include "runtime/interfaceSupport.inline.hpp"
#include "runtime/javaThread.inline.hpp"
@@ -53,6 +54,7 @@
#include "runtime/sharedRuntime.hpp"
#include "services/threadService.hpp"
#include "utilities/dtrace.hpp"
+#include "utilities/globalDefinitions.hpp"
#include "utilities/macros.hpp"
#include "utilities/preserveException.hpp"
#if INCLUDE_JFR
@@ -312,7 +314,70 @@ void ObjectMonitor::ClearSuccOnSuspend::operator()(JavaThread* current) {
// -----------------------------------------------------------------------------
// Enter support
+bool ObjectMonitor::enter_for(JavaThread* locking_thread) {
+ // Used by ObjectSynchronizer::enter_for to enter for another thread.
+ // The monitor is private to or already owned by locking_thread which must be suspended.
+ // So this code may only contend with deflation.
+ assert(locking_thread == Thread::current() || locking_thread->is_obj_deopt_suspend(), "must be");
+
+ // Block out deflation as soon as possible.
+ add_to_contentions(1);
+
+ bool success = false;
+ if (!is_being_async_deflated()) {
+ void* prev_owner = try_set_owner_from(nullptr, locking_thread);
+
+ if (prev_owner == nullptr) {
+ assert(_recursions == 0, "invariant");
+ success = true;
+ } else if (prev_owner == locking_thread) {
+ _recursions++;
+ success = true;
+ } else if (prev_owner == DEFLATER_MARKER) {
+ // Racing with deflation.
+ prev_owner = try_set_owner_from(DEFLATER_MARKER, locking_thread);
+ if (prev_owner == DEFLATER_MARKER) {
+ // Cancelled deflation. Increment contentions as part of the deflation protocol.
+ add_to_contentions(1);
+ success = true;
+ } else if (prev_owner == nullptr) {
+ // At this point we cannot race with deflation as we have both incremented
+ // contentions, seen contention > 0 and seen a DEFLATER_MARKER.
+ // success will only be false if this races with something other than
+ // deflation.
+ prev_owner = try_set_owner_from(nullptr, locking_thread);
+ success = prev_owner == nullptr;
+ }
+ } else if (LockingMode == LM_LEGACY && locking_thread->is_lock_owned((address)prev_owner)) {
+ assert(_recursions == 0, "must be");
+ _recursions = 1;
+ set_owner_from_BasicLock(prev_owner, locking_thread);
+ success = true;
+ }
+ assert(success, "Failed to enter_for: locking_thread=" INTPTR_FORMAT
+ ", this=" INTPTR_FORMAT "{owner=" INTPTR_FORMAT "}, observed owner: " INTPTR_FORMAT,
+ p2i(locking_thread), p2i(this), p2i(owner_raw()), p2i(prev_owner));
+ } else {
+ // Async deflation is in progress and our contentions increment
+ // above lost the race to async deflation. Undo the work and
+ // force the caller to retry.
+ const oop l_object = object();
+ if (l_object != nullptr) {
+ // Attempt to restore the header/dmw to the object's header so that
+ // we only retry once if the deflater thread happens to be slow.
+ install_displaced_markword_in_object(l_object);
+ }
+ }
+
+ add_to_contentions(-1);
+
+ assert(!success || owner_raw() == locking_thread, "must be");
+
+ return success;
+}
+
bool ObjectMonitor::enter(JavaThread* current) {
+ assert(current == JavaThread::current(), "must be");
// The following code is ordered to check the most common cases first
// and to reduce RTS->RTO cache line upgrades on SPARC and IA32 processors.
diff --git a/src/hotspot/share/runtime/objectMonitor.hpp b/src/hotspot/share/runtime/objectMonitor.hpp
index 281067fe55c..d69cf5eba8c 100644
--- a/src/hotspot/share/runtime/objectMonitor.hpp
+++ b/src/hotspot/share/runtime/objectMonitor.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -295,6 +295,7 @@ class ObjectMonitor : public CHeapObj {
int contentions() const;
void add_to_contentions(int value);
intx recursions() const { return _recursions; }
+ void set_recursions(size_t recursions);
// JVM/TI GetObjectMonitorUsage() needs this:
ObjectWaiter* first_waiter() { return _WaitSet; }
@@ -329,6 +330,7 @@ class ObjectMonitor : public CHeapObj {
void operator()(JavaThread* current);
};
public:
+ bool enter_for(JavaThread* locking_thread);
bool enter(JavaThread* current);
void exit(JavaThread* current, bool not_suspended = true);
void wait(jlong millis, bool interruptible, TRAPS);
diff --git a/src/hotspot/share/runtime/objectMonitor.inline.hpp b/src/hotspot/share/runtime/objectMonitor.inline.hpp
index 36790925b71..b371663eeb3 100644
--- a/src/hotspot/share/runtime/objectMonitor.inline.hpp
+++ b/src/hotspot/share/runtime/objectMonitor.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1998, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -102,6 +102,12 @@ inline void ObjectMonitor::add_to_contentions(int value) {
Atomic::add(&_contentions, value);
}
+inline void ObjectMonitor::set_recursions(size_t recursions) {
+ assert(_recursions == 0, "must be");
+ assert(has_owner(), "must be owned");
+ _recursions = checked_cast(recursions);
+}
+
// Clear _owner field; current value must match old_value.
inline void ObjectMonitor::release_clear_owner(void* old_value) {
#ifdef ASSERT
diff --git a/src/hotspot/share/runtime/os.cpp b/src/hotspot/share/runtime/os.cpp
index dad10a7b9b4..9af30e28538 100644
--- a/src/hotspot/share/runtime/os.cpp
+++ b/src/hotspot/share/runtime/os.cpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -30,7 +30,6 @@
#include "classfile/vmClasses.hpp"
#include "classfile/vmSymbols.hpp"
#include "code/codeCache.hpp"
-#include "code/icBuffer.hpp"
#include "code/vtableStubs.hpp"
#include "gc/shared/gcVMOperations.hpp"
#include "interpreter/interpreter.hpp"
@@ -1289,7 +1288,7 @@ void os::print_location(outputStream* st, intptr_t x, bool verbose) {
st->print_cr(INTPTR_FORMAT " is an unknown value", p2i(addr));
}
-bool is_pointer_bad(intptr_t* ptr) {
+static bool is_pointer_bad(intptr_t* ptr) {
return !is_aligned(ptr, sizeof(uintptr_t)) || !os::is_readable_pointer(ptr);
}
@@ -1816,10 +1815,10 @@ char* os::reserve_memory(size_t bytes, bool executable, MEMFLAGS flags) {
return result;
}
-char* os::attempt_reserve_memory_at(char* addr, size_t bytes, bool executable) {
+char* os::attempt_reserve_memory_at(char* addr, size_t bytes, bool executable, MEMFLAGS flag) {
char* result = SimulateFullAddressSpace ? nullptr : pd_attempt_reserve_memory_at(addr, bytes, executable);
if (result != nullptr) {
- MemTracker::record_virtual_memory_reserve((address)result, bytes, CALLER_PC);
+ MemTracker::record_virtual_memory_reserve((address)result, bytes, CALLER_PC, flag);
log_debug(os)("Reserved memory at " INTPTR_FORMAT " for " SIZE_FORMAT " bytes.", p2i(addr), bytes);
} else {
log_debug(os)("Attempt to reserve memory at " INTPTR_FORMAT " for "
@@ -2114,33 +2113,37 @@ void os::pretouch_memory(void* start, void* end, size_t page_size) {
// We're doing concurrent-safe touch and memory state has page
// granularity, so we can touch anywhere in a page. Touch at the
// beginning of each page to simplify iteration.
- char* cur = static_cast(align_down(start, page_size));
+ void* first = align_down(start, page_size);
void* last = align_down(static_cast(end) - 1, page_size);
- assert(cur <= last, "invariant");
- // Iterate from first page through last (inclusive), being careful to
- // avoid overflow if the last page abuts the end of the address range.
- for ( ; true; cur += page_size) {
- Atomic::add(reinterpret_cast(cur), 0, memory_order_relaxed);
- if (cur >= last) break;
+ assert(first <= last, "invariant");
+ const size_t pd_page_size = pd_pretouch_memory(first, last, page_size);
+ if (pd_page_size > 0) {
+ // Iterate from first page through last (inclusive), being careful to
+ // avoid overflow if the last page abuts the end of the address range.
+ last = align_down(static_cast(end) - 1, pd_page_size);
+ for (char* cur = static_cast(first); /* break */; cur += pd_page_size) {
+ Atomic::add(reinterpret_cast