make `memorynew` intrinsic #55913

oscardssmith · 2024-09-28T05:08:09Z

This speeds up making new Memorys and allow the compiler to better understand what's going on, allowing for LLVM level escape analysis in some cases. There is more room to grow this (currently this only optimizes for fairly small Memory since bigger ones would require writing some more LLVM code, and we probably want a size limit on putting Memory on the stack to avoid stackoverflow. For larger ones, we could potentially inline the free so the Memory doesn't have to be swept by the GC, etc.

Benchmarks:

julia> f(T) = Memory{T}(undef, 2);

julia> @btime f(Int);
  7.879 ns (1 allocation: 48 bytes) #before
  3.959 ns (1 allocation: 48 bytes) #after 

julia> @btime f(Union{Int, Nothing})
  8.426 ns (1 allocation: 48 bytes) #before
  4.017 ns (1 allocation: 48 bytes) #after 

julia> @btime f(Any);
  8.637 ns (1 allocation: 48 bytes) #before
  3.920 ns (1 allocation: 48 bytes) #after 

julia> function g()
           m = Memory{Int}(undef, 2)
           for i in 1:2
              m[i] = i
           end
           m[1]+m[2]
       end

julia> @btime g()
  9.735 ns (1 allocation: 48 bytes) #before
  1.719 ns (0 allocations: 0 bytes) #after

src/cgutils.cpp

oscardssmith · 2024-10-04T03:54:41Z

@gbaraldi so with LLVM assertions enabled I'm getting

julia: /home/oscardssmith/Documents/Code/julia/deps/srccache/llvm-julia-18.1.7-2/llvm/lib/IR/Instructions.cpp:685: void llvm::CallInst::init(llvm::FunctionType*, llvm::Value*, llvm::ArrayRef<llvm::Value*>, llvm::ArrayRef<llvm::OperandBundleDefT<llvm::Value*> >, const llvm::Twine&): Assertion `(i >= FTy->getNumParams() || FTy->getParamType(i) == Args[i]->getType()) && "Calling a function with a bad signature!"' failed.

[398134] signal 6 (-6): Aborted
in expression starting at REPL[3]:1
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7108b6c2871a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN4llvm8CallInst4initEPNS_12FunctionTypeEPNS_5ValueENS_8ArrayRefIS4_EENS5_INS_17OperandBundleDefTIS4_EEEERKNS_5TwineE at /home/oscardssmith/Documents/Code/julia/usr/bin/../lib/libLLVM.so.18.1jl (unknown line)
CallInst at /home/oscardssmith/Documents/Code/julia/usr/include/llvm/IR/Instructions.h:1692 [inlined]
Create at /home/oscardssmith/Documents/Code/julia/usr/include/llvm/IR/Instructions.h:1540 [inlined]
CreateCall at /home/oscardssmith/Documents/Code/julia/usr/include/llvm/IR/IRBuilder.h:2398
CreateCall at /home/oscardssmith/Documents/Code/julia/usr/include/llvm/IR/IRBuilder.h:2420 [inlined]
emit_memorynew at /home/oscardssmith/Documents/Code/julia/src/cgutils.cpp:4555 [inlined]

which is on the line that does ctx.builder.CreateCall(prepare_call(jl_alloc_genericmemory_unchecked_func), {ptls, nbytes, cg_typ});. Any ideas?

gbaraldi · 2024-10-04T12:07:56Z

I'd print everyone involved here with the way I showed you yesterday thing->print(dbgs()), So print the func and the arguments

oscardssmith · 2024-10-04T20:40:44Z

This now works! For simple examples like Memory{Int}(undef, 3) it's about 7ns compared to 11 previously. Also this version is almost able to be analyzed by our escape analysis to delete the Memory when it doesn't (we just need to teach the llvm-alloc-helpers about gc_loaded, and for non @inbounds accesses we need to teach LLVM that the boundschecks are dead).

gbaraldi · 2024-10-04T20:49:49Z

As an example of what is possible.

Allocopt was able to go from

define i64 @julia_f_769() #0 !dbg !5 {
top:
  %pgcstack = call ptr @julia.get_pgcstack()
  %current_task1 = getelementptr inbounds i8, ptr %pgcstack, i64 -112, !dbg !14
  %memoryref_mem = call dereferenceable(40) ptr addrspace(10) @julia.gc_alloc_obj(ptr nonnull %current_task1, i64 40, ptr addrspace(10) addrspacecast (ptr @"+Core.GenericMemory#771.jit" to ptr addrspace(10))), !dbg !14
  %0 = addrspacecast ptr addrspace(10) %memoryref_mem to ptr addrspace(11), !dbg !14
  %1 = getelementptr inbounds { i64, ptr }, ptr addrspace(11) %0, i64 0, i32 1, !dbg !14
  %2 = call nonnull ptr @julia.pointer_from_objref(ptr addrspace(11) %0) #4, !dbg !14
  %3 = getelementptr inbounds i8, ptr %2, i64 16, !dbg !14
  store ptr %3, ptr addrspace(11) %1, align 8, !dbg !14
  store i64 3, ptr addrspace(11) %0, align 8, !dbg !14
  %memoryref_data4 = call ptr addrspace(13) @julia.gc_loaded(ptr addrspace(10) %memoryref_mem, ptr %3), !dbg !15
  store i64 2, ptr addrspace(13) %memoryref_data4, align 8, !dbg !15, !tbaa !20, !alias.scope !24, !noalias !27
  %memoryref_data11 = getelementptr inbounds i8, ptr addrspace(13) %memoryref_data4, i64 8, !dbg !32
  store i64 4, ptr addrspace(13) %memoryref_data11, align 8, !dbg !32, !tbaa !20, !alias.scope !24, !noalias !27
  %memoryref_data18 = getelementptr inbounds i8, ptr addrspace(13) %memoryref_data4, i64 16, !dbg !34
  store i64 5, ptr addrspace(13) %memoryref_data18, align 8, !dbg !34, !tbaa !20, !alias.scope !24, !noalias !27
  ret i64 11, !dbg !36
}

to. Removing the allocation. Which likely would allow it to just return the 11

define i64 @julia_f_769() #0 !dbg !5 {
top:
  %memoryref_mem = alloca [40 x i8], align 16
  %pgcstack = call ptr @julia.get_pgcstack()
  %current_task1 = getelementptr inbounds i8, ptr %pgcstack, i64 -112, !dbg !14
  call void @llvm.lifetime.start.p0(i64 40, ptr %memoryref_mem)
  %0 = freeze [40 x i8] undef, !dbg !14
  store [40 x i8] %0, ptr %memoryref_mem, align 1, !dbg !14
  %1 = getelementptr inbounds { i64, ptr }, ptr %memoryref_mem, i64 0, i32 1, !dbg !14
  %2 = getelementptr inbounds i8, ptr %memoryref_mem, i64 16, !dbg !14
  store ptr %2, ptr %1, align 8, !dbg !14
  store i64 3, ptr %memoryref_mem, align 8, !dbg !14
  %memoryref_data4 = call ptr addrspace(13) @julia.gc_loaded(ptr addrspace(10) null, ptr %2), !dbg !15
  store i64 2, ptr addrspace(13) %memoryref_data4, align 8, !dbg !15, !tbaa !20, !alias.scope !24, !noalias !27
  %memoryref_data11 = getelementptr inbounds i8, ptr addrspace(13) %memoryref_data4, i64 8, !dbg !32
  store i64 4, ptr addrspace(13) %memoryref_data11, align 8, !dbg !32, !tbaa !20, !alias.scope !24, !noalias !27
  %memoryref_data18 = getelementptr inbounds i8, ptr addrspace(13) %memoryref_data4, i64 16, !dbg !34
  store i64 5, ptr addrspace(13) %memoryref_data18, align 8, !dbg !34, !tbaa !20, !alias.scope !24, !noalias !27
  ret i64 11, !dbg !36
}

oscardssmith · 2024-10-04T20:55:28Z

julia> @btime Memory{Int8}(undef, 3)
  7.169 ns (1 allocation: 32 bytes) # before
  3.986 ns (1 allocation: 32 bytes) # after

giordano · 2024-10-04T20:56:11Z

FixedSizeArrays.jl is going to be much nicer 🙂

src/cgutils.cpp

@inbounds

combined with #55913, the compiler is smart enough to fully remove ``` function f() m = Memory{Int}(undef, 3) @inbounds m[1] = 2 @inbounds m[2] = 2 @inbounds m[3] = 4 @inbounds return m[1] + m[2] + m[3] end ```

src/ccall.cpp

@inbounds

combined with #55913, the compiler is smart enough to fully remove ``` function f() m = Memory{Int}(undef, 3) @inbounds m[1] = 2 @inbounds m[2] = 2 @inbounds m[3] = 4 @inbounds return m[1] + m[2] + m[3] end ```

giordano · 2024-10-11T19:56:13Z

Can you please add an llvm pass test for #56030 (comment) (removing all memory for a simple case where the Memory object doesn't escape)?

oscardssmith · 2024-10-11T20:01:18Z

Do you want an actual LLVM pass, or can I just write a test for 0 allocations?

giordano · 2024-10-11T20:04:19Z

I think an llvm test would be more robust, but probably a simple zero-allocation test would do the job as well.

oscardssmith · 2024-10-11T21:42:42Z

LOL. This test is so good it broke a doctest in performance tips. We're testing to show that you get allocations if you have "bad" code that allocates arrays, but now it doesn't allocate :laughing

oscardssmith · 2024-10-14T16:45:10Z

This is now on top of #55995 (to figure out why we weren't optimizing correctly), but other than that, I think this is good to go!

giordano · 2024-10-15T01:05:15Z

Maybe a test of no allocations in simple cases as discussed above? 🙂

The Value printer LLVM uses just prints the kind of instruction so it just shows call.

base/compiler/tfuncs.jl

vtjnash · 2024-10-23T18:57:57Z

base/essentials.jl

@@ -905,6 +906,12 @@ macro goto(name::Symbol)
 end

 # linear indexing
+function getindex(A::GenericMemory, i::Int)
+    @_noub_if_noinbounds_meta
+    @boundscheck ult_int(bitcast(UInt, sub_int(i, 1)), bitcast(UInt, A.length)) || throw_boundserror(A, (i,))


can we just make this the bootstrap method too? The boundscheck macro is just expanded simply to @_boundscheck &&, so this seems compatible

sounds reasonable

Co-authored-by: Jameson Nash <vtjnash@gmail.com>

oscardssmith commented Sep 28, 2024

View reviewed changes

src/cgutils.cpp Outdated Show resolved Hide resolved

nsajko added the arrays [a, r, r, a, y, s] label Sep 28, 2024

oscardssmith added the performance Must go faster label Sep 29, 2024

oscardssmith force-pushed the os/make-memorynew-tfunc branch from 6222082 to b65a483 Compare October 5, 2024 01:12

oscardssmith commented Oct 5, 2024

View reviewed changes

src/cgutils.cpp Outdated Show resolved Hide resolved

This was referenced Oct 7, 2024

teach llvm-alloc-helpers about gc_loaded #56030

Merged

don't capture arrays/tuples in BoundsError #43738

Open

vtjnash reviewed Oct 8, 2024

View reviewed changes

src/ccall.cpp Outdated Show resolved Hide resolved

oscardssmith force-pushed the os/make-memorynew-tfunc branch from b65a483 to 724b8c5 Compare October 11, 2024 19:51

oscardssmith mentioned this pull request Oct 13, 2024

Constructing a 67108864 TiB array succeeds incorrectly, corrupting memory #54328

Open

giordano mentioned this pull request Oct 13, 2024

Memory stores aren't vectorised in a for loop unless explicit at-inbounds is used JuliaArrays/FixedSizeArrays.jl#70

Open

oscardssmith force-pushed the os/make-memorynew-tfunc branch 3 times, most recently from 2c2b098 to e6e26ab Compare October 14, 2024 16:43

oscardssmith requested a review from vtjnash October 14, 2024 16:45

oscardssmith mentioned this pull request Oct 15, 2024

Make BoundsError lazy and move Memory boundscheck to Julia #56167

Draft

gbaraldi and others added 23 commits October 22, 2024 09:42

Improve remarks of the alloc opt pass slightly.

53616ec

The Value printer LLVM uses just prints the kind of instruction so it just shows call.

Update llvm-alloc-opt.cpp

81755e6

make memorynew intrinsic

cb79529

no segfault

7b6d78e

empty mem optimization

e3d60b6

optimized!

4b9c7b3

fix zeroinit

a9b4fae

it works

933770b

It's alive!

c18b684

use checked rather than overflow

86a7ba9

fix error path

21be744

sink ptls

3d614f0

debugs

80c58b6

fixes

5d1051a

fix optimization

4094ae9

typo

38f262c

cleanup and test

2ff0656

mark some tests broken

7e32bef

mark some as broken

516a702

move boundscheck to julia and improve codegen

8631d2a

fix-bootsrap

9dcaf13

off by 1

311f5ef

update overflow message

59d0d2f

oscardssmith mentioned this pull request Oct 22, 2024

Performance issue with typed_hvcat #56292

Open

switch wideint_t to overflowing operations

db95887

oscardssmith force-pushed the os/make-memorynew-tfunc branch from f280252 to db95887 Compare October 22, 2024 19:49

oscardssmith requested a review from gbaraldi October 22, 2024 19:49

vtjnash reviewed Oct 23, 2024

View reviewed changes

base/compiler/tfuncs.jl Outdated Show resolved Hide resolved

vtjnash reviewed Oct 23, 2024

View reviewed changes

address review

94f569a

Co-authored-by: Jameson Nash <vtjnash@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make `memorynew` intrinsic #55913

make `memorynew` intrinsic #55913

oscardssmith commented Sep 28, 2024 •

edited

Loading

oscardssmith commented Oct 4, 2024

gbaraldi commented Oct 4, 2024

oscardssmith commented Oct 4, 2024

gbaraldi commented Oct 4, 2024

oscardssmith commented Oct 4, 2024

giordano commented Oct 4, 2024

giordano commented Oct 11, 2024

oscardssmith commented Oct 11, 2024

giordano commented Oct 11, 2024

oscardssmith commented Oct 11, 2024

oscardssmith commented Oct 14, 2024

giordano commented Oct 15, 2024

vtjnash Oct 23, 2024

oscardssmith Oct 23, 2024

make memorynew intrinsic #55913

Are you sure you want to change the base?

make memorynew intrinsic #55913

Conversation

oscardssmith commented Sep 28, 2024 • edited Loading

oscardssmith commented Oct 4, 2024

gbaraldi commented Oct 4, 2024

oscardssmith commented Oct 4, 2024

gbaraldi commented Oct 4, 2024

oscardssmith commented Oct 4, 2024

giordano commented Oct 4, 2024

giordano commented Oct 11, 2024

oscardssmith commented Oct 11, 2024

giordano commented Oct 11, 2024

oscardssmith commented Oct 11, 2024

oscardssmith commented Oct 14, 2024

giordano commented Oct 15, 2024

vtjnash Oct 23, 2024

Choose a reason for hiding this comment

oscardssmith Oct 23, 2024

Choose a reason for hiding this comment

make `memorynew` intrinsic #55913

make `memorynew` intrinsic #55913

oscardssmith commented Sep 28, 2024 •

edited

Loading