WIP: Upgrade arrow to 0.9.0 and create shared libraries

[saulshanabrook]

No description provided.

[saulshanabrook]

Right now I can't get the mutable_data from the arrow::Buffer. I think this is because it is an inline function and so the compiler removes it after compiling. I guess it depends on the compiler, and I can't find any definite reference for this behavior, but I did find this stackoverflow answer

The linker may not see inline functions at all. They are usually compiled straight into the code that calls them (i.e., the code is used in place of a function call).

If the compiler chooses not to inline the function (since it is merely a hint), I'm not sure, but I think the compiler emits it as a normal non-inline function and somehow annotates it so the linker just picks the first copy it sees and ignores the others.

As far as I can tell, the mutable_data method doesn't show up in libarrow.a:

$ llvm-nm arrow/release/release/libarrow.a | c++filt
...
0000000000001154 short GCC_except_table8
                 U __Unwind_Resume
00000000000009f0 T arrow::PoolBuffer::Resize(long long, bool)
0000000000000940 T arrow::PoolBuffer::Reserve(long long)
0000000000000750 T arrow::PoolBuffer::PoolBuffer(arrow::MemoryPool*)
0000000000000690 T arrow::PoolBuffer::PoolBuffer(arrow::MemoryPool*)
00000000000008d0 T arrow::PoolBuffer::~PoolBuffer()
0000000000000860 T arrow::PoolBuffer::~PoolBuffer()
00000000000007e0 T arrow::PoolBuffer::~PoolBuffer()
0000000000000c30 T arrow::MutableBuffer::MutableBuffer(std::__1::shared_ptr<arrow::Buffer> const&, long long, long long)
0000000000000ba0 T arrow::MutableBuffer::MutableBuffer(std::__1::shared_ptr<arrow::Buffer> const&, long long, long long)
0000000000000e40 T arrow::MutableBuffer::~MutableBuffer()
0000000000000cc0 T arrow::MutableBuffer::~MutableBuffer()
00000000000003b0 T arrow::AllocateBuffer(arrow::MemoryPool*, long long, std::__1::shared_ptr<arrow::Buffer>*)
0000000000000720 T arrow::ResizableBuffer::~ResizableBuffer()
0000000000000ad0 T arrow::SliceMutableBuffer(std::__1::shared_ptr<arrow::Buffer> const&, long long, long long)
                 U arrow::default_memory_pool()
0000000000000cf0 T arrow::AllocateResizableBuffer(arrow::MemoryPool*, long long, std::__1::shared_ptr<arrow::ResizableBuffer>*)
0000000000000500 T arrow::Buffer::FromString(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<arrow::Buffer>*)
0000000000000240 T arrow::Buffer::FromString(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, arrow::MemoryPool*, std::__1::shared_ptr<arrow::Buffer>*)
0000000000000eb0 T arrow::Buffer::~Buffer()
0000000000000e80 T arrow::Buffer::~Buffer()
0000000000000680 T arrow::Buffer::CheckMutable() const
0000000000000000 T arrow::Buffer::Copy(long long, long long, arrow::MemoryPool*, std::__1::shared_ptr<arrow::Buffer>*) const
0000000000000170 T arrow::Buffer::Copy(long long, long long, std::__1::shared_ptr<arrow::Buffer>*) const
0000000000000200 T arrow::Buffer::Equals(arrow::Buffer const&) const
00000000000001c0 T arrow::Buffer::Equals(arrow::Buffer const&, long long) const
                 U std::__1::__shared_weak_count::__get_deleter(std::type_info const&) const
...

Steps to reproduce

Right now, I am just testing development on my Mac.

git clone git@github.com:plures/pxnd.git
cd pxnd
git checkout upgrade-arrow-version
make libplasma/plasma.so

On my machine, this successfully downloads and compiles Arrow, but then it fails to find the arrow::Buffer::mutable_data() symbol:

clang++ -std=c++11 -Weverything -pedantic  --system-header-prefix=_xnd.c --system-header-prefix=xnd.h --system-header-prefix=ndtypes.h --system-header-prefix=arrow/ --system-header-prefix=plasma/ --system-header-prefix=Python.h -O0 -g  -shared -fPIC -o libplasma/plasma.so -I ../arrow/cpp/src/ libplasma/plasma.cc arrow/release/release/libplasma.a arrow/release/release/libarrow.a
Undefined symbols for architecture x86_64:
  "arrow::Buffer::mutable_data()", referenced from:
      _objectBuffer_data in plasma-66ded1.o
      _objectBuffer_metadata in plasma-66ded1.o
      _plasmaClient_create in plasma-66ded1.o
ld: symbol(s) not found for architecture x86_64
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libplasma/plasma.so] Error 1

[skrah]

On Linux with gcc this works (I had to change the include path for client.h):

stefan@canberra:~/tmp/pxnd$ clang++ -std=c++11 -Weverything -pedantic  --system-header-prefix=_xnd.c --system-header-prefix=xnd.h --system-header-prefix=ndtypes.h --system-header-prefix=arrow/ --system-header-prefix=plasma/ --system-header-prefix=Python.h -O0 -g  -shared -fPIC -o libplasma/plasma.so -I arrow/cpp/src/ libplasma/plasma.cc arrow/release/release/libplasma.a arrow/release/release/libarrow.a

But my buffer.h shows this (does not look inline):

  uint8_t* mutable_data() {
#ifndef NDEBUG
    CheckMutable();
#endif
    return mutable_data_;
  }

[saulshanabrook]

@skrah it did work for me too, when I re-cloned the repo and tried it. I must have had some files left over from arrow 0.8.0.

I think it's inline because it's defined in the class definition. It sets it as inline without any keyword.

[saulshanabrook]

I think arrow 0.9.0 has been pulled from pypi, because I can't install it anymore. I know they had some trouble with the last release.

[saulshanabrook]

It has been released, so it should be possible to continue working on this. I am new with building these types of C libraries, so if anyone wants to work on this, I would base your work off this branch and feel free to change around how things are built if you have better ideas.

Also, we should figure out what APIs the xnd/ndtypes Python c apis should expose, so we don't have to include their files.