Skip to content

Instantly share code, notes, and snippets.

@ax3l

ax3l/symbols_cpp.md

Last active April 23, 2024 08:57
Show Gist options
  • Save ax3l/ba17f4bb1edb5885a6bd01f58de4d542 to your computer and use it in GitHub Desktop.
Save ax3l/ba17f4bb1edb5885a6bd01f58de4d542 to your computer and use it in GitHub Desktop.
Download ZIP
Symbol Visibility 101 by Boris Staletic @bstaletic
Raw
symbols_cpp.md

Visible Symbols in C++ Projects

Intro

This is a spontaneous and verbatime log of a conversion with Boris Staletic @bstaletic from March, 24th 2018 on the pybind11 gitter. Thank you so much, Boris!

To confuse future readers, we decided to write down the discussion and I added typos and unnecessarily long sentences.

The issue came up with linking a CMake (library) target into a pybind11 module (in openPMD-api). pybind11 by default does extensive symbol hiding: -fvisibility=hidden and -fvisibility-inlines-hidden.

The Setup

A C++11 project that also wants to build a python module via pybind11.

CMakeLists.txt snippet:

# ...

#  library
add_Library(openPMD
    src/openPMD.cpp
    # ...
)

# ...

pybind11_add_module(openPMD.py MODULE
    src/binding/python/openPMD.cpp
    # ...
)

# ...

target_link_libraries(openPMD.py PRIVATE openPMD)

# ...

The Warning / Issue

On OSX with apple clang 8.1.0.8020042 (travis build matrix):

ld: warning:
direct access in function
  'pybind11::detail::erase_all(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)'
from file
  'CMakeFiles/openPMD.py.dir/src/binding/python/openPMD.cpp.o'
to global weak symbol
  'std::__1::char_traits<char>::eq(char, char)'
from file
  'lib/libopenPMD.a(Series.cpp.o)'
means the weak symbol cannot be overridden at runtime.
This was likely caused by different translation units being compiled with different visibility settings.

Principles

So should I do symbol hiding and explicit symbol exposure for my library in general as well?

B: "Do it for all OS's. It will make your shared object smaller, so it will also load/import slightly faster. I even had a measurable performance gain after hiding symbols by default."

Why does it warn me in the first place? Could something go wrong?

B: "Why is it warning you? Because the linker found something odd - having an exposed symbol that contains a hidden one (or is it the other way around?). How did that happen? Pybind11 forced visibility of a symbol. Can something go wrong? Assuming every target is consistent, no. What I mean by consistent? Everything that was intended to be used "from outside" should be exposed in the symbol-hidden target. Why did GCC not warn you? Could be a number of reasons."

B:

It decided you know what you are doing and that everything is fine.
It decided to fix the visibility.
It decided that the symbol makes no sense and deleted it.

B: "At this point I am speculating."

How to Expose Symbols Explicitly as a Library Author

First of all one needs to hide the symbols, generally. GCC, ICC & Clang accept -fvisibility=hidden and -fvisibility-inlines-hidden. MSVC performs symbol hiding by default.

The according cmake target properties are CXX_VISIBILITY_PRESET and VISIBILITY_INLINES_HIDDEN.

When does symbol hiding happen? Compile-time or link-time?

B: "At link time."

How do I explicitly expose symbols if everything is -fvisibility=hidden by default?

B: "Usually it is VISIBILITY_MACRO struct foo{};, with exceptions it is struct VISIBILITY_MACRO foo : std::exception {};.

If you go for the former form for exceptions the compiler will be confused and discard the visibility attribute. As for downstream projects, well... you should have enough tests to make sure you have exposed everything.

If you hide [user-defined] exceptions they will just not get thrown [in downstream code]."

Don't forget to expose globals, enums, etc. as well!

VISIBILITY_MACRO

Since symbol visibility is controled by compiler-specific flags, here is how to define a common macro, e.g. VISIBILITY_MACRO or VISIBILITY_EXPORT or PROJECTNAME_EXPORT:

compiler attribute notes
g++ __attribute__((visibility("default")))
clang++ __attribute__((visibility("default")))
icpc __attribute__((visibility("default")))
pgc++ unsupported, not planned as of 04/2018 unsupported
nvcc unsupported as of 10.2.89 Nvidia RFE 2767443
xlc++ __attribute__((visibility("default")))
msvc __declspec( dllexport ) symbols hidden by default

Standardization Efforts

  • C++17 proposal for [[visible]]: P0276R0 -> not adopted? -> probably managed via C++20 modules

Checking Visibility

B:

nm -C <shared_object> | grep <symbol_name>

and

nm -CD <shared_object> | grep <symbol_name>

B:

If the symbol doesn't exist at all it won't be anywhere.
If the symbol appears only in the first output, it's hidden.
If the symbol appears in both outputs, it is exposed.

Useful links

Epilogue

Again, thank you so much, Boris!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment