Management of dependencies in wrapper

In this tutorial we will show how an external package can define its dependencies. There is basically 2 ways:

• dependencies to other external packages.
• explicit platform requirement through configurations.

Step 5 : add a new managed version

We continue to work on the yaml-cpp project defined in previous steps. This time we want to port another existing version of this external project: version 0.5.2. In this earlier version than the previously defined one, yaml-cpp depends on another external package called boost. boost is a famous project aiming at providing useful APIs to C++ programmers.

5.1 : create a new version folder in the wrapper

cd <yaml-cpp>/src
mkdir 0.5.2
touch 0.5.2/CMakeLists.txt
touch 0.5.2/deploy.cmake

5.2 : define deployment procedure (in src/0.5.2/deploy.cmake):

install_External_Project( PROJECT yaml-cpp
                          VERSION 0.5.2
                          URL https://github.com/jbeder/yaml-cpp/archive/release-0.5.2.tar.gz
                          ARCHIVE yaml_0.5.2.tar.gz
                          FOLDER yaml-cpp-release-0.5.2)

build_CMake_External_Project( PROJECT yaml-cpp FOLDER yaml-cpp-release-0.5.2 MODE Release
                        DEFINITIONS BUILD_GMOCK=OFF BUILD_GTEST=OFF BUILD_SHARED_LIBS=ON YAML_CPP_BUILD_TESTS=OFF
                        YAML_CPP_BUILD_TESTS=OFF YAML_CPP_BUILD_TOOLS=OFF YAML_CPP_BUILD_CONTRIB=OFF gtest_force_shared_crt=OFF
                        COMMENT "shared libraries")

if(NOT EXISTS ${TARGET_INSTALL_DIR}/lib OR NOT EXISTS ${TARGET_INSTALL_DIR}/include)
  message("[PID] ERROR : during deployment of yaml-cpp version 0.5.2, cannot install yaml-cpp in worskpace.")
  return_External_Project_Error()
endif()

As you may have notice there is less user entries set by build_CMake_External_Project. This is because version 0.5.2 of yaml-cpp does not define these cache entries.

5.3 : define resulting components (in src/0.5.2/CMakeLists.txt):

PID_Wrapper_Version(
  VERSION 0.5.2 DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
PID_Wrapper_Dependency(PACKAGE boost FROM VERSION 1.55.0 TO VERSION 1.65.1) #After boost 1.65.1 => cannot compile
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      EXPORT boost/boost-headers)

5.4 build the version 0.5.2

Now that the description is completed, we can try configuring and building the package.

cd <pid-workspace>/wrappers/yaml-cpp
pid build version=0.5.2

You can face two situations:

• the build failed. This is surely explainable by the fact that boost is not installed in your operating system. This way you saw that there is an implicit dependency between yaml-cpp and boost projects.
• the build succeeded. Well this is in fact not a good situation because you did not see that there is an implicit dependency !

Step 6 : add a dependency to an external package

What we need to do now is to specify the dependency between yaml-cpp and boost projects. We will specify this dependency as an external dependency. This is possible because boost is provided as an external package in PID, in other words a wrapper exists for this project.

6.1 Describe the dependency (in src/0.5.2/CMakeLists.txt):

PID_Wrapper_Version(
  VERSION 0.5.2 
  DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
PID_Wrapper_Dependency(PACKAGE boost FROM VERSION 1.55.0 TO VERSION 1.65.1)
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp)
PID_Wrapper_Component_Dependency(COMPONENT libyaml EXPORT EXTERNAL boost-headers PACKAGE boost)

or shorter (recommended):

PID_Wrapper_Version(
  VERSION 0.5.2 DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
PID_Wrapper_Dependency(PACKAGE boost FROM VERSION 1.55.0 TO VERSION 1.65.1)
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      EXPORT boost/boost-headers)

There are two operation to do, quite the same way as for native packages:

• define the dependency between external packages using the PID_Wrapper_Dependency command. In the example yaml-cpp depends on a specific range of version of the boost project.
• define the dependencies between components of yaml-cpp and components of boost, using either PID_Wrapper_Component_Dependency or EXPORT/DEPEND argument of PID_Wrapper_Component. In the example libyaml depends on boost-headers. We use the keyword EXPORT because some headers of boost are included in public headers of yaml-cpp. To detect that we did:

cd <pid-workspace>/wrappers/yaml-cpp/build/0.5.2/yaml-cpp-release-0.5.2
grep -nr boost include/

In output you should see something like:

./yaml-cpp/noncopyable.h:12:	// this is basically boost::noncopyable
./yaml-cpp/node/detail/node_ref.h:13:#include <boost/utility.hpp>
./yaml-cpp/node/detail/node_ref.h:19:		class node_ref: private boost::noncopyable
./yaml-cpp/node/detail/impl.h:11:#include <boost/type_traits.hpp>
./yaml-cpp/node/detail/impl.h:25:		struct get_idx<Key, typename boost::enable_if_c<boost::is_unsigned<Key>::value && !boost::is_same<Key, bool>::value>::type> {
./yaml-cpp/node/detail/impl.h:40:		struct get_idx<Key, typename boost::enable_if<boost::is_signed<Key> >::type> {
./yaml-cpp/node/detail/node_iterator.h:11:#include <boost/iterator/iterator_facade.hpp>
./yaml-cpp/node/detail/node_iterator.h:12:#include <boost/utility/enable_if.hpp>
./yaml-cpp/node/detail/node_iterator.h:54:		class node_iterator_base: public boost::iterator_facade<
...

For instance we see that the header boost/utility.hpp is included in yaml-cpp/node/detail/node_ref.h.

6.2 Modify deploy script to manage the dependency at build time:

install_External_Project( PROJECT yaml-cpp
                          VERSION 0.5.2
                          URL https://github.com/jbeder/yaml-cpp/archive/release-0.5.2.tar.gz
                          ARCHIVE yaml_0.5.2.tar.gz
                          FOLDER yaml-cpp-release-0.5.2)

get_External_Dependencies_Info(PACKAGE boost ROOT root_folder INCLUDES boost_include)

build_CMake_External_Project( PROJECT yaml-cpp FOLDER yaml-cpp-release-0.5.2 MODE Release
                        DEFINITIONS BUILD_GMOCK=OFF BUILD_GTEST=OFF BUILD_SHARED_LIBS=ON YAML_CPP_BUILD_TESTS=OFF
                        YAML_CPP_BUILD_TESTS=OFF YAML_CPP_BUILD_TOOLS=OFF YAML_CPP_BUILD_CONTRIB=OFF gtest_force_shared_crt=OFF
                        Boost_NO_SYSTEM_PATHS=ON Boost_INCLUDE_DIR=${boost_include} BOOST_INCLUDEDIR=${boost_include} BOOST_ROOT=${root_folder} BOOSTROOT=${root_folder}
                        COMMENT "shared libraries")

if(NOT EXISTS ${TARGET_INSTALL_DIR}/lib OR NOT EXISTS ${TARGET_INSTALL_DIR}/include)
  message("[PID] ERROR : during deployment of yaml-cpp version 0.5.2, cannot install yaml-cpp in worskpace.")
  return_External_Project_Error()
endif()

In previous description we changed 2 things:

• call to get_External_Dependencies_Info(PACKAGE boost ROOT root_folder INCLUDES boost_include) is used to get all necessary information about boost package in use. Here what is needed are the include and root boost project folders.
• arguments like Boost_NO_SYSTEM_PATHS=ON Boost_INCLUDE_DIR=${boost_include} BOOST_INCLUDEDIR=${boost_include} BOOST_ROOT=${root_folder} BOOSTROOT=${root_folder} are passed to CMake definitions of the external project. It simply sets the include and root folders where to find boost. These arguments are required by the CMake script of the original project to configure boost adequately and so be able to compile the yaml-cpp code with the adequzte version of boost. Here the boost version used will be the one provided by PID.

6.3 build again the version 0.5.2

cd <pid-workspace>/wrappers/yaml-cpp
pid build version=0.5.2

This time everything should work as expected and furthermore the build process may have automatically deploy the boost external package.

Step 7 : define a required configuration for target platform

Another way to deal with dependencies is by using platform configurations, as for native packages. Using this kind of dependencies should be limited as far as possible but is sometimes required or more easy to use than external packages.

This last step of the tutorial simply shows how to write the dependency to boost as a check of the platform configuration. Indeed the boost wrapper also defines a way to test / and install systems packages. But in this example we will use it directly to get also all the compilation/linker flags used to configrue libyaml.

The tutorial consists in rewriting the previous example for version 0.5.2 of yaml-cpp.

7.1 : describe the dependency (in src/0.5.2/CMakeLists.txt):

PID_Wrapper_Version(
  VERSION 0.5.2 
  DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
PID_Wrapper_Configuration(CONFIGURATION boost)
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      INCLUDES include boost_INCLUDE_DIRS)

What changed:

• the call to PID_Wrapper_Configuration explicitly checks if the target platform has boost system library installed. This call generates various variables, including boost_INCLUDE_DIRS that contains the path to the include folder for boost.
• the call to PID_Wrapper_Component now adds the variable boost_INCLUDE_DIRS to the set of includes exported by the project. We use the variable instead of its value (i.e. boost_INCLUDE_DIRS rather than ${boost_INCLUDE_DIRS}) because we want the resulting binary package to be relocatable: the variable boost_INCLUDE_DIRS will be interpreted anytime the binary package of yaml-cpp will be used (while if its value was used this variable would be only interpreted when the external project is built and so will match only the current build environment in use).

7.2 : build the project

There is no more modification required because the deployment procedure is still valid. So we simply build the project:

cd <pid-workspace>/wrappers/yaml-cpp
pid build version=0.5.2

Everything should work as expected. The procedure may have installed the boost system package automatically if it was not already present in OS.

7.3 : Discussion about dependencies version management

One very bad thing about the previous design is that we will face troubles anytime we need to use yaml-cpp because it will require to use system version of boost which can conflict with other packages using the PID version. Furthermore it is more convenient to use external project wrapper because they define the components and so dependency are far more simple to write.

We would rather prefer use the system version “as if” it was a PID built version. This way we would be able to use the description of boost components. To do this we need to change a bit the description:

PID_Wrapper_Version(
  VERSION 0.5.2 
  DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
# PID_Wrapper_Configuration(CONFIGURATION boost) #now optional
PID_Wrapper_Dependency(boost VERSION SYSTEM)
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      EXPORT boost/boost-headers)

Then an explicit dependency to the boost external package is specified (PID_Wrapper_Dependency), and this time there is a version constraint to SYSTEM version. The SYSTEM keyword simply tells that the dependency version is the version installed in operating system and retrieved (automatically) using boost configuration. So we have to memorize that using the SYSTEM keyword for version constraint requires the configuration with same name to exist. The call to PID_Wrapper_Configuration becomes optional because the call to PID_Wrapper_Dependency with SYSTEM version constraint will do the job automatically.

Using the SYSTEM version constraint is possible if external project wrappers are provided with a way to build equivalent system version of a given PID version. You can test it on boost wrapper by doing:

cd <pid-workspace>/wrappers/boost
pid build version=1.58.0 os_variant=true

The use of os_variant argument:

• will check if the version of boost installed in operating system matches the version to build.
• if everything is OK then the command generates an external package for this boost version but without using the deploy script : instead of compiling boost it rather generates symlinks to equivalent binary components and include folders that are installed in system.

This way users can use packages description provided for the given version of boost while in fact using the boost version installed in operating system.

Hete if OS installed version (that you do not control) does not match the required version to build (here 1.58.0), an error is generated. To know the currently installed version of boost we can do:

cd <pid-workspace>/wrappers/boost
pid eval_system_check

The output should be something like:

--- Returned variables ---
- boost_VERSION = 1.74.0
- boost_LIBRARY_DIRS = 
- boost_INCLUDE_DIRS = /usr/include
- boost_RPATH = /usr/lib/x86_64-linux-gnu/libboost_atomic.so;/usr/lib/x86_64-linux-gnu/libboost_chrono.so;/usr/lib/x86_64-linux-gnu/libboost_container.so;/usr/lib/x86_64-linux-gnu/libboost_context.so;...
- boost_LINK_OPTIONS = -lboost_atomic;-lboost_chrono;...
- boost_COMPONENTS = atomic;chrono;container;context;coroutine;date_time;exception;fiber;filesystem;graph;graph_parallel;iostreams;locale;log;log_setup;
 --- Final contraints in binary ---
- libraries=atomic,chrono,container,context,coroutine,date_time,exception,fiber,filesystem,graph,graph_parallel,iostreams,locale,log,log_setup,math_c99,math_c99f,...
- soname=libboost_atomic.so.1.74.0,libboost_chrono.so.1.74.0,libboost_container.so.1.74.0,libboost_context.so.1.74.0,libboost_coroutine.so.1.74.0,...
- version=1.74.0
Built target eval_system_check

Now the boost_VERSION variable gives you the version of the boost installed version, on the PC used to write this doc 1.74.0. So to correctly build the os variant of version 1.74.0 the good command is:

cd <pid-workspace>/wrappers/boost
pid build version=1.74.0 os_variant=true

Now if you adapt previous example with your OS installed version of boost the wrapper build process should be OK.

Main problem with previous description is sill that the yaml-cpp version defined must use the OS installed version of boost in the end, which is not so flexible. We want to be able to use any version of boost whether it is OS installed or PID built one. To do this we simply have to go back to the initial description:

PID_Wrapper_Version(
  VERSION 0.5.2 DEPLOY deploy.cmake
  SONAME 0.5 #define the extension name to use for shared objects
)
PID_Wrapper_Dependency(PACKAGE boost FROM VERSION 1.55.0 TO VERSION 1.65.1) #After boost 1.65.1 => cannot compile
PID_Wrapper_Component(COMPONENT libyaml  ALIAS yaml-cpp
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      EXPORT boost/boost-headers)

Indeed with this description we just say that any version of boost in the range of allowed versions (here 1.55 to 1.65.1) can be used to build yaml-cpp and this also include OS installed version !!

The choice of the OS variant at yaml-cpp level:

• can be decided by the build process : the OS or non os variant will be used depending on constraints coming from packages using yaml-cpp as a dependency. For instance if a package using yaml-cpp also forces the use of system variant of boost (e.g. PID_Wrapper_Dependency(boost VERSION SYSTEM)) then the os variant version will be used OR an error will be generated if the OS version is not in the range of allowed versions (here 1.55 to 1.65.1).
• can be forced using dedicated variables:

pid build version=0.5.2 -D0.5.2_boost_ALTERNATIVE_VERSION_USED=SYSTEM

Previous build command just ask PID to selection othe OS variant for boost dependency when building yaml-cpp version 0.5.2.

If now you just want to use the version 1.59.0 of boost instead:

pid build version=0.5.2 -D0.5.2_boost_ALTERNATIVE_VERSION_USED=1.59.0