In this tutorial we learn how to define new wrappers. A wrapper is a specific deployment unit used to port existing third party projects into PID in such a way that using these external projects becomes very close to using native packages from CMake perspective. In the remaining parts of this tutorial, we show as an example how to wrap the project named yaml-cpp.

Step 1: create the wrapper repository and project

First of all we need to create the project for the wrapper and its online repository.

1.1: create online repository

Go into your git hosting server (gitlab for LIRMM) and create a new repository with same name as the external project wrapped.

Important Note: when creating the repository please ensure that the repository is empty after creation. Some hosting service propsoe to automatically generate some stuff, but deactivate all these options because PID needs an empty repository for first connection with a package. For instance by default Gitlab may propose to initialize the repository with a README. Please unselect this option.

In this example we create a repository named yaml-cpp. Then copy the SSH address of this git repository.

1.2: create the wrapper

First of all remove any contribution (find and reference files) of the yaml-cpp project in official contribution space. Indeed as yaml-cpp already exists PID will not accept to create it again. We suggest to start this tutorial from an empty workspace and then removing existing references to yaml-cpp. If you do not already have one, for the official contribution space pid-contributions:

cd <somewhere> && git clone https://gite.lirmm.fr/pid/pid-workspace.git
cd pid-workspace && pid configure
pid contributions cmd=churl space=pid publish=<address of pid-contributions fork>
pid contributions cmd=delete space=pid content=yaml-cpp

Now create the wrapper:

cd <pid-workspace>
pid create wrapper=yaml-cpp url=<url previously copied>
cd <pid-workspace>/wrappers/yaml-cpp

The wrapper project should have been created and put into folder <pid-workspace>/wrappers. In the following sections we use <yaml-cpp> as a short name for the path to <pid-workspace>/wrappers/yaml-cpp.

Step 2: describe the content of the wrapper

Now first thing to do is to define adequate meta information of the wrapper. Edit the CMakeListst.txt file in <yaml-cpp> and paste the code:

cmake_minimum_required(VERSION 3.15.7)
set(WORKSPACE_DIR ${CMAKE_SOURCE_DIR}/../.. CACHE PATH "root of the PID workspace")
list(APPEND CMAKE_MODULE_PATH ${WORKSPACE_DIR}/cmake) # using generic scripts/modules of the workspace
include(Wrapper_Definition NO_POLICY_SCOPE)

project(yaml-cpp)

PID_Wrapper(
		AUTHOR     		Your Name
		INSTITUTION		Your Institution
		MAIL 			    your mail
		ADDRESS 		  <address of the online repository>
		YEAR 			    2018
		LICENSE 		  MIT
		DESCRIPTION 	"wrapper for yaml-cpp project : A YAML parser and emitter in C++")

PID_Original_Project(
		AUTHORS "yaml-cpp authors"
		LICENSES "MIT License"
		URL https://github.com/jbeder/yaml-cpp)

build_PID_Wrapper()

Explanations:

  • include(Wrapper_Definition NO_POLICY_SCOPE) is used to import the API for writing wrappers.

  • PID_Wrapper command (equivalent to declare_PID_Wrapper) transform the current CMake project into a PID wrapper. Arguments to provide are more or less the same as for native packages:

    • the LICENSE argument must be provided. One good approach is to use the same license than the external project itself, but you can choose any available license because the wrapper is considered as another project than the external project it wraps.
    • AUTHOR, INSTITUTION, MAIL refer to the contact author of the wrapper project and NOT to authors of the external project being wrapped.

  • PID_Original_Project (equivalent to define_PID_Wrapper_Original_Project_Info) provides meta-data about original external project being wrapped:

    • AUTHORS argument defines the authors of external project.
    • LICENSES argument lists the licenses that apply to the external project. You can use any string to describe the license.

    • URL argument define the address of the web site where we can find information about the project. You can typically use it to target a github/gitlab project.

    • build_PID_Wrapper configures the wrapper and create its build commands.

Step 3: wrapping a version of yaml-cpp

Now the first thing to decide is which version(s) of the external project you have to port. Indeed, unlike native package, we have to explicitly manage versions of the existing project one by one. Indeed depending on version the content of the external project (components it creates for instance) may differ more or less.

For now we decide to port version 0.6.2 of yaml-cpp.

3.1: create the source folder for version 0.6.2

cd <yaml-cpp>/src
mkdir 0.6.2
touch 0.6.2/CMakeLists.txt
touch 0.6.2/deploy.cmake

Wrapping a version of an external project simply consists in creating a folder with target version as name, in th example the folder 0.6.2. Then we need at least two files:

  • a CMakeLists.txt that describes the content of yaml-cpp for version 0.6.2.
  • a script file that define the deployment procedure. We decide to call it deploy.cmake.

3.2: Editing the version

Now we have to edit those two files to describe the wrapper. First edit the file src/0.6.2/CMakeLists.txt:

PID_Wrapper_Version(VERSION 0.6.2 DEPLOY deploy.cmake)

That’s it for now, the PID_Wrapper_Version simply tells that the version currently wrapped in the 0.6.2 and the CMake script that implement deployment procedure is named deploy.cmake. For now we keep the deploy script empty.

3.3: Download and Build the external project for first time

First thing to do is to test a complete build of the external project “the normal way” (i.e. without using PID). This way you can:

  • find from where sources of the project can be downloaded
  • understand what is the procedure to build and install the code.

For instance yaml-cpp basic install procedure looks like something as:

cd <somewhere>
mkdir yamlcpp_build
cd yamlcpp_build
wget https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.2.tar.gz
tar xvf yaml-cpp-0.6.2.tar.gz
cd yaml-cpp-yaml-cpp-0.6.2
mkdir build && mkdir install
cd build && cmake -DCMAKE_INSTALL_PREFIX=../install ..
make
make install

You can have a look in the install folder we created, it contains binary artefacts generated by the yaml-cpp project. You should see at least to folder: include and lib. When looking into lib we see that the library libyaml-cpp.a has been generated. We can also see that there are some unwanted artefact that have been generated: libgmock.a and libgtest.a. These libraries are not related to yaml-cpp directly but are used to test the project itself, so they are not usefull for end users like us. We simply want to remove any content that is not useful for end users. Also we prefer using shared object rather than archive libraries, so we need to see if there is a way to generate the shared version of the library instead. We have to look at CMake options we can use to do that:

cd <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/build
ccmake .. #type q to quit

From available configuration entries we deduce that we should deactivate a set of options:

cd <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/install
rm -Rf *
cd ../build
cmake -DBUILD_SHARED_LIBS=ON -DBUILD_GMOCK=OFF -DBUILD_GTEST=OFF -Dgmock_build_tests=OFF -DYAML_CPP_BUILD_TESTS=OFF -DYAML_CPP_BUILD_CONTRIB=OFF -DYAML_CPP_BUILD_TOOLS=OFF ..
make
make install

Now look into the install/lib folder: only the library libyaml-cpp.so.0.6.2. (and its symlinks) has been generated, exactly what we expected.

It is most of time a good idea to understand what are the dependencies of the external project. In the current example yaml-cpp is provided with a .pc file (look in <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/install/lib/pkg-config folder) so we can use it to quickly now if yaml-cpp have any.

cd <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/install/lib/pkg-config
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:`pwd`
pkg-config --static --libs yaml-cpp
pkg-config --static --cflags yaml-cpp
  • --cflags can help you know what are the required include folders and so deduce if yaml-cpp requires headers of third party packages. In the example the only include folder exported by yaml-cpp contain its own headers so we deduce it has no compile time dependency.
  • --libs can help you know what are the required libraries and so deduce if yaml-cpp requires binaries of third party packages at linktime or runtime. In the example the output provides flags (e.g. -lyaml-cpp) that only target yaml-cpp content which means the package does not a priori require third party binaries.

Another more general technic to get information about binaries dependency is to use commands to consult content of shared object:

  • listing all (direct and undirect) runtime dependencies:
cd <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/install/lib
ldd libyaml-cpp.so.0.6

output will be something like:

linux-vdso.so.1 =>  (0x00007ffc441b3000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f976243b000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9762225000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9761e5b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9761b52000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9762a37000)

This command is usefull to see how the system globally resolve runtime dependencies.

  • listing only the direct runtime dependencies:
cd <somewhere>/yamlcpp_build/yaml-cpp-yaml-cpp-0.6.2/install/lib
readelf -d libyaml-cpp.so.0.6

output will be something like:

...
0x0000000000000001 (NEEDED)             Bibliothèque partagée: [libstdc++.so.6]
0x0000000000000001 (NEEDED)             Bibliothèque partagée: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Bibliothèque partagée: [libc.so.6]
0x000000000000000e (SONAME)             Bibliothèque soname: [libyaml-cpp.so.0.6]
0x000000000000000c (INIT)               0x14c60
0x000000000000000d (FINI)               0x688cc

This command is useful to know only the direct dependencies (all those marked as NEEDED), here for instance libstdc++.so.6. Here we can deduce that there is no other binary dependency than the standard ones.

It is also useful to know the exact soname of the shared object, in this case libyaml-cpp.so.0.6. From this we can deduce that the SONAME is written with the 2 first digits of the version number (0.6). This is an important information because PID enforce a policy for external package: the name of the shared object binary file must exactly match the SONAME in object binary. So in PID using libyaml-cpp.so or even libyaml-cpp.so.0 is not correct because these SONAME extension are not sufficiently restrictives on version, while using libyaml-cpp.so.0.6.2 is too restrictive.

We have all required informations to start writing the wrapper, now its is time to automate this procedure using a PID wrapper for the yaml-cpp project.

3.4: Write the deployment procedure using PID wrapper API

Now we go back to our PID wrapper yaml-cpp to start writing this procedure:

pid cd yaml-cpp

We reproduce the previous deployment procedure into the deploy script src/0.6.2/deploy.cmake. Edit the file and paste the code:

install_External_Project( 
  PROJECT yaml-cpp
  VERSION 0.6.2
  URL https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.2.tar.gz
  ARCHIVE yaml-cpp-0.6.2.tar.gz
  FOLDER yaml-cpp-yaml-cpp-0.6.2)

build_CMake_External_Project( 
  PROJECT yaml-cpp FOLDER yaml-cpp-yaml-cpp-0.6.2 MODE Release
  DEFINITIONS BUILD_GMOCK=OFF BUILD_GTEST=OFF BUILD_SHARED_LIBS=ON YAML_CPP_BUILD_TESTS=OFF YAML_CPP_BUILD_TESTS=OFF YAML_CPP_BUILD_TOOLS=OFF YAML_CPP_BUILD_CONTRIB=OFF gtest_force_shared_crt=OFF
  COMMENT "shared libraries"
)

if(NOT EXISTS ${TARGET_INSTALL_DIR}/lib OR NOT EXISTS ${TARGET_INSTALL_DIR}/include)
  message("[PID] ERROR : during deployment of yaml-cpp version 0.6.2, cannot install yaml-cpp in worskpace.")
  return_External_Project_Error()
endif()

The deploy script is basically divided in two parts:

  • First part consists in downloading and extracting the project. This is achieved using the install_External_Project script:

    • PROJECT argument specifies the name of the project.
    • VERSION specifies the version being installed.
    • URL is the url where the archive for that version can be found. In the example the address https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.2.tar.gz has been used.
    • ARCHIVE is the name of the downloaded archive. In the example yaml-cpp-0.6.2.tar.gz.
    • FOLDER is the name of the root folder contained in archive. In the example yaml-cpp-yaml-cpp-0.6.2.

  • Second part consists in configuring the external project to make it build properly. Since the yaml-cpp project is based on CMake we can use the build_CMake_External_Project command provided by PID to automate this part. This command will configure then build and finally install the external project directly into the PID workspace:

    • PROJECT argument specifies the name of the project.
    • FOLDER is the name of the root folder contained in archive. In the example yaml-cpp-yaml-cpp-0.6.2.
    • MODE specifies the chosen build mode. In the example we want only Release binaries to be generated.
    • DEFINITIONS argument specifies the list of CMake options that must be set before building the external project. We simply reproduce options values we defined in previous section, so that the build generates only the libyaml-cpp.so library.

The function return_External_Project_Error is used to automatically exit the install procedure if the package has not been correctly installed (here simply test that include and lib folders exist).

3.4: Write the description

Last step before building consists in writing the description of the external package generated by the wrapper. In the current example we need to tell PID that the package yaml-cpp contains a library libyaml-cpp.so. Edit src/0.6.2/CMakeLists.txt and paste the code:

#declaring a new known version
PID_Wrapper_Version(
  VERSION 0.6.2 DEPLOY deploy.cmake
  CMAKE_FOLDER lib/cmake/yaml-cpp
  SONAME 0.6 #define the extension name to use for shared objects
)

#now describe the content
PID_Wrapper_Environment(LANGUAGE CXX[std=11])#requires a full capable c++11 toolchain

#component for shared library version
PID_Wrapper_Component(
  COMPONENT libyaml ALIAS yaml-cpp
  CXX_STANDARD 11
  INCLUDES include
  SHARED_LINKS yaml-cpp
)

We perform some changes:

  • PID_Wrapper_Version specifies the SONAME that will be used by default for all shared objects generated by the external project. It also define the current version using VERSION and may also define the path (relative to the install root directory) to the folder that contains cmake module files generated by yaml-cpp projet (using CMAKE_FOLDER).
  • PID_Wrapper_Environment defines a constraint on C++ compiler toolchain that can be used to build the project. Here we need a C++11 compatible toolchain.
  • call to PID_Wrapper_Component is used to define the component libyaml. Headers of the component can be found in include folder (path relative to the install root). To specificy library the simplest way is to give its base name (here yaml-cpp). This base name is used to compute the final name of the shared library, using soname information, standard path for libraries (lib) and prefix/suffix rules specific to the OS. For instance on macosx the result will be lib/liblibyaml.dylib and lib/liblibyaml.so.0.6 on linux.

Note if component have different SONAME

Sometimes you may face external project where components have non uniform SONAMEs. In this case you can directly set the SONAME in PID_Wrapper_Component, in this case this soname will be used only for that component. We could have something like:

#declaring a new known version
PID_Wrapper_Version(
  VERSION 0.6.2 DEPLOY deploy.cmake
  CMAKE_FOLDER lib/cmake/yaml-cpp
)

#now describe the content
PID_Wrapper_Environment(LANGUAGE CXX[std=11])#requires a full capable c++11 toolchain

#component for shared library version
PID_Wrapper_Component(COMPONENT libyaml ALIAS yaml-cpp
                      CXX_STANDARD 11
                      INCLUDES include
                      SHARED_LINKS yaml-cpp
                      SONAME 0.6
)

3.5: Build the version

Now that the description is completed, we can try configuring and building the package.

cd <pid-workspace>/wrappers/yaml-cpp
pid build version=0.6.2

The build command does all the job as for native packages. The main difference is that, when using wrappers, you need to specify which version you want to build (e.g. version=0.6.2). Indeed many versions of the same external project can be managed by a single wrapper and each version may require different configure/build actions and may produce different software artefacts.

Now look into the install folder of yaml-cpp and see that everything goes well:

cd <pid-workspace>/install/<platform>/yaml-cpp/0.6.2
ls include
ls lib

If you want to save your current work:

cd <pid-workspace>/wrappers/yaml-cpp
git add --all && git commit -m "first commit"
git push origin master

Now if everything goes as expected you may want to mark the last commit as being the last one that modified the wrapper for version 0.6.2.

cd <pid-workspace>/wrappers/yaml-cpp
pid memorizing version=0.6.2

This last command (re)tags the wrapper repository with the version tag v0.6.2. This is used to launch the continuous integration process so that the wrapper will regenerate binaries and eventually publish to a framework. But this is beyond the topic of this tutorial.

Step 4 : register the external package

Now your external package is ready to be used, because the PID system can find and deploy a released version of your code. But since the wrapper is not distributed, no one except yourself can use it, which is not so interesting ! As for native packages, the registering is done in 2 steps.

Step 4.1 : referencing the external package in private workspace

The registering phase is achieved by using a dedicated PID command:

cd <pid-worskspace>
pid register package=yaml-cpp

This last command performs two different operations:

  • the first operation consists in generating the reference files of the package and putting them into adequate contribution space, exactly the same way as for native packages. These are the CMake script files that are used by PID to identify the package: Findyaml-cpp.cmake (use to find the external package in local workspace) and ReferExternalyaml-cpp.cmake (use to find the external package online). This operation can be achieved manually by doing:
cd <yaml-cpp>
pid referencing
  • the second operation consists in generating a commit for the pid contribution space and publishing them to your fork repository of PID official contribution space:
cd <pid-worskspace>
pid contributions cmd=publish space=pid

You can have a look at your fork of pid-contributions, it contains the latest commit that added yaml-cpp contribution files (probably just after the commit that removed them). Now every user of your own PID contribution space is able to know the existence of your package and where to find it. But of course there are certainly a limited number of persons who work with it !

Step 4.2 : publishing the external package in official workspace

The real publication of your external package will be done when it will be referenced into the official contribution space. This is simply achieved by using the gitlab/github server interface to propose a merge request between your fork of pid-contributions of original repository.

Once your merge request is accepted by the administrator of the official pid-contributions repository, your external package is registered and anyone can use the wrapper you just defined ! Please do not do that for this tutorial as yaml-cpp is already (well) defined in official

Now you know how to manage an external package wrapper, let’s see how to manage many versions and dependencies.