A framework is a specific deployment unit used to group a set of packages that share common topics or mechanisms. For instance there is a framework called pid, that groups a set of native packages and external wrappers implementing runtime mechanisms of PID (e.g. the pid-rpath package) or general utilities for software developpers (e.g. wrappers for boost and yaml-cpp, package pid-os-utilities). The result of building a framework is a static site that provides a centralized documentation on these packages. Frameworks have been primarily be designed to work together a continuous integration (CI) process in a Gitlab environment.

Contributing to an existing framework

This first tutorial explains how a package or wrapper can contribute to a framework. To keep it simple we want a new package <newpack> to be able to contribute to the pid framework.

Step 1: edit a package that contributes to a framework

Let’s make the package my-first-package, created in introduction tutorials, contributing to the framework pid. We suppose that this package is already in your local workspace (or you can simply create a new one). In the root CMakeLists.txt of my-first-package:

cmake_minimum_required(VERSION 3.8.2)
set(WORKSPACE_DIR ${CMAKE_SOURCE_DIR}/../.. CACHE PATH "root of the packages workspace directory")
list(APPEND CMAKE_MODULE_PATH ${WORKSPACE_DIR}/cmake) # using generic scripts/modules of the workspace
include(Package_Definition NO_POLICY_SCOPE)

project(my-first-package)

PID_Package(AUTHOR 		    Robin Passama
            INSTITUTION	  LIRMM
            YEAR          2015
            ADDRESS       git@gite.lirmm.fr:passama/my-first-package.git
            LICENSE       MIT
            DESCRIPTION   TODO: input a short description of package toto utility here
            VERSION       0.2.0
		)

Simply change the fields with your personnal information and set adequate address for git repository.

# publication of package
PID_Category(programming/tutorial)
PID_Publishing(
      PROJECT https://gite.lirmm.fr/own/my-first-package
			FRAMEWORK pid
			DESCRIPTION my-first-package is used in tutorial.
			ALLOWED_PLATFORMS x86_64_linux_stdc++11)

build_PID_Package()

We added two lines to the previous content of CMakeLists.txt:

  • PID_Publishing (or equivalent declare_PID_Publishing) tells that the package produces a static site. The FRAMEWORK argument specifies that it contributes to the pid framework generated static site. PROJECT argument gives the address of the package online project site and DESCRIPTION provides a long description of package utility. Finally the ALLOWED_PLATFORMS argument tells for which platforms the publishing will be performed, for now only one platform can be specified due to CI process limitations.

  • PID_Category (or equivalent add_PID_Package_Category) tells to which categories the package belongs to. If such categories exist in the framework definitions then the framework static site classifies the package is given categories, in left side panel of the resulting static site. Look at pid-rpath package in the left side panel, it belongs to the category programming/resources management that is also defined by the framework so it has been put in the corresponding section.

The same logic applies for wrappers by using calls to PID_Wrapper_Category (or equivalent add_PID_Wrapper_Category) and PID_Wrapper_Publishing (or equivalent declare_PID_Wrapper_Publishing) which makes it possible to publish wrappers more or less the same way as native packages.

Step 2: make a package contributing to a framework

Now the package will really contribute to pid framework:

cd <pid-worskspace>/packages/my-first-package
pid site synchro=false

The site command simply publishes the static site of the package. In the present case it published it inside the framework. The argument synchro=false is used to avoid to update of the framework git repository (which anyway will be impossible because you have no write access to this project).

To have a look at the final result:

cd <pid-workspace>/sites/frameworks/pid
pid serve

The framework being a deployment unit is has its own git repository, CMake description and related commands. The serve command is used to generate local web server that hosts the generated static site of the framework. This way you can see the result locally anytime you modifiy the content of a framework.

The output gives something like:

[100%] [PID] Serving the static site of the framework pid ...
Configuration file: none
            Source: /home/robin/soft/PID/pid-workspace/sites/frameworks/pid/build/generated
       Destination: /home/robin/soft/PID/pid-workspace/sites/frameworks/pid/build/generated/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 0.749 seconds.
 Auto-regeneration: enabled for '/home/robin/soft/PID/pid-workspace/sites/frameworks/pid/build/generated'
Configuration file: none
    Server address: http://127.0.0.1:4000/pid-framework/
  Server running... press ctrl-c to stop.

As explained simply do a ctrl-c to stop the web server. To see the result use a web browser to open the address http://127.0.0.1:4000/pid-framework.

The package my-first-package should appear only in the section All Packages of the left side panel because the category used programming/tutorial is not defined in pid framework. If you click on the link to my-first-package in left side panel, the center panel now shows the documentation of the package.

Step 3: add developper info to the static site

Warning: keep the serve command running in another terminal !

For now the package related pages do not give access to API doc or other developper info. To do this you have to modify a bit the package description:

...
PID_Category(programming/tutorial)
PID_Publishing(
      PROJECT https://gite.lirmm.fr/passama/my-first-package
      FRAMEWORK pid
      DESCRIPTION my-first-package is used in tutorial.
      PUBLISH_DEVELOPMENT_INFO
      ALLOWED_PLATFORMS x86_64_linux_stdc++11)


build_PID_Package()

The PUBLISH_DEVELOPMENT_INFO simply tells PID that all available development info, like doxygen APID documentation website, will be published and referenced by the framework.

Remmebr to set your personnal info in PROJECT argumument.

Then you have to rebuild your package with adequate options activated and then republish.

cd <pid-worskspace>/packages/my-first-package
pid configure -DBUILD_API_DOC=ON -DBUILD_STATIC_CODE_CHECKING_REPORT=ON
#you can set to ON: BUILD_AND_RUN_TESTS,
# BUILD_TESTS_IN_DEBUG and BUILD_COVERAGE_REPORT if your package has tests defined
pid build
pid site synchro=false

After this last command the site should have been updated, so simply reload the static site in the web browser to see the result. In the my-first-package panel, the top bar should now provide API and Static checks from Developpers menu. By clicking on API item a new window open the API doc doxygen generated site (since BUILD_API_DOC package option has been activated).

Step 4: publish binary archives using the framework

Frameworks can also be used as respositories for binary archives of native and external packages. The main benefit of binary archives is that they really speed up the deployment process by avoiding unecessary compilation.

To do so, modify a bit the CMakeLists.txt of the package:

...
PID_Category(programming/tutorial)
PID_Publishing(	PROJECT https://gite.lirmm.fr/own/my-first-package
                FRAMEWORK pid
                DESCRIPTION my-first-package is used in tutorial.
                PUBLISH_BINARIES PUBLISH_DEVELOPMENT_INFO
                ALLOWED_PLATFORMS x86_64_linux_stdc++11)


build_PID_Package()

The PUBLISH_BINARIES option simply tells PID that if binary archive have been generated for the package they will be put and referenced into the framework static site.

Then you have to rebuild your package with adequate options activated and then republish.

cd <pid-worskspace>/packages/my-first-package
pid configure -DBUILD_API_DOC=ON -DBUILD_STATIC_CODE_CHECKING_REPORT=ON -DGENERATE_INSTALLER=ON
pid build
pid site synchro=false

After this last command the site should have been updated, so simply reload the static site in the web browser to see the result. In the my-first-package panel, the item Binaries of the top bar menu Developpers item now prints a page that lists all available binaries for the package. Binary archives are classififed according to version first and target platforms. You should see an archive for the version/platform you are currently building your package with (since GENERATE_INSTALLER option is active binary archives have been generated by build process and site command has published them into framework).

Automatic contribution to a framework

Anytime a package contributes to a framework using the PID_Publishing command, a CI file is generated (.gitlab-ci.yml). This file is used to configure the continuous integration process of the package: anytime the package is released its contribution to the framework will be automatically updated. So for instance if the package publishes binary archives using the PUBLISH_BINARIES option, then corresponding archives for the version released will be generated by the build process taking place in the CI process and then put into the framework.

Create a new framework

Now we now how a package contributes to a framework we have to see how to define new frameworks. A framework is a deployment unit in PID so there is a dedicated CMake API to describe frameworks content and also a specific development process for them.

Step 1: create the framework project

Let’s suppose we want to define a framework called myframework, we first need to create the repository. In gitlab, create a new project called myframework in your gitlab workspace. Then we need to create the framework in your local workspace and connect it to this repository:

cd <pid-worskspace>
pid create framework=myframework url=git@gite.lirmm.fr:passama/myframework.git

This command create the framework repository of myframework in local workspace in <pid-workspace>/sites/frameworks. The framework branching model is rather simple: there is only one master branch and only one origin remote.

Then look at the generated file <pid-workspace>/sites/frameworks/myframework/CMakeLists.txt, it should look something like:

cmake_minimum_required(VERSION 3.8.2)
set(WORKSPACE_DIR ${CMAKE_SOURCE_DIR}/../../.. CACHE PATH "root of the frameworks workspace directory")
list(APPEND CMAKE_MODULE_PATH ${WORKSPACE_DIR}/cmake) # using generic scripts/modules of the workspace
include(Framework_Definition NO_POLICY_SCOPE)

project(myframework)

PID_Framework(AUTHOR 		robin
			        ADDRESS git@gite.lirmm.fr:passama/myframework.git
              YEAR 		2019
              LICENSE 	MIT
              DESCRIPTION 	"TODO: input a short description of framework myframework utility here"
              SITE		"TODO: input the web site address "
		)


build_PID_Framework()

The logic is more or less the same as for packages, but this time we use the frameworks API (see include(Framework_Definition NO_POLICY_SCOPE)) to declare a framework using the PID_Framework command. Arguments have xactly the same logic as for packages. The SITE argument is used to specify the address of the resulting website. First set adequate author name/institution as well as year, description and license information.

The folder <pid-workspace>/sites/frameworks/myframework is organized according to a predefined scheme:

  • the share folder contains miscellaneous elements, like CI scripts
  • the src folder contains the whole static site description for myframework:

    • _external folder will contain all information about external packages that contribute to the framework.
    • _packages folder will contain all information about native packages that contribute to the framework.
    • assets folder contains global data that will be used “as is” and will not be interpreted by the jekyll generation process.
    • pages folder contains global data that will be interpreted by the jekyll generation process. It typically contains markdown files that themselves contain the general information about the framework. For instance, the page you are currently reading as been generated from a markdown file contained in the pid framework. This folder contains some default pages:

      • introduction.md that generates an introduction page for the framework
      • install.md that describes the installation procedure for the framework.
      • tutorial.md that gives access to some tutorials.
      • more.md that provides access to advanced topics and detailed information.

Each of these markdown files corresponds to an entry point of the global framework documentation. More precisely in the finally generated static site, each of these files will be accessible from an item of the Documentation menu of top menu bar. As a first example you can simply look in the current web page top menu bar, because this static site has been generated from the pid framework.

Step 2: writing framework documentation content

Each of these files can contain any number of reference to any number of other pages or section. The pages folder can as well contain any number of files, either mardown or html files, you can also add other kind of element like images and of course subfolders. This way you can structure your documentation’s filesystem more or less the way you want.

When writing a web page file (html or markdown) you have to use some specific jekyll headers. For instance for now the introduction.md file should be like this:

---
layout: page
title: Introduction
---

The lines between --- is the jekyll header:

  • the layout attribute must be page.
  • the title of default generated markdown files must be let unchanged.
  • any title of user specific files can have any title.

Headers are use by the jekyll generation process to input the content of the file into the final web site. All pages of the framework can also contain Liquid tags, that are interpreted by jekyll to generate specific actions. The most usefull tag is highlight:

## Example code
{% highlight c %}
i = i +1;
{% endhighlight %}

The text you are currently reading has itself been generated from a markdown page and the beginning of this tutorial is defined by the following text:

## Create a new framework

Now we now how a package contributes to a framework we have to see how to define new frameworks. A framework is a deployment unit in PID so there is a dedicated CMake API to describe frameworks content and also a specific development process for them.

### Step 1: create the framework project

Let's suppose we want to define a framework called `myframework`, we first need to create the repository. In gitlab, create a new project called myframework in your gitlab workspace. Then we need to create the framework in your local workspace and connect it to this repository:

{% highlight bash %}
cd <pid-worskspace>
pid create framework=myframework url=git@gite.lirmm.fr:passama/myframework.git
{% endhighlight %}

This command create the framework repository of `myframework` in local workspace in `<pid-workspace>/sites/frameworks`. The framework branching model is rather simple: there is only one **master** branch and only one **origin** remote.

The previous content is enough to start working on a simple framework description. First let’s consider <framework> as the path to myframework folder in your workstation’s filesystem.

1) Edit the file /src/pages/introduction.md and write this content:

---
layout: page
title: Introduction
---

## Talking about your code


## Example code
{% highlight c %}
i = i +1;
{% endhighlight %}

2) Regenerate the static site:

cd <framework>
pid configure
pid build

3) Print the result

In another terminal launch the serve command

cd <framework>
pid serve

That’s it, look at the address provided by the serve command output with your web browser and click on Introduction item of the top menu Documentation to see the result. You simply have to reproduce this process to populate your framework documentation.

You can also improve a bit the look of the resulting static site by specifying images such as a logo and a banner.

  • the logo image is used as an icon at left of the top menu bar. The icon has an hyperlink that points to the welcome page. Save the image of the PID icon as /src/assets/icon.jpg (right click on the icon then "save as ...").
  • the banner is a bigger image that is put at the top of the welcome page. GO to the welcome page and save the image of the PID banner as /src/assets/banner.jpg.

Modify a bit the CMakeLists.txt:

...
PID_Framework(AUTHOR 		    Robin Passama
              ADDRESS       git@gite.lirmm.fr:passama/myframework.git
              YEAR          2019
              LICENSE       MIT
              DESCRIPTION   "TODO: input a description of framework myframework utility here"
              SITE          "TODO: input the web site address "
              LOGO          icon.jpg
              BANNER        banner.jpg
		)
...

The arguments LOGO and BANNER are used to set the path to the corresponding images. All path are expressed relative to the assets folder.

Then regenerate the static site:

cd <framework>
pid configure
pid build

And see the result by reloading the resulting static site in your web browser. The welcome page and the top menu bar of myframework should look like the ones of pid.

Step 4: structuring the contribution process with categories

Except global documentation all the contributions will be generated by packages and wrappers that belongs to this framework (anytime their site command is executed). These units will finally fill the folders _packages and _external respectively.

Categories can then be used to structure to present contributing packages in a more structured way in the left side panel. The goal of this panel is to have a quick overview of available packages and wrappers and thanks to their categorization, to have an indication about their utility. For instance if you look at the left side panel of the current website, you can quickly understand that the package pid-rpath provides some programming utilities to manage runtime resources. To generate such a classification in left side panel there are 2 things to do:

  • Define a category test in myframework. Modify again a bit the CMakeLists.txt:
...
PID_Framework(...)

PID_Framework_Category(test) #simply add this line to previous code
#keep following lines UNCHANGED
...
project(my-first-package)
PID_Package(
  ...
  )

# CHANGE the following lines
PID_Category(test)
PID_Publishing(PROJECT           https://gite.lirmm.fr/passama/my-first-package
               FRAMEWORK         myframework
               DESCRIPTION       my-first-package is used in tutorial.
               ALLOWED_PLATFORMS x86_64_linux_stdc++11)

build_PID_Package()#keep it unchanged as usual

In the previous code we changed the framework of my-first-package to be now myframework. Furthermore the package now belong to the test category (call to PID_Category(test)) that has been defined in myframework.

Now to contribute to the framework just do:

cd <pid-worskspace>/packages/my-first-package
pid configure
pid site synchro=false

And see the result after reloading your static site in your web browser. You should now see that your package is referenced in left side panel under the test category.

Step 5: Publishing the framework

5.1 Providing online addresses

Until now all we did is to generate the framework site locally. Obviously what we trully want is to publih it online so that other people can read your doc ! This is achieved while using the default CI process of frameworks, that consists in generating pages of the web site.

One important thing to do is to specify the address of the finally generated static site. This requires to set the SITE argument of PID_Framework command. You should also give all the necessary meta information : the description of the framework and contact author information are generally important (but not so mush in this tutorial of course):

...
PID_Framework(AUTHOR        Robin Passama
              MAIL          passama@lirmm.fr
              ADDRESS       git@gite.lirmm.fr:passama/myframework.git
              YEAR          2019
              LICENSE       MIT
              DESCRIPTION   "This is a test frameork..."
              SITE          http://passama.lirmm.net/myframework
              PROJECT       https://gite.lirmm.fr/passama/myframework
)
...

So use the previous pattern and simply changes values for author, description, static site and online project addresses, etc. Let’s remember that CI process is based on gitlab-CI so naming convention of gitlab applies everywhere you need to set an address:

  • Gitlab project pages (in PROJECT) generate a path with pattern https://<gitlab server address>/<namespace>/<project>.
  • Gitlab pages generates by default static site in a path with pattern: http://<namespace>.<gitlab pages server address>/<project>. See SITE argument for an example.

Now simply regenerate the static site:

cd <pid-worskspace>/sites/frameworks/myframework
pid configure
pid build

And see the result ! Everything should more or less look the same as previously.

5.2 Configuring the CI in your framework project

For the gitlab pages to work, you need to activate the CI process in your gitlab project.

Go to your project’s settings and allow for pipelines to run (in “Permissions” section). This way the CI process can be automatically launched whenever you push to the framework repository.

Another important configuration is to add the runner as a member of the framework. Indeed the CI runners will update the framework anytime a package or wrapper contributes to the framework (new doc, new binaries) and so thoses runners must have rights to push to framework packages to do this update. The administrator of your PID environment should have configured a specific CI user in your gitlab server that is the identity used by CI runners whenever they need to push to a specific repository. So:

  • ask your administrator the name of this specific CI user.
  • add it as a member of myframework. Its role must give it permission to push to protected branches, in Gitlab 11.7 the role is called Maintainer.

Once done your are now ready to publish your framework !

5.3 Publishing the framework

As already said, the publishing process is automatically managed by the CI process, so everything you have to do is to commit and push:

cd <pid-worskspace>/sites/frameworks/myframework
git add -A
git commit -m "first published version"
git push origin master

The CI process takes place. When finished the static site has been generated and can be accessed at http://passama.lirmm.net/myframework (change with your own address).

Step 6: Publishing packages and wrappers

Now that the CI process is in place at framework level, the publishing of packages and wrappers can be automated.

6.1 Automatic update of framework static site

First update the content of myframework with content coming from my-first-package:

cd <pid-worskspace>/packages/my-first-package
pid site

This last command simply updates the content of myframework with pages and binaries related to my-first-package and automatically push the framework repository (because the argument synchro=false has been removed).

You can notice that the CI process has been automatically launched and generates the static website.

6.2 Automated publishing on release

One nice features of PID is that it automatically manage the publishing of packages anytime they are released. This is also true for wrappers, anytime they memorize a new/updated version.

This mechanism is also achieved throught CI process, but is automated from end user point of view. So first you need to activate pipelines for my-first-package the same way as for my-framework. No need to add the CI Runner identity as a member of your project because the runners will not have to push to my-first-package repository.

Then if we want to use the framework as a repository for binaries we have to declare it (in root CMakeLists.txt of my-first-package). Simply add the PUBLISH_BINARIES argument to the PID_Publishing command:

PID_Publishing(PROJECT           https://gite.lirmm.fr/passama/my-first-package
               FRAMEWORK         myframework
               DESCRIPTION       my-first-package is used in tutorial.
               PUBLISH_BINARIES
               ALLOWED_PLATFORMS x86_64_linux_stdc++11)

Now reconfigure and rebuild my-first-package to see if everything OK:

cd <my-first-package>
pid build

Then commit and push as usual to the inegration branch:

cd <my-first-package>
git add -A
git commit -m "ready to publish"
git push origin integration
# OR DO: pid integrate

Now let’s test that the release process automates the publishing of package versions:

cd <pid-workspace>
pid release package=my-first-package

If release command succeeded then:

  • in gitlab you should see that the CI process for my-first-package has just been started
  • at the end of this CI process, runners automatically update myframework repository with content coming from my-first-package.
  • This in turn launches the CI process of myframework and so finally generates the static site.

Step 7: Nice naming of all components into a workspace

Most of time, frameworks are used to organized a set of related packages, each one defining its own set of components. Each component is supposed to provide a new set of functionalities or extending framework functionalities. It is very convenient to have a common naming pattern for all those component belonging to the same framework.

For instance, take the well known project boost as example. This project is not xritten in PID but if it was the case it would probably have been defined as a framework with each library corresponding to a package. When we use boost components in a PID project, we just say something like:

PID_Component(my_comp MODULE
              DEPEND boost/boost-filesystem
                     boost/boost-python)

All components names are relative to the same name boost. What is possible in PID only because all components are defined in the same package. We can achieve same behavior in all components were defined in the same unique package what we really don’t want for many reason, the first ones being that such big projects are really hard to maintain, long to compile, etc. We prefer using little packages, that can be regrouped intoa same framework.

For instance if we refer to packages my-first-package and my-second-packages they define their libraries like that:

  • for my-first-package:
PID_Component(hello-shared SHARED DIRECTORY hello
              INTERNAL DEFINITIONS DO_NOT_PRINT_HELLO)
  • for ``my-second-packages`:
PID_Component(hello-user SHARED DIRECTORY hello_user
              DEPEND my-first-package/hello-shared)

Anytime we want to use them in a third party project we must do:

PID_Component(my-app APP DIRECTORY an_app
              DEPEND my-first-package/hello-shared
                     my-second-package/hello-user)

Let’s suppose those two package are member of the same framework myframework as explained previously. What we would like is to have a common naming pattern when using those components and frameworks can be used for that. So instead of refering to package names we can use framework name instead:

PID_Component(my-app APP DIRECTORY an_app
              DEPEND myframework/hello-shared
                     myframework/hello-user)

or with a longer signature:

PID_Component(my-app APP DIRECTORY an_app)
PID_Component_Dependency(COMPONENT my-app DEPEND hello-shared FRAMEWORK myframework)
PID_Component_Dependency(COMPONENT my-app DEPEND hello-user FRAMEWORK myframework)

This makes it a bit more straithforward to understand and use for a framework user.

Also we sometime needs to define a simple and strict naming of components according to a framework predefined pattern. We can use the ALIAS feature of components to give them meaningfull names in the context of the framework namespace.

Now let’s suppose the two components have been defined this way:

  • for my-first-package:
PID_Component(hello-shared SHARED ALIAS base DIRECTORY hello
              INTERNAL DEFINITIONS DO_NOT_PRINT_HELLO)
  • for ``my-second-packages`:
PID_Component(hello-user SHARED ALIAS user DIRECTORY hello_user
              DEPEND my-first-package/hello-shared)

Now we can use those names like this:

PID_Component(my-app APP DIRECTORY an_app
              DEPEND myframework/base
                     myframework/user)

or with a longer signature:

PID_Component(my-app APP DIRECTORY an_app)
PID_Component_Dependency(COMPONENT my-app DEPEND base FRAMEWORK myframework)
PID_Component_Dependency(COMPONENT my-app DEPEND user FRAMEWORK myframework)

From user point of view it’s far simpler and nice looking to handle such names.

That’s it you know more or less everything about frameworks !