Tutorials for frameworks
A framework is a specific deployment unit used to group a set of packages that share common topics or mechanisms. For instance there is a framework called pid
, that groups a set of native packages and external wrappers implementing runtime mechanisms of PID (e.g. the pid-rpath
package) or general utilities for software developpers (e.g. wrappers for boost
and yaml-cpp
, package pid-os-utilities
). The result of building a framework is a static site that provides a centralized documentation on these packages. Frameworks have been primarily be designed to work together a continuous integration (CI) process in a Gitlab environment.
Contributing to an existing framework
This first tutorial explains how a package or wrapper can contribute to a framework. To keep it simple we want a new package <newpack>
to be able to contribute to the pid
framework.
Step 1: edit a package that contributes to a framework
Let’s make the package my-first-package
, created in introduction tutorials, contributing to the framework pid
. We suppose that this package is already in your local workspace (or you can simply create a new one). In the root CMakeLists.txt of my-first-package
:
We added two lines to the previous content of CMakeLists.txt:
-
PID_Publishing
(or equivalentdeclare_PID_Publishing
) tells that the package produces a static site. TheFRAMEWORK
argument specifies that it contributes to thepid
framework generated static site.PROJECT
argument gives the address of the package online project site andDESCRIPTION
provides a long description of package utility. Finally theALLOWED_PLATFORMS
argument tells for which platforms the publishing will be performed, for now only one platform can be specified due to CI process limitations. -
PID_Category
(or equivalentadd_PID_Package_Category
) tells to which categories the package belongs to. If such categories exist in the framework definitions then the framework static site classifies the package is given categories, in left side panel of the resulting static site. Look atpid-rpath
package in the left side panel, it belongs to the categoryprogramming/resources management
that is also defined by the framework so it has been put in the corresponding section.
The same logic applies for wrappers by using calls to PID_Wrapper_Category
(or equivalent add_PID_Wrapper_Category
) and PID_Wrapper_Publishing
(or equivalent declare_PID_Wrapper_Publishing
) which makes it possible to publish wrappers more or less the same way as native packages.
Step 2: make a package contributing to a framework
Now the package will really contribute to pid
framework:
The site
command simply publishes the static site of the package. In the present case it published it inside the framework. The argument synchro=false
is used to avoid to update of the framework git repository (which anyway will be impossible because you have no write access to this project).
To have a look at the final result:
The framework being a deployment unit is has its own git repository, CMake description and related commands. The serve
command is used to generate local web server that hosts the generated static site of the framework. This way you can see the result locally anytime you modifiy the content of a framework.
The output gives something like:
As explained simply do a ctrl-c to stop the web server. To see the result use a web browser to open the address http://127.0.0.1:4000/pid-framework.
The package my-first-package
should appear only in the section All Packages of the left side panel because the category used programming/tutorial
is not defined in pid
framework. If you click on the link to my-first-package
in left side panel, the center panel now shows the documentation of the package.
Step 3: add developper info to the static site
Warning: keep the serve
command running in another terminal !
For now the package related pages do not give access to API doc or other developper info. To do this you have to modify a bit the package description:
The PUBLISH_DEVELOPMENT_INFO
simply tells PID that all available development info, like doxygen APID documentation website, will be published and referenced by the framework.
Remmebr to set your personnal info in PROJECT
argumument.
Then you have to rebuild your package with adequate options activated and then republish.
After this last command the site should have been updated, so simply reload the static site in the web browser to see the result. In the my-first-package
panel, the top bar should now provide API and Static checks from Developpers menu. By clicking on API
item a new window open the API doc doxygen generated site (since BUILD_API_DOC
package option has been activated).
Step 4: publish binary archives using the framework
Frameworks can also be used as respositories for binary archives of native and external packages. The main benefit of binary archives is that they really speed up the deployment process by avoiding unecessary compilation.
To do so, modify a bit the CMakeLists.txt of the package:
The PUBLISH_BINARIES
option simply tells PID that if binary archive have been generated for the package they will be put and referenced into the framework static site.
Then you have to rebuild your package with adequate options activated and then republish.
After this last command the site should have been updated, so simply reload the static site in the web browser to see the result. In the my-first-package
panel, the item Binaries of the top bar menu Developpers item now prints a page that lists all available binaries for the package. Binary archives are classififed according to version first and target platforms. You should see an archive for the version/platform you are currently building your package with (since GENERATE_INSTALLER
option is active binary archives have been generated by build process and site command has published them into framework).
Automatic contribution to a framework
Anytime a package contributes to a framework using the PID_Publishing
command, a CI file is generated (.gitlab-ci.yml
). This file is used to configure the continuous integration process of the package: anytime the package is released its contribution to the framework will be automatically updated. So for instance if the package publishes binary archives using the PUBLISH_BINARIES
option, then corresponding archives for the version released will be generated by the build process taking place in the CI process and then put into the framework.
Create a new framework
Now we now how a package contributes to a framework we have to see how to define new frameworks. A framework is a deployment unit in PID so there is a dedicated CMake API to describe frameworks content and also a specific development process for them.
Step 1: create the framework project
Let’s suppose we want to define a framework called myframework
, we first need to create the repository. In gitlab, create a new project called myframework in your gitlab workspace. Then we need to create the framework in your local workspace and connect it to this repository:
This command create the framework repository of myframework
in local workspace in <pid-workspace>/sites/frameworks
. The framework branching model is rather simple: there is only one master branch and only one origin remote.
Then look at the generated file <pid-workspace>/sites/frameworks/myframework/CMakeLists.txt
, it should look something like:
The logic is more or less the same as for packages, but this time we use the frameworks API (see include(Framework_Definition NO_POLICY_SCOPE)
) to declare a framework using the PID_Framework
command. Arguments have xactly the same logic as for packages. The SITE
argument is used to specify the address of the resulting website. First set adequate author name/institution as well as year, description and license information.
The folder <pid-workspace>/sites/frameworks/myframework
is organized according to a predefined scheme:
- the
share
folder contains miscellaneous elements, like CI scripts -
the
src
folder contains the whole static site description formyframework
:_external
folder will contain all information about external packages that contribute to the framework._packages
folder will contain all information about native packages that contribute to the framework.assets
folder contains global data that will be used “as is” and will not be interpreted by the jekyll generation process.-
pages
folder contains global data that will be interpreted by the jekyll generation process. It typically contains markdown files that themselves contain the general information about the framework. For instance, the page you are currently reading as been generated from a markdown file contained in thepid
framework. This folder contains some default pages:introduction.md
that generates an introduction page for the frameworkinstall.md
that describes the installation procedure for the framework.tutorial.md
that gives access to some tutorials.more.md
that provides access to advanced topics and detailed information.
Each of these markdown files corresponds to an entry point of the global framework documentation. More precisely in the finally generated static site, each of these files will be accessible from an item of the Documentation
menu of top menu bar. As a first example you can simply look in the current web page top menu bar, because this static site has been generated from the pid
framework.
Step 2: writing framework documentation content
Each of these files can contain any number of reference to any number of other pages or sections. The pages folder can as well contain any number of files, either mardown or html files, you can also add other kind of elements like images and of course subfolders. This way you can structure your documentation’s filesystem more or less the way you want.
When writing a web page file (html or markdown) you have to use some specific jekyll headers. For instance for now the introduction.md file should be like this:
The lines between ---
is the jekyll header:
- the
layout
attribute must bepage
. - the
title
of default generated markdown files must be let unchanged. - any title of user specific files can have any title.
Headers are use by the jekyll generation process to input the content of the file into the final web site. All pages of the framework can also contain Liquid tags, that are interpreted by jekyll to generate specific actions. The most usefull tag is highlight
:
The text you are currently reading has itself been generated from a markdown page and the beginning of this tutorial is defined by the following text:
The previous content is enough to start working on a simple framework description. First let’s consider <framework>
as the path to myframework
folder in your workstation’s filesystem.
1) Edit the file
2) Regenerate the static site:
3) Print the result
In another terminal launch the serve
command
That’s it, look at the address provided by the serve
command output with your web browser and click on Introduction
item of the top menu Documentation
to see the result. You simply have to reproduce this process to populate your framework documentation.
Step 3: Adding a banner and a logo
You can also improve a bit the look of the resulting static site by specifying images such as a logo and a banner.
- the logo image is used as an icon at left of the top menu bar. The icon has an hyperlink that points to the welcome page. Save the image of the PID icon as
/src/assets/icon.jpg (right click on the icon then "save as ..."). - the banner is a bigger image that is put at the top of the welcome page. GO to the welcome page and save the image of the PID banner as
/src/assets/banner.jpg.
Modify a bit the CMakeLists.txt:
The arguments LOGO
and BANNER
are used to set the path to the corresponding images. All path are expressed relative to the assets folder.
Then regenerate the static site:
And see the result by reloading the resulting static site in your web browser. The welcome page and the top menu bar of myframework
should look like the ones of pid
.
Step 4: structuring the contribution process with categories
Except global documentation all the contributions will be generated by packages and wrappers that belongs to this framework (anytime their site
command is executed). These units will finally fill the folders _packages
and _external
respectively.
Categories can then be used to structure to present contributing packages in a more structured way in the left side panel. The goal of this panel is to have a quick overview of available packages and wrappers and thanks to their categorization, to have an indication about their utility. For instance if you look at the left side panel of the current website, you can quickly understand that the package pid-rpath
provides some programming utilities to manage runtime resources. To generate such a classification in left side panel there are 2 things to do:
- Define a category
test
inmyframework
. Modify again a bit the CMakeLists.txt:
- Define the corresponding
test
category in the contributing package./ Let’s suppose we reuse the packagemy-first-package
defined in tutorial on how to contribute to a framework:
In the previous code we changed the framework of my-first-package
to be now myframework
. Furthermore the package now belong to the test
category (call to PID_Category(test)
) that has been defined in myframework
.
Now to contribute to the framework just do:
And see the result after reloading your static site in your web browser. You should now see that your package is referenced in left side panel under the test
category.
Step 5: Publishing the framework
5.1 Providing online addresses
Until now all we did is to generate the framework site locally. Obviously what we trully want is to publih it online so that other people can read your doc ! This is achieved while using the default CI process of frameworks, that consists in generating pages of the web site.
One important thing to do is to specify the address of the finally generated static site. This requires to set the SITE
argument of PID_Framework
command. You should also give all the necessary meta information : the description of the framework and contact author information are generally important (but not so mush in this tutorial of course):
So use the previous pattern and simply changes values for author, description, static site and online project addresses, etc. Let’s remember that CI process is based on gitlab-CI so naming convention of gitlab applies everywhere you need to set an address:
- Gitlab project pages (in
PROJECT
) generate a path with patternhttps://<gitlab server address>/<namespace>/<project>
. - Gitlab pages generates by default static site in a path with pattern:
http://<namespace>.<gitlab pages server address>/<project>
. SeeSITE
argument for an example.
Now simply regenerate the static site:
And see the result ! Everything should more or less look the same as previously.
5.2 Configuring the CI in your framework project
For the gitlab pages to work, you need to activate the CI process in your gitlab project.
Go to your project’s settings and allow for pipelines to run (in “Permissions” section). This way the CI process can be automatically launched whenever you push to the framework repository.
Another important configuration is to add the runner as a member of the framework. Indeed the CI runners will update the framework anytime a package or wrapper contributes to the framework (new doc, new binaries) and so thoses runners must have rights to push to framework packages to do this update. The administrator of your PID environment should have configured a specific CI user in your gitlab server that is the identity used by CI runners whenever they need to push to a specific repository. So:
- ask your administrator the name of this specific CI user.
- add it as a member of
myframework
. Its role must give it permission to push to protected branches, in Gitlab 11.7 the role is calledMaintainer
.
Once done your are now ready to publish your framework !
5.3 Publishing the framework
As already said, the publishing process is automatically managed by the CI process, so everything you have to do is to commit and push:
The CI process takes place. When finished the static site has been generated and can be accessed at http://passama.lirmm.net/myframework (change with your own address).
Step 6: Publishing packages and wrappers
Now that the CI process is in place at framework level, the publishing of packages and wrappers can be automated.
6.1 Automatic update of framework static site
First update the content of myframework
with content coming from my-first-package
:
This last command simply updates the content of myframework
with pages and binaries related to my-first-package
and automatically push the framework repository (because the argument synchro=false
has been removed).
You can notice that the CI process has been automatically launched and generates the static website.
6.2 Automated publishing on release
One nice features of PID is that it automatically manage the publishing of packages anytime they are released. This is also true for wrappers, anytime they memorize a new/updated version.
This mechanism is also achieved throught CI process, but is automated from end user point of view. So first you need to activate pipelines for my-first-package
the same way as for my-framework
. No need to add the CI Runner identity as a member of your project because the runners will not have to push to my-first-package
repository.
Then if we want to use the framework as a repository for binaries we have to declare it (in root CMakeLists.txt
of my-first-package
). Simply add the PUBLISH_BINARIES
argument to the PID_Publishing
command:
Now reconfigure and rebuild my-first-package
to see if everything OK:
Then commit and push as usual to the inegration branch:
Now let’s test that the release process automates the publishing of package versions:
If release command succeeded then:
- in gitlab you should see that the CI process for
my-first-package
has just been started - at the end of this CI process, runners automatically update
myframework
repository with content coming frommy-first-package
. - This in turn launches the CI process of
myframework
and so finally generates the static site.
Step 7: Nice naming of all components into a workspace
Most of time, frameworks are used to organized a set of related packages, each one defining its own set of components. Each component is supposed to provide a new set of functionalities or extending framework functionalities. It is very convenient to have a common naming pattern for all those component belonging to the same framework.
For instance, take the well known project boost
as example. This project is not xritten in PID but if it was the case it would probably have been defined as a framework with each library corresponding to a package. When we use boost components in a PID project, we just say something like:
All components names are relative to the same name boost
. What is possible in PID only because all components are defined in the same package. We can achieve same behavior in all components were defined in the same unique package what we really don’t want for many reason, the first ones being that such big projects are really hard to maintain, long to compile, etc. We prefer using little packages, that can be regrouped intoa same framework.
For instance if we refer to packages my-first-package
and my-second-packages
they define their libraries like that:
- for
my-first-package
:
- for
my-second-packages
:
Anytime we want to use them in a third party project we must do:
Let’s suppose those two package are member of the same framework myframework
as explained previously. What we would like is to have a common naming pattern when using those components and frameworks can be used for that. So instead of refering to package names we can use framework name instead:
or with a longer signature:
This makes it a bit more straithforward to understand and use for a framework user.
Also we sometime needs to define a simple and strict naming of components according to a framework predefined pattern. We can use the ALIAS
feature of components to give them meaningfull names in the context of the framework namespace.
Now let’s suppose the two components have been defined this way:
- for
my-first-package
:
- for
my-second-packages
:
Now we can use those names like this:
or with a longer signature:
From user point of view it’s far simpler and nice looking to handle such names.
That’s it you know more or less everything about frameworks !