Tutorial - Management of runtime resources -

One important but too often forgotten aspect when managing software is how to manage files and folders at runtime. For instance, we often need to find a configuration file in filesystem and basically we either write the path to these files directly into the code, or we need to manage parameters of a program to let the user providing path to the application. Both technics suffers limitation: the first one is not relocatable and requires to modify source code simply to change path according to the workstation filesystem ; the second may be unusable in practice if you have many path to handle.

To overcome these problems, PID is provided with a mechanism to easily manage relocatable runtime resources. By runtime resource we mean any kind of file, or folder, that can be found on a filesystem at runtime (when binary code is executed).

Let’s see how it works with the package my-first-package defined in previous tutorial !

Step 1: Identify and create required runtime resources

Let’s suppose the application hello-app needs to read a specific file at a given moment of its execution. This file is named hello_data.txt.

In every native PID package there is a specific share/resources folder. It is used to contain all runtime resources required by the components of the package. So we simply create the file hello_data.txt in this folder:

cd <my-first-package>/share/resources
echo "important data" | cat > hello_data.txt

That’s it, the file hello_data.txt is now part of the package:

it will be installed in a dedicated place everytime my-first-package is built. The install folder of the package version contains a share/resources subfolder that itself contains hello_data.txt.
it can be updated and commited as any other file of the project so in the end it is easy to release new version of runtime resources together with new version of the code that use them.

For now we just have a file whose lifecycle is bound to the lifecycle of component that will use it, but nothing more so we are still far from being able to write C++ code without boring with filesystem path.

Step 2: Register runtime resource into components

Now we have to configure the application hello-app so that it will be able to find hello_data.txt in any situation. This is achieved by modifying a bit the description of hello-app in apps/CMakeLists.txt.

PID_Component(hello-app APP DIRECTORY hello
              DEPEND hello-static boost/boost-filesystem posix
              RUNTIME_RESOURCES hello_data.txt)

Using RUNTIME_RESOURCES argument of PID_Component allows to specify which runtime resources of the package a component is using at runtime. In the example, hello-app simply uses hello_data.txt. Just remember that all path are expressed relative to the share/resources folder.

For now nothing in the code of hello-app refers to this runtime resource so we do not need to do more. Simply remember that runtime resources of a component are accessible to any component that uses (i.e. depends on) it. So we can imagine to create components that provide runtime resources to other components without using them directly. We typically use this scheme to provide some configuration files with libraries in order for instance to define alternative possible configuration of the library that may be used by a third party component.

Step 3: use the runtime resource mechanism in your code

But for now we keep things simple, and we just want to directly read hello_data.txt content during execution.

3.1 Write the code

First edit the file named hello_main.cpp in apps/hello folder and paste the following code:

#include <hello.h>
#include <iostream>
#include <boost/filesystem.hpp>
#include <pthread.h>
#include <pid/rpath.h>

using namespace std;
using namespace boost::filesystem;

static void* threaded(void* data){
  if (data != NULL){
    path p((char*)data);
    std::string str((char*)data);
    if(! exists(p)){
      str+=" is NOT a valid path !!"
    }
    else{
      str+=" is a valid path !!"
    }
    print_Hello(str);
  }
  return (NULL);
}

int main(int argc, char* argv[]){
    PID_EXE(argv[0]);//optional with recent compilers !!
    if(argc == 1){
	   cout<<"Only one argument ... error !! Please input a string as argument of the program"<<endl;
	   return 1;
    }

    ifstream myfile(PID_PATH("hello_data.txt").c_str());//resolve the runtime path using PID_PATH macro
    string line;
    if (myfile.is_open()){
      while (getline (myfile,line) ){
        pthread_t the_thread;//using the posix API for threads
        if(! pthread_create (& the_thread, NULL, threaded, line.c_str())){
          std::cout << "Error: cannot create thread" << std::endl;
          return (-1);
        }
        pthread_join (the_thread, NULL);
      }
      myfile.close();
    }
    else{
      cout<<"Error !! cannot find resource file hello_data.txt"<<endl;
 	    return 1;
    }
    return 0;
}

Now instead of printing something depending on an explicit input of the user, the code prints “hello” depending on the content of hello_data.txt. Important part in code are:

#include <pid/rpath.h> indicates that the code uses the runtime path resolution API. This API provides some C++ preprocessor macros that are use after.
PID_EXE(argv[0]); is used to configure the runtime resource resolution system. Call to PID_EXE is mandatory only if you work with old GNU GCC or clang versions, or with other compilers than gcc or clang. Otherwise you can omit this call because it will be automatically placed into static section of the excutable (i.e. will be executed before main).
PID_PATH("hello_data.txt") is a call to the resolution algorithm and finally returns a canonical path to the hello_data.txt file, that can in turn be used. In the example the target file is finally read with a c++ ifstream.

3.2 : Description of the package

We see that for using the runtime resources mechanisms we need a specific c++ API. This API if provided by a library called rpathlib that is provided by the package called pid-rpath (for PID runtime path). We need to describe the dependency to pid-rpath in my-first-package since its component hello-app needs to use rpathlib. In root CMakeLists.txt of my-first-package, write:

cmake_minimum_required(VERSION 3.15.7)
set(WORKSPACE_DIR ${CMAKE_SOURCE_DIR}/../.. CACHE PATH "root of the packages workspace directory")
list(APPEND CMAKE_MODULE_PATH ${WORKSPACE_DIR}/cmake) # using generic scripts/modules of the workspace
include(Package_Definition NO_POLICY_SCOPE)

project(my-first-package)

PID_Package(AUTHOR 		    My Name
			      INSTITUTION	  LIRMM
            YEAR 		      2015
            ADDRESS		    git@gite.lirmm.fr:own/my-first-package.git
            LICENSE 	    CeCILL
            DESCRIPTION   TODO: input a short description of package toto utility here
            VERSION       0.2.0
		)

check_PID_Platform(CONFIGURATION posix)
PID_Dependency(boost)
PID_Dependency(pid-rpath VERSION 2.0) #simply adding this line
build_PID_Package()

We simply add the dependency to pid-rpath as explained in previous tutorials.

Then we need to declare the dependency between component hello-app and rpathlib. This is achieved by modifying a bit the description of hello-app in apps/CMakeLists.txt:

PID_Component(hello-app APP DIRECTORY hello
              DEPEND hello-static boost/boost-filesystem pid-rpath/rpathlib posix
              RUNTIME_RESOURCES hello_data.txt)

We simply added the dependency to rpathlib as usual by adding pid-rpath/rpathlib to the DEPEND argument. Now the code is ready to be built.

Step 4: Build and run the code

build the package:

cd <my-first-package>
pid build

At runtime, the resource resolution mechanism automatically finds the file hello_data.txt either in install tree (if the hello-app binary called is in the install tree) or in build tree (if the hello-app binary called is in the build tree):

run from install tree:

cd <pid-workspace>/install/<platform>/my-first-package/0.2.0
./bin/hello-app

run from build tree:

cd <my-first-package>/build
./release/apps/hello-app

Both should produce the same result something like:

Hello important data is NOT a valid path !!

. Now edit hello_data.txt located in <my-first-package>/share/resources:

cd <my-first-package>/share/resources
echo /lib | cat >> hello_data.txt

We simply add a new line with the string /lib, which is a valid path in the filesystem. Now run again both binaries.

output when run from install tree: nothing change

Hello important data is NOT a valid path !!

output when run from build tree: new output

Hello important data is NOT a valid path !!
Hello /lib is a valid path !!

The difference is due to the fact that when run from build tree the resolution mechanism finds the runtime resource in source tree first, while when run from install tree it finds it in the install tree. So difference is because runtime resource is not synchronized between build and install trees. To resolve this situation simply build the package:

cd <my-first-package>
pid build

Now for both binaries the result is the same:

Hello important data is NOT a valid path !!
Hello /lib is a valid path !!

Explanation is simple: the resource file has been reinstalled because it was modified in source tree compared to the version in install tree.

Step 5: Working with references on runtime resource that do not exist

Now we will see that is is also possible to declare runtime resources that do not exist in package filesystem. The main utility is to declare virtual places where content will be added. For instance if your code needs to write to a specific file you may not want this file to preexist (either in source or install tree).

5.1 Declare the “virtual” runtime resource

First declare your runtime resource as usual:

PID_Component(hello-app APP DIRECTORY hello
              DEPEND hello-static boost/boost-filesystem posix
              RUNTIME_RESOURCES hello_data.txt hello_report.txt)

We simply added hello_report.txt file, but we did not create corresponding file in the share/resources folder of the package.

5.2 Write the code

Then now modify a bit the previous code to make it write this file. Edit the file named hello_main.cpp in apps/hello folder and paste the following code:

#include <hello.h>
#include <iostream>
#include <boost/filesystem.hpp>
#include <pthread.h>
#include <pid/rpath.h>

using namespace std;
using namespace boost::filesystem;

static void* threaded(void* data){
  if (data != NULL){
    path p((char*)data);
    std::string str((char*)data);
    if(! exists(p)){
      str+=" is NOT a valid path !!"
    }
    else{
      str+=" is a valid path !!"
    }
    print_Hello(str);
  }
  return (NULL);
}

int main(int argc, char* argv[]){
    PID_EXE(argv[0]);//optional with recent compilers !!
    if(argc == 1){
	   cout<<"Only one argument ... error !! Please input a string as argument of the program"<<endl;
	   return 1;
    }

    ifstream myfile(PID_PATH("hello_data.txt").c_str());//resolve the runtime path using PID_PATH macro

    ofstream myreport(PID_PATH("hello_report.txt").c_str(), std::ofstream::out | std::ofstream::trunc);//ADDING: output file stream to non existing file hello_report.txt

    string line;
    if (myfile.is_open()){
      while (getline (myfile,line) ){
        pthread_t the_thread;//using the posix API for threads
        if(! pthread_create (& the_thread, NULL, threaded, line.c_str())){
          std::cout << "Error: cannot create thread" << std::endl;
          return (-1);
        }
        pthread_join (the_thread, NULL);
        myreport << line << ": OK"<< std::endl;//ADDING: write the report in output file stream
      }
      myfile.close();
      myreport.close();//ADDING: close report output file stream

    }
    else{
      cout<<"Error !! cannot find resource file hello_data.txt"<<endl;
 	    return 1;
    }
    return 0;
}

The code is mostly the same as previously, lined added are marked with comments //ADDED: .... PID_PATH macro is used is used to resolve the path to hello_report.txt.

5.3 Build and run the code

build the code:

cd <my-first-package>
pid build

run the code : the program hello-app should crash on an exception. This situation is normal. Indeed when the runtime mechanism tries to solve PID_PATH("hello_report.txt") it does not find the corresponding resource (normal because we created none) !

5.4 Modify the source code

The resolution mechanism can handle such situations if they are explicitly specified by the source code. This is achieved by using a specific syntax. Edit the code :

Replace the line :

ofstream myreport(PID_PATH("hello_report.txt").c_str(), std::ofstream::out | std::ofstream::trunc);//ADDING: output file stream to non existing file hello_report.txt

By the line:

ofstream myreport(PID_PATH("+hello_report.txt").c_str(), std::ofstream::out | std::ofstream::trunc);//ADDING: output file stream to non existing file hello_report.txt

Then rebuild and run hello-app again: it executes correctly and produces adequate output. The file hello_report.txt now exists:
- if your ran hello-app from build tree then is has been created in <my-first-package>/share/resources. So now it lies in the source tree so you have to take care to avoid publishing it with git ! Simply add the file to your .gitignore rules. This should be done any time a “virtual” resource is defined.
- if your ran hello-app from install tree then is has been created in <pid-workspace/install/<platform>/my-first-package/0.2.0/share/resources.

The only thing we changed between both codes has been to add a + character at the beginning of the path expression. + has a special meaning for the PID_PATH macro: it tells the system to resolve the path even if the finally pointed file does not exist yet. To get a detailed description of path expressions that can be used with PID_PATH macro you can consult this page.

Step 6: Working with folders

This last part of the tutorial now simply explains how to use folder as runtime resources instead of regular files. Indeed most of time it is a good idea to structure runtime resources in folders rather than putting all these files directly into share/resources folder. Furthermore it tends to reduce collision of resources names between packages.

6.1 Declare a folder as a runtime resource

As an example, configurating a program often requires many files so it is preferable to put all this file into one specific folder and then simply reference the folder as a runtime resource. Something like:

PID_Component(hello-app APP DIRECTORY hello
              DEPEND hello-static boost/boost-filesystem posix
              RUNTIME_RESOURCES hello_data.txt hello_report.txt hello_config_files)

We added hello_config_files folder to runtime resources declared by the component.

We need to add the corresponding folder in share/resources folder of the package as well as files in share/resources/hello_config_files:

cd <my-first-package>/share/resources
mkdir hello_config_files
echo "some data" | cat > hello_config_files/data1.txt
echo "some more data" | cat > hello_config_files/data2.txt

6.2 Write and build the code

Then now modify a bit the previous code to make it use this folder. Edit the file named hello_main.cpp in apps/hello folder and paste the following code:

#include <hello.h>
#include <iostream>
#include <boost/filesystem.hpp>
#include <pthread.h>
#include <pid/rpath.h>

using namespace std;
using namespace boost::filesystem;

static void* threaded(void* data){
  if (data != NULL){
    path p((char*)data);
    std::string str((char*)data);
    if(! exists(p)){
      str+=" is NOT a valid path !!"
    }
    else{
      str+=" is a valid path !!"
    }
    print_Hello(str);
  }
  return (NULL);
}

int main(int argc, char* argv[]){
    PID_EXE(argv[0]);//optional with recent compilers !!
    if(argc == 1){
	   cout<<"Only one argument ... error !! Please input a string as argument of the program"<<endl;
	   return 1;
    }
    string line;

    //ADDING: read the files in hello_config_files folder
    ifstream myconfig1(PID_PATH("hello_config_files/data1.txt").c_str());
    ifstream myconfig2(PID_PATH("hello_config_files/data2.txt").c_str());
    if(myconfig1.is_open() and myconfig2.is_open()){
      while (getline (myconfig1,line) ){
        std::cout<<"config1: "<<line<<std::endl;
      }
      while (getline (myconfig2,line) ){
        std::cout<<"config2: "<<line<<std::endl;
      }
    }

    ifstream myfile(PID_PATH("hello_data.txt").c_str());//resolve the runtime path using PID_PATH macro
    ofstream myreport(PID_PATH("hello_report.txt").c_str(), std::ofstream::out | std::ofstream::trunc);


    if (myfile.is_open()){
      while (getline (myfile,line) ){
        pthread_t the_thread;//using the posix API for threads
        if(! pthread_create (& the_thread, NULL, threaded, line.c_str())){
          std::cout << "Error: cannot create thread" << std::endl;
          return (-1);
        }
        pthread_join (the_thread, NULL);
        myreport << line << ": OK"<< std::endl;
      }
      myfile.close();
      myreport.close();

    }
    else{
      cout<<"Error !! cannot find resource file hello_data.txt"<<endl;
 	    return 1;
    }
    return 0;
}

We simply added 2 input file streams to read the two files contained in hello_config_files. Expression will be correctly interpreted by PID_PATH without generating exceptions because:

hello_config_files is referenced as a runtime resource for the component so path to it can be resolved.
files data1.txt and data2.txt exist in this folder.

Simply build again the package and see the result.

6.3 Write the code to allow using non existing files in referenced folders.

It is also possible to use reference folders as containers for files that do not exist at build time. This is even a far better practice that directly writing files in the share/resources folder of a package: let’s suppose your program is generating a big amount of log files, possibly with log files whose name is labelled with a date for instance, then it is a natural way to put all the logs into a specific folder.

modify the description by referencing a new hello_logs folder as a runtime resources:

PID_Component(hello-app APP DIRECTORY hello
              DEPEND hello-static boost/boost-filesystem posix
              RUNTIME_RESOURCES hello_data.txt hello_report.txt hello_config_files hello_logs)

create hello_logs folder in share/resources of the package:

cd <my-first-package>/share/resources
mkdir hello_logs
echo "*" | cat > hello_logs/.gitignore

This folder is supposed to be empty by default because there is no logs that is why we added a .gitignore rule excluding all its content from version control.

modify previous code:

Replace:

//ADDING: read the files in hello_config_files folder
ifstream myconfig1(PID_PATH("hello_config_files/data1.txt").c_str());
ifstream myconfig2(PID_PATH("hello_config_files/data2.txt").c_str());
if(myconfig1.is_open() and myconfig2.is_open()){
  while (getline (myconfig1,line) ){
    std::cout<<"config1: "<<line<<std::endl;
  }
  while (getline (myconfig2,line) ){
    std::cout<<"config2: "<<line<<std::endl;
  }
}

By:

ofstream mylogfile(PID_PATH("+hello_logs/log.txt").c_str(), std::ofstream::out | std::ofstream::trunc);//ADDING: write access to the log file
ifstream myconfig1(PID_PATH("hello_config_files/data1.txt").c_str());
ifstream myconfig2(PID_PATH("hello_config_files/data2.txt").c_str());
if(myconfig1.is_open() and myconfig2.is_open()){
  while (getline (myconfig1,line) ){
    mylogfile<<"config1: "<<line<<std::endl;
  }
  while (getline (myconfig2,line) ){
    mylogfile<<"config2: "<<line<<std::endl;
  }
}

Build and run the code:

Now you can see that a file log.txt with updated content has been added to the hello_logs folder.

6.4 Test the resolution without using write access specifier (i.e. `+`)

Remove the file share/resources/hello_logs/logs.txt (in source or install tree dependencing on from where you ran the program).
In the code: