When we run a program inside the OS, a process is asked to handle the task. If our code isn't designed in a concurrent way, then the process uses only one thread to run the main function. This makes the program to perform its actions sequentially, but we can take advantage of threads to perform more than one thing at a time if needed.

Modern microprocessors are built with multiple processors (cores). To achieve programming concurrency we can face two scenarios:

• Multiple threads running inside one process.
• Multiple processes running at the same time.

Concurrent programming defines an environment where created tasks can be performed at the same time, but it doesn't mean that all tasks are going to be executed in parallel.

A process consists in a running program plus the resources that allow the program's execution. Processes can have multiple threads running inside them.

We can check running processes inside *nix using commands like ps, pstree or top.

In this article, we are going to focus in the first case scenario: multiple threads running inside one process.

What is a thread?

A thread is a separate dynamic set of code executions or instructions that run alongside the main process in a program, and it can be scheduled.

Threads give us concurrency without isolation, working in the same process and sharing memory space, which makes the ability for threads to intercommunicate.

Creating a thread is cheaper than creating a process, and ending a thread is faster than ending a process.

Until now, all the examples shown in previous articles have been made using serial or sequential computation. That's not wrong, but we were using only one thread in one process to achieve our functionality.

Sequential commands run like this:

start -> job_a -> job_b -> job_c -> ... -> end

which in code is as we usually call functions in main:

int main() {
    job_a();
    job_b();
    job_c();
    ...
    
    return 0;
}

A thread set is executed like this:

       -> job_a ->
      /           \
start ->  job_b   -> end
      \           /
       -> job_c ->

which in pseudo code would look like this:

int main() {
    createThread(job_a());
    createThread(job_b());
    createThread(job_c());
    
    ...
    
    join_thread(job_a());
    join_thread(job_b());
    join_thread(job_c());
    
    return 0;
}

In order for a program to take advantage of threads, it needs to be able to be organized into discrete, independent tasks which can execute concurrently.

|-- job_a --| |-- job_b --| ... |-- job_n --|

Considering our sequential code from above, we can check three situations to check if threading is possible in our program:

• Jobs or routines can be interchanged and result is not modified.

|-- job_b --| |-- job_a --| ... |-- job_n --|

• Jobs or routines can be interleaved and result is not modified.

|- rA -| |- rB -| |- rA -| |- rB -| |- rA -| |- rN -|

• Jobs or routines can be overlapped and result is not modified.

|-- job_a --|        |-- job_n --|
        |-- job_b --|

We can take a look at the internal workflow inside an IDE (Integrated Development Environment). An IDE usually contains various spaces inside a workspace.

When we launch the program, a process is created by the operating system. That process contains the required threads for the IDE to run the multiple operations it needs, like the integrated terminal emulator, the file explorer, the text editor, or the syntax checker.

We can implement threading in our program as a matter of trial and error, or to specific task only at the beginning, incrementing the number of tasks and threads as the program evolve and grow up the thread model. If we want to start from a proven ground, the POSIX threads standard offers some existing models for threaded programs, which are not designed for any specific application kind, but are worth knowing, like:

— Pipeline model

The pipeline model takes a long input stream and process each of the inputs through a series of stages or sub-operations. Each stage can handle a different unit of input at a time.

input -> thread_a -> thread_b -> thread_n -> output

The overall throughput of a pipeline is limited by the thread that processes its slowest stage, meaning threads that follow it in the pipeline are stopped until it has finished. In this type of threading model is good to design the program in a way where all stages take about the same amount of time to finish.

The standard Graphics Pipeline uses this threading model.

— Thread pool

In this model, one thread is in charge of work assignments for the other threads. The thread in charge deals with requests and communications that arrive in an asynchronous way, while the other threads perform how to handle the requests and process the data.

This model is also known as the manager-worker model.

input_a ->              -> worker_a
          \            /
input_b    -> manager  ->  worker_b
          /            \
input_c ->              -> worker_c

This model fits well in database servers, or desktop related tasks like window managing.

— Peer model

In this model, a thread must create all the other peer threads when the program starts but after that, all threads work concurrently on their tasks without a specific leader. This makes each thread responsible for its own input.

       -> thread_a ->
      /              \
input ->  thread_b   -> output
      \              /
       -> thread_c ->

Given the lack of a manager thread, peers need to synchronize their access to common input sources.

Implement threading in C

We can work with threads in C, however there isn't any built-in solution for this. Inside unix-like machines we have a set of POSIX types and calls wrapped in a header named pthread.h that let us access threading functions in C. So before we even start, we need to add the header to our code.

#include <pthread.h>

Let's create a first threads' boilerplate. It's easier than you may expect.

In short, we need a function we want to execute in parallel to our main() one, then we need to create a thread, assign the desired function to it, ensure that we are executing it, and terminate the thread once we're done.

Create a function to execute an entry point

The standard prototype for a function that is going to be passed to a thread follows the scheme void *function_name(void *arg)

void *thread_job() { 
    printf("We are in a new thread\n"); 
    return NULL;
}

Create a thread

pthread_t thread;
pthread_create(&thread, NULL, function_to_execute, &value_to_pass);

We need to pass the following parameters to the thread creation:

• The ID from the created thread.
• The attributes we want to use to create the thread. Pass NULL if you don't need any special ones, so defaults are applied.
• A pointer to the function to execute by the thread.
• A pointer to the thread argument.

This returns 0 if thread creation is successful and nonzero if not.

It's a good practice to avoid code errors checking the returning value of the thread creation function.

pthread_create(&thread, NULL, function_to_execute, &value_to_pass) != 0 ? printf("Failed to create Thread\n") : printf("Thread created!\n");

— Note that we are creating threads from the main() function of the program, but we can create them from inside actual threads too.

            -> thread_c ->
           /              \
thread_a ->    thread_b    -> thread_d ...

Once a thread is created, it has a life cycle that consists in four states:

• Ready state, meaning the thread is waiting for a processor, and able to run.
• Running state, when the thread is currently executing.
• Blocked state, meaning the thread is waiting for a synchronization mechanism or an I/O operation to complete.
• Terminated state, once the thread is done or cancelled.

blocked --> ready <---> running --> terminated
   |                       |
   └----------<------------┘

Ensure thread execution

We can use pthread_join() as a thread synchronization call to ensure that our main thread waits until the second thread finishes:

pthread_join(thread, NULL);

Note we are passing NULL as the second argument. We'll use this second argument in a few lines below to return data from our thread.

Terminate a thread

Threads normally terminate once they done their inside work correctly. However there are more options to terminate a thread.

• We can explicitly tell a thread to terminate using pthread_exit():

pthread_exit(NULL);

• We can specify which thread to terminate using pthread_cancel():

pthread_cancel(thread);

— After following the steps, our code should look like this:

#include <stdio.h>
#include <pthread.h>

int error_close() {
    printf("Failed to create Thread\n");
    return 1;
}

void *thread_job() { 
    printf("We are in a new thread\n"); 

    pthread_exit(NULL); /* optional, but recommended */
    return NULL;        /* optional, but recommended */
}

int main() {
    pthread_t thread;
    pthread_create(&thread, NULL, function_to_execute, &value_to_pass) != 0 ? error_close : printf("Thread created!\n");
    
    printf("We are inside Main()\n");
    
    pthread_join(thread, NULL);
    
    pthread_exit(NULL);  /* optional, but recommended */
    
    return 0;
}

Tell the compiler to use pthread lib

To compile our program using threads we need to link it along with the POSIX thread library. Adding the -pthread flag to the compiler should work.

$ gcc -pthread -o test_threads main.c
$ ./test_threads

Sharing data between threads

Threads can communicate each other, but they need fast communication methods. Most thread communication involves using memory, since all threads created by the program live in the same process and share the same memory space.

We have three types of memory to work with (Refer here to read about managing memory in C) and to place data to be shared between threads.

— Global memory

If we know that we are only going to have an instance of an object inside our multi threaded program like a mutex, which we don't want to be inside individual threads.

— Stack memory

Storing data in this memory location is recommended for thread routines since its lifetime is the same of the routine execution.

— Dynamic memory

Storing data dynamically requires some memory management routine like malloc(). Data stored in this type of memory has a lifetime scoped between memory allocation and memory deallocation.

This is usually recommended to manage persistent context, since it's independent from all program's threads.

We can find the following shared data between threads in a process:

• Memory space.
• Global variables.
• Opened files.
• Children processes.
• Timers.
• Semaphores and signals.

Threads also have private data. Variables declared within the thread function are local to the thread.

Other private data from a thread is:

• Thread ID.
• Registers.
• Thread status.
• Thread context when it's not executing.

— A thread doesn't keep track of the other created threads, nor does it know the thread that created it. As part of the POSIX thread header functions, we can take advantage of pthread_self() to get the running thread's id.

Inside the thread_job() function we can add the following lines:

void *thread_job() { 
    printf("We are in a new thread with ID: %ld\n", pthread_self()); 
    pthread_exit();
}

Since pthread_self() returns the thread handle of the calling thread, we can use it in combination to pthread_equal() to identify a thread when entering a routine.

Passing arguments to threads

Thread functions take a void pointer as an argument, and return a void pointer as result. Since this is generic data, it leaves us almost total freedom to operate with our data.

Let's modify our actual code. We are going to define a thread count number, and we are going to create as much threads as the defined value has.

We are going to print which thread are we in when running the thread_job() function. To know which one is the working one, we are passing the thread counter as the argument value.

#include <stdio.h>
#include <pthread.h>

#define THREAD_COUNT 10

void *thread_job(void *value) { 
    long t_num;
    t_num = (long)value;    
    
    printf("Thread %ld with ID %ld is working...\n", t_num, pthread_self());
    
    /* sleep acts as a dummy, simulating some work being made */
    sleep(2);

    printf("Thread %ld with ID %ld is done!\n", t_num, pthread_self());
    
    pthread_exit(NULL);
    return NULL;
}

int main() {
    pthread_t *threads = (pthread_t*)malloc(sizeof(pthread_t));
    long i;
    
    for(i = 0; i < THREAD_COUNT; i++){
        printf("We are inside Main()\n");
        
        if(pthread_create(&threads[i], NULL, thread_job, (void *)i) != 0){
            printf("error creating thread[%ld]", i);
            return 1;
        }
    }

    for(i = 0; i < THREAD_COUNT; i++) {
        pthread_join(thread[i], NULL);
    }
    
    return 0;
}

This is nice but in real life we'll probably need to pass more than one argument to our thread on creation. We can collect all the data we need to pass to a thread inside a struct type.

typedef struct tdata {
    int      amount;
    char     *account_name;
    e_action action;
}tdata_t;

When we pass a struct into the thread job function, we can access its data by simply casting the type of the struct:

void *thread_job(void *data) { 
    tdata_t received_data;
    
    received_data.amount = ((tdata_t*)data)->amount;
    received_data.account_name = ((tdata_t*)data)->account_name;
    received_data.action = ((tdata_t*)data)->action;

    
    printf("Thread job. Account name is: %s\n", received_data.account_name);

    /* free data struct before leaving if not needed anymore */
    free(data);
    return NULL;
}

Returning values from threads

Sometimes we may need our thread to make some operations and return something from it.

We can return almost anything since thread functions are type of void pointer. The important point here is to allocate memory to the local value we want to return. Otherwise it will cause a segmentation fault since it's going to be on the stack memory of the function.

Allocate some memory in the thread function.

void *thread_job(void *value) { 

    /* allocate some memory for our desired return value */ 
    int *t_int = (int *)malloc(sizeof(int));
    
    for(int i = 0; i < (int)value; i++)
        (*t_int)++;

    /* return the value */
    return t_int;
}

Inside our external function that controls the thread creation and execution, we can create a variable to hold what is returned from the thread job.

int *ext_result;

Using pthread_join() we can get the return value from the function using the second argument of the function:

pthread_join(entry_point, (void*)&ext_result);

Now we can use the returned value in the rest of our program.

Following the good practice of freeing memory up when we are done using it. Inside the external function we have to free *ext_result after using it (since we cannot do it inside the thread job function, and both variables point at the same memory address).

free(ext_result);

Explicit synchronization

In concurrent programs is not possible to determine what is going to happen when we execute it just by looking at it. Threads run concurrently and the execution order depends on the scheduler, but we can manage to intentionally make a thread wait for another one to finish.

If more than one thread is asked to access or write a memory location we can run into a situation known as race condition.

 race condition between two threads accessing and writing the same memory
   
   thread_a      memory      thread_b           threads' steps
     
              |00|0A|0B|0C|
             /             \
|00|0A|0B|0C|               |00|0A|0B|0C|    1. read the value
     |                           |
   |08|                        |0E|          2. modify the value
     |                           |
|00|08|0B|0C|               |00|0E|0B|0C|    3. write the value
             \             /
              |00|0E|0B|0C|
              
          this time thread_b wins

Avoiding these situations can be achieved via mechanisms that manage read/write locks and barriers such as mutexes or semaphores.

— If we use threads to run completely independent functions that have no correlation from each other, synchronization isn't a problem, and we would choose to skip this process.

MUTEX

A mutex is the basic pthread synchronization mechanism. Its name stands for mutual exclusion lock. It's useful to solve unpredictable race conditions by serializing the execution of threads.

If a thread succeeds calling a mutex lock, it will block the other threads to execute the code below until the owner thread unlocks the mutex.

The pthreads API provides mutex functions and operations to work with.

In order to create a mutex we need to declare a pthread_mutex_t. We can do it in an static or a dynamic way:

• Static, declaring it outside any function:

/* just the mutex */
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;


/* A mutex inside a struct, holding protected data */
typedef struct m_data {
    pthread_mutex_t mutex;
    int             value;
} m_data_t;

m_data_t data = {PTHREAD_MUTEX_INITIALIZER, 0};

• Dynamic, declaring it when we allocate memory to it:

/* A mutex inside a struct, holding protected data */
typedef struct m_data {
    pthread_mutex_t mutex;
    int             value;
} m_data_t;

...

foo(){
    m_data_t *data;
    data = (m_data_t*)malloc(sizeof(data_t));
    pthread_mutex_init(&data->mutex, NULL);
    ...
}

Remember to initialize the mutex before creating any threads.

Once its initialized, we can lock it and unlock it using the following functions:

pthread_mutex_lock(&mutex); 

/* code to execute in between */

pthread_mutex_unlock(&mutex);

If a thread calls the mutex lock, the code between the lock function call and the unlock function call can only be accessed by a single thread until the mutex is unlocked.

This kills parallelism, but allow to make responsiveness in places like user interfaces. We can have a thread doing the I/O and the rest calculating whatever needed in the back.

The example below calculates the first 21 Fibonacci numbers using a separate thread for each one. Try commenting out the mutex lock and run several times the program. Different results in the numbering order may occur.

#include <stdio.h>
#include <pthread.h>

#define THREAD_COUNT 21

int result;
pthread_mutex_t result_mutex = PTHREAD_MUTEX_INITIALIZER;

int calc_fibonacci (long num) {
    if (num <= 1) {
        return 1;
    }
    return calc_fibonacci(num -1) + calc_fibonacci(num -2);
}

void *thread_job(void *value) { 
    pthread_mutex_lock(&result_mutex); 
    
    result = calc_fibonacci((long)value);
    
    pthread_mutex_unlock(&result_mutex);
    
    printf("We are in thread num %ld, and result is %d\n", (long)value, result);
    
    sleep(1);
    return NULL;
}


int main() {
    pthread_t thread;
    long i;
    
    for(i = 0; i < THREAD_COUNT; i++){
        
        if(pthread_create(&threads[i], NULL, thread_job, (void *)i) != 0){
            printf("error creating thread[%ld]", i);
            return 1;
        }
    }

    for(i = 0; i < THREAD_COUNT; i++){
    
        pthread_join(thread[i], NULL);
    
        pthread_exit(NULL);
    }
    
    return 0;
}

When implementing mutexes, we need to take care of a few factors:

• Waiting threads are not good for performance. It's a good practice to apply several small mutexes to unrelated code executions rather than using a single mutex that locks them all at once.

If the data to lock is independent, is a good idea to use separate mutexes. Complications face up when data isn't independent at all.

• It takes time to lock and unlock mutexes. This means performance issues, so the first factor should be guided by the common sense of mutexing only critical parts.

CONDITION VARIABLES

Condition variables are a signal mechanism associated with mutexes and their protected shared data. They control threads' access to data, and let threads synchronize between them based on the value of the data.

We can think about condition variables as a notification system among threads.

To create a condition variable, the process is fairly familiar:

pthread_cond_t;

/* using an initializer macro */
condition_var = PTHREAD_COND_INITIALIZER;

/* or using the function call */
int pthread_cond_init(&condition_var, NULL);

Once a condition variable has been initialized, we can use it with a thread in the following two ways:

• Make the thread wait on the condition variable.

pthread_cond_wait(&condition_var, &mutex);

/* or specifying a timeout with */

pthread_cond_timedwait();

Calling any of the waiting functions require to pass a locked mutex next to the condition variable.

• Make the thread signal other threads waiting on the condition variable.

/* signal only one of the waiting threads */
pthread_cond_signal(&condition_var); 

/* singal all the waiting threads */
pthread_cond_broadcast(&condition_var);

Both functions make the thread calling them to hold the mutex. The mutex must be unlocked after the call.

SEMAPHORE

A semaphore is a synchronization mechanism made from an unsigned int whose changes can't be interrupted. It's stored in a memory location accessible by all the processes that need to synchronize their operations.

Semaphores' header is separated from pthreads. In order to implement semaphores in our project, the header semaphore.h is required.

The main difference with a mutex, is that semaphores don't have a concept of ownership. While we cannot use a thread to lock a mutex and another one to unlock it, since the mutex expect the same thread to unlock it, it's possible to do the same using semaphores.

In most case scenarios, using mutexes and condition variables is more than enough to solve synchronization problems.

— In order to have a semaphore inside our code we need to declare it and start it:

sem_t *semaphore;

sem_init(&semaphore, 0, N);

We can work with semaphores using two operations:

• WAIT operation which will try to decrease the semaphore value if its value is greater than zero. If not, it'll wait.

sem_wait();

• SIGNAL operation which will increment the value of the semaphore, and return.

sem_post();

As most of the data structures in C, we need to create it before using it, and destroy it after using it so we avoid garbage.

A complete overview on how to implement a semaphore could look like this:

#include <semaphore.h>

#define N 6 /* can be any positive value */

/* create a semaphore */
sem_t semaphore;

/* initialize semaphore */
sem_init(&semaphore, 0, N); 

/* allocate a resource */
sem_wait(&semaphore);

...

/* return semaphore to pool */
sem_post(&semaphore);

...

sem_destroy(&semaphore);

return 0;

— We can use a semaphore in a similar way to a mutex by using a binary semaphore (define N as 1), to protect critical parts of the code from race conditions.

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <semaphore.h>

#define THREADS 4
sem_t semaphore;
int counter = 0;

void* thread_job(void* args) {
    printf("Hi from thread %d\n", *(int*)args);
    
	sem_wait(&semaphore);
	counter++;
    printf("Counter value is: %d\n",counter);
	sem_post(&semaphore);
	
	free(args);
}

int main(void) {
	
	pthread_t *threads = malloc(sizeof(pthread_t) * THREADS);
	
	sem_init(&semaphore, 0, 1); //we can change 1 to other value and have more threads running at a time
	int i;
	
	for(i = 0; i < THREADS; i++) {
		int *a = malloc(sizeof(int));
		*a = i;
		if(pthread_create(&thread[i], NULL, &thread_job, a) !=0){
			printf("cannot create thread.\n");
		}
	}
	
	for(i = 0; i < THREADS; i++) {
		if(pthread_join(thread[i], NULL) !=0){
			printf("cannot join thread.\n");
		}
	}
	
	sem_destroy(&semaphore);
	return 0;
}

A working example

In the previous article we worked on a fictitious weather forecast program to explain how to save files. Let's grow our program a bit.

The single thread program

In a serialized way, if we'd want the user to make interaction with the program, we can think of three main functions to implement:

• Add data to the program.
• Return data from the program.
• Generate new data from existing data.
• Exit the program when done, or requested.

This can be translated into code this way:

typedef enum {
    EXIT = 0,
    WRITE,
    READ,
    OPERATE,
} e_action;

And so, our main function can deal with the type of action, one at a time:

/* simple error message handling */
int handle_error(char* msg) {
    printf("%s\n", msg);
    return 1;
}

int main(int argc, char *argv[]) {
    
    /* Get the desired action (this time from argv[1]) */
    int action;
    if(argc > 1)
        action = atoi(argv[1]);
    else
        action = -1;

    
    switch(action) {
        case WRITE:
            func_write();
            break;
        
        case READ:
            func_read();
            break;
        
        case OPERATE:
            func_operate();
            break;
        
        case EXIT:
            func_end_program();
            break;
            
        default:
            handle_error("No action passed to argv[1]");
            break;
    }

    return 0;
}

If the program is going to be used from a single terminal by a single user, there is no much complication, but let's scale our fictitious program a bit.

Let's take in consideration that weather's forecast data is coming from several automatic stations around a country's region. That data is sent to a server along with the action to perform and once done, the server responses back.

If we maintain a serial version of the program, the moment many automatic weather forecast stations send actions, the performance of the server is going to degrade quickly.

The multi-threaded program

If we want to keep server performance in a good state, one solution is to add threads to our program, so looking at the general tasks we can make threads that operate independent from each other.

Since we need to pass more than one argument to the threads we create, we can use a struct to do so:

typedef struct tdata {
    int              action;    /* the action to perform */
    e_operation      operation; /* the operation to perform, if any */
    daily_forecast_t day;       /* the data to work with */
} tdata_t;


int main(int argc, char *argv[]) {
    tdata_t   *thread_data;
    pthread_t *thread;
    
    int action;
    if(argc > 1)
        action = atoi(argv[1]);
    else
        action = -1;
    
    thread_data = (tdata_t*)malloc(sizeof(tdata_t));
    
    thread = (pthread_t*)malloc(sizeof(pthread_t));
    
    ...

This way the data handling falls into the thread's function:

void *thread_job(void *data) { 
   
    tdata_t received_data;
    received_data.action = ((tdata_t*)data)->action;
    received_data.operation = ((tdata_t*)data)->operation;
    received_data.day = ((tdata_t*)data)->day;

 
    switch(received_data) {
        case WRITE:
            func_write();
            break;
        
        case READ:
            func_read();
            break;
        
        case OPERATE:
            func_operate();
            break;
            
        default:
            handle_error("No valid action passed to argv[1]");
            break;
    }
 
    free(data);
    return NULL;
}

— Now instead of creating a new thread each time a station needs to perform an action, we can define a maximum number of threads, initialize them at the beginning of the program, and reuse them in a thread pool.

A thread pool needs to take care of the following things:

• The total number of available threads, so we can limit the number of data requests at the same time.

#define NUM_THREADS 10

• The max size for the data queue, so we can limit the number of requests waiting for service.

#define QUEUE_SIZE 10

Since the queue is a critical part, we need some sort of control over it. We can have a counter to keep track of it, and a mutex to avoid other threads to run over the same queue at the same time.

int queue_count = 0;
pthread_mutex_t data_mutex;

• A way to behave when all threads are working and the data queue is full, so we don't loose data.
• A way to behave if the data queue is empty so we don't overheat the processor.

pthread_cond_t data_cond;

— In terms of design, we could figure out the main behavior of the program in the following steps:

1. The thread pool is waiting until a job is created.
2. The main thread creates a job and signals the thread pool.
3. The thread pool gets the task and executes it.
4. If required, a result is sent back to the main thread.

First of all, we need to define what our threads are going to do when created.

void* start_thread() {

    /* create a struct var to hold data */
    tdata_t data;
        
    /* lock critical part with mutex */
    pthread_mutex_lock(&data_mutex); 
    
    /* if we don't have any data in the queue, we tell the threads to wait */    
    while (data_count == 0) {
        pthread_cond_wait(&data_cond, &data_mutex);
    }
    
    /* if we receive data, then we assign the first element of the queue 
     * to our data holder, and shift the data queue */    
    data = data_queue[0];
    for(int i = 0; i < data_count -1; i++) {
        data_queue[i] = data_queue[i +1];
    }
    
    /* keep track of the data slots */
    data_count--;
    
    /* unlock mutex when done */    
    pthread_mutex_unlock(&data_mutex); 
    
    /* execute the thread job */
    thread_job(&data);
}

Our function thread_job() does not require anymore to be a void* so we can leave it just as a void function.

void thread_job(void *data) { 
   
    tdata_t received_data;
    received_data.action = ((tdata_t*)data)->action;
    received_data.operation = ((tdata_t*)data)->operation;
    received_data.day = ((tdata_t*)data)->day;

 
    switch(received_data) {
        case WRITE:
            func_write();
            break;
        
        case READ:
            func_read();
            break;
        
        case OPERATE:
            func_operate();
            break;
            
        default:
            handle_error("No valid action passed to argv[1]");
            break;
    }
}

Then we need a function to submit jobs with data to the waiting threads:

void submit_job(tdata_t data) {
    
    /* managing the data queue is a critical part so let's lock it
     * before doing anything */
    pthread_mutex_lock(&data_mutex); 
    
    /* assign the data to our data queue and
     * keep track of the data slots */
    data_queue[data_count] = data;
    data_count++;
    
    /* unlock the mutex when done */
    pthread_mutex_unlock(&data_mutex);
 
    /* Wake up one thread */
    pthread_cond_signal(&data_cond);
}

Inside the main function, we can create an infinite loop that listens to user input after we create the thread pool:

The expression for ( ;; ) is the same as while(1)

for ( ;; ) {
    
    printf("\nAutomatic weather forecast station\nWrite action to take: ");
    scanf("%s", buffer);
    action = atoi(buffer);
    
    if(action == EXIT) {
        printf("\nExiting...\n");
        
        free(buffer);
        free(thread_data);
        free(thread);
        
        break;
    }
    
    thread_data->action = action;
    
    submit_data(*thread_data);
}

If we run the code right now, text in the terminal emulator is going to overlap. We need to signal the menu when we are done executing a thread job so the text appears in order.

There are many ways to handle this. Since in this article we talked about semaphores, let's create a binary semaphore that signals when our thread job is done.

Using a simple integer that changes from 0 to 1 can do the trick too.

/* create the semaphore */
sem_t ready_sem;

/* initialize it in the main function, before using it 
 * note that the value is 1, so we can print the menu for the first time */
int main(int argc, char *argv[]) {
    sem_init(&ready_sem, 0, 1);
    ...
}

We need the semaphore to wait before printing the menu:

for ( ;; ){
    sem_wait(&ready_sem);
    ...

And we need to signal once our thread job has finished:

void thread_job(void *data) {
    
    ...
    sem_post(&ready_sem);
}

Now we can operate from the command line without overlapping text messages.

Another option could be not printing any confirmation message from the thread_jobs, leading only errors to prone in the terminal emulator, and that way we can experiment with multiple tasks at a time from a single machine.

Working examples along with compiling instructions are going to be uploaded at unixworks' repo.

Summing up

Threading in computer programs is an extensive field. Covering in depth threads would require more than an article to do it right however, after diving a bit through threading, applied to POSIX and C in this article, we can see that most of it is a game on locking and releasing, waiting and signaling.

Although using threads is not always the best idea to make a program faster, knowing how to implement them can help in our programming design workflows.

There is a newer header for threads, designed for C11 named threads.h which maybe substitutes pthread.h in a future. Right now using it reduces portability and is only available in major C compilers.

Also OpenMP is a multi threading implementation worth mentioning for larger projects. It is an industry standard and is portable and multi-platform.

There are two main types of data we can store in a file: ASCII or text data, and binary data. Binary serialization involves taking some arbitrary set of structured data and transform that data into a consistent stream of bytes.

Many high-level programming languages have custom solutions to achieve binary serialization, while in C there's not a standard solution at all.

Binary data encoding and decoding can be useful in two main fields:

• Networking.
• File saving.

We saw in the previous article how to encode and decode text data, using JSON as our main data structure type.

Text data files are portable and can be moved between computers with ease but, when we ask our computer to work with a text file, it needs to convert it to binary data somehow, which can be rather slow depending the situation.

In the other hand, working with binary data files needs no conversion at all, and usually these kind of files are smaller than text-based ones. We break down complex structures apart and write their individual properties into the buffer. Once it's done, when requested we can read the same data to reconstruct the complex structure.

The main downside is that we cannot print directly their content to our console.

Note that we are using specific byte sizes when defining data. This is important since when we design or use an actually binary file format, its contents are (or should be) ordered in structs specifying which data is inside. The way to know each part of the file is by interpreting those structs which contain the data values.

A practical example with data

Let's suppose we have to create a structured data format for a program that takes daily weather forecast data from the user.

We need at least some constant variables to be filled, such as sunlight hours, cloud formation's type, precipitation's type, rate and amount (if any), minimum and maximum humidity, minimum and maximum temperature, etc. Data the program can later use to make weather predictions and estimations, monthly statistics, etc.

— We can think of our structure as:

typedef enum {
    CLEAR = 0,
    CIRRUS,
    CUMULUS,
    STRATUS,
    NIMBUS
} e_cloud;

typedef enum {
    NONE = 0,
    RAIN,
    DRIZZLE,
    SNOW,
    SLEET,
    HAIL
} e_precipitation;

typedef struct daily_forecast_t {
    float           sunlight_hours;
    e_cloud         cloud_type;
    e_precipitation prec_type;
    float           prec_rate;
    float           prec_hours;
    float           prec_amount;
    float           min_temp;
    float           max_temp;
    float           average_temp;
    float           min_humidity;
    float           max_humidity;
    float           average_humidity;
} daily_forecast_t;

— Now that we have our data structures defined we can create some data to work with. In a real program this data would come from the terminal emulator args or inputs, or from the input fields of a window made for the purpose.

daily_forecast_t day = {
        .sunlight_hours = 14.2,
        .cloud_type = 2,
        .prec_type = 0,
        .prec_rate = 0.0,
        .prec_hours = 0.0,
        .prec_amount = 0.0,
        .min_temp = 7.0,
        .max_temp = 21.2,
        .average_temp = 0.0,
        .min_humidity = 18.3,
        .max_humidity = 25.6,
        .average_humidity = 0.0
    };

day.average_temp = (day.min_temp + day.max_temp) / 2;
day.average_humidity = (day.min_humidity + day.max_humidity) / 2;

— The next step is to store this data so we don't loose any. In a binary format, using fwrite() should be enough. From the previous article, we know we need a FILE handler in write mode where to save our binary data.

FILE *out = fopen("day1.bin", "w");
if(out != NULL) {
    fwrite(&day, sizeof(daily_forecast_t), 1, out);
    fclose(out);
} else {
    printf("%s\n", "there's been an error with the file.");
}

— If we compile and execute our program, we should have a new file named day1.bin in the program's directory.

Trying to read it as plain text is going to print some weird stuff to the console:

$ less day1.bin
33cA^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<E0>@<9A><99><A9>A<9A><99>aAff<92>A<CD><CC><CC>A<9A><99><AF>A
day1.bin (END)

not so useful for the human eye.

Since we are inside a *nix machine, let's use hd to properly inspect the file.

$ hd day1.bin
00000000  33 33 63 41 02 00 00 00  00 00 00 00 00 00 00 00  |33cA............|
00000010  00 00 00 00 00 00 00 00  00 00 e0 40 9a 99 a9 41  |...........@...A|
00000020  9a 99 61 41 66 66 ca 41  66 66 12 42 99 99 f7 41  |..aAff.Aff.B...A|
00000030

This still isn't much useful for us as console readers, but the program can interpret the information contained in it much faster as if it were plain text.

— We suppose the file's content is fine, but we need a way to check it, and to read that data in a workable way. Let's create a read function to do so.

• Since we know how data is structured and it needs to be deconstructed in that specific way, first create a daily_forecast_t variable where to hold the read data.

daily_forecast_t day_data;

• Then create a FILE handler in read mode that reads the file we just saved. To prove it's working let's print the average data values we calculated.

Note that once we've read the file, we have the data stored in memory so it's a good practice to close the file before doing anything else.

FILE *in = fopen("day1.bin", "r");
        if(in != NULL) {
                fread(&day_data, sizeof(daily_forecast_t), 1, in);
                fclose(in);
                printf("Average temp from day has been: %.2f\nAverage humidity from day has been: %.2f\n", day_data.average_temp, day_data.average_humidity);
        } else {        
                printf("%s\n", "there's been an error reading the file.");
        }

• After compiling and running it we should have something like this in the command line:

$ ./weather
Average temp from day has been: 14.10
Average humidity from day has been: 30.95

— Right now, that data structure is filled daily and we have the ability to write a file per day. It'd be great if it could to be stored in a way we can manage days in months, and months in years.

So a month data structure could be:

typedef struct month_forecast_t {
    int days;        /* total days in month */
    void *days_data; /* an array containing all days' data */
} month_forecast_t;

and a year data structure could be:

typedef struct year_forecast_t {
    int months;        /* total months in a year */
    void *months_data; /* an array containing all months' data */
} year_forecast_t;

— If we want to pass our created day's data into a month_forecast_t first we need to allocate some memory for the month's data:

month_forecast_t january;
january.days = 31;
january.days_data = malloc(january.days * sizeof(daily_forecast_t));

and then we can pass the data to our desired day:

january.days_data[0] = day1;

Always remember to free() allocated memory once it's no longer needed.

Creating file formats

At this point, we have our structured data stored in a file, yet we have no way to determine which file is the correct one for our weather forecast program to load and read (except for the file extension if we decide a unique one, but that alone isn't so reliable as an identifier).

Getting it a bit worse, when we apply serialization we need to take care of the compiler's padding system and the way the computer and the OS represent data in binary form. That is Big Endian or Little Endian representation.

— From a technical view, Big Endian store the most significant byte at lower addresses while Little Endian does it the opposite way, storing the most significant byte at higher addresses.

  Little Endian     |       Big Endian
32bit int  memory   |   memory    32bit int
0A0B0C0D   | .. |   |   | .. |    0A0B0C0D
| | | |___ | 0D |   |   | 0A | ___| | | |
| | |_____ | 0C |   |   | 0B | _____| | |
| |_______ | 0B |   |   | 0C | _______| |
|_________ | 0A |   |   | 0D | _________|
           | .. |   |   | .. |

In plain text we can say that Big Endian just represents the data the way we read it, and Little Endian represents the data flipped.

— File formats can help us defining and maintaining how our data should be interpreted, no matter which machine or OS the program is running in.

When creating file formats it's important to take care of the following aspects:

• Unique file identification

It's common between binary file formats to use the first bytes of the file to include a unique number that identifies the format. That is, the magic number or signature of the file.

• Versioning

Our program can grow in the future, and maybe it can handle new data, or maybe some parts are refactored and so the file format. In order to avoid wrong data parsing when this events happen, it's important to keep track of the file format's version.

There's no need to go crazy with conventions. One or two digits should be enough. A more professional approach can be starting at version 1.0.0 use the last digit for patch releases, the middle digit for minor releases and the left one for major releases.

• Header checksum

This is optional (depending on who you ask to) to add into a file format. It mainly let us know that the file isn't damaged.

• Offset to data

This provides a hint on where to start reading the data contained in the file.

Making a simple try for a file format is less scary than you may think. To make things easier from this point, let's use the extension .fct for our forecast program files.

— ADDING A FILE FORMAT HEADER

Diving into it, we have to create a HEADER struct that has to be read before our actual file's data.

typedef struct fct_header_t {
    char     identifier[12];
    char     version;
    char     data_offset;
} fct_header_t;

We can also define a file_id in a way that can be almost unique:

const unsigned char fct_id[12] = {
    //'«', 'F', 'C', 'T', ' ', '1', '0', '»', '\r', '\n', '\x1A', '\n' 
    0xAB, 0x46, 0x43, 0x54, 0x20, 0x31, 0x30, 0xBB, 0x0D, 0x0A, 0x1A, 0x0A
};

— DEFINING A FILE STRUCTURE

Right now we have evolved our program, and just writing out the forecast structure for a day is no longer useful. We have to give our file a structure where we can handle both the header and the data.

typedef struct fct_file_t {
    fct_header_t     header;
    daily_forecast_t data;
} fct_file_t;

Then we can implement our write and read functions separately to make things tidier.

— MAKING A WRITE FUNCTION

Moving the write components out of the main() function allows us to call the function where needed, when needed in a cleaner way.

We can pass to it a fct_file_t parameter as well as a filename parameter (in the form of a char*).

One cool addition is to implement an auto extension for the file name. This is almost useless in terms of functionality, but gives the user the ability to fast check the file in a file explorer visually.

int write_to_file(fct_file_t fct, char *filename) {
    
    /* auto append file extension */
    char *ext = ".fct";
    strcat(filename, ext);
    
    /* open in write mode*/
    FILE *out = fopen(filename, "w");
    
    /* check if there's a problem with the file before doing anything*/
    if(out != NULL) {
        fwrite(&fct, sizeof(fct_file_t), 1, out);
        fclose(out);
        printf("%s written!\n", filename);
        return 0;
    } else {
        printf("there's been an error with the file %s", filename);
        return 1;
    }
}

We can handle specific chunks of data by creating buffers and passing the info through functions like memcpy() .

Let's say we have the following header:

fct_file_t file1 = {
        .file_header = header,
        .data = day1
};

and we want to copy just the header to a buffer:

unsigned char *header_buffer = (unsigned char*)malloc(sizeof(fct_header_t));

The code would perform something like this:

identifier (12) -> ab 46 43 54 20 31 30 bb 0d 0a 1a 0a -> header_buffer
version (1) -> 01 -> header_buffer
data_offset (1) -> 0e -> header_buffer

which we can confirm by printing out the value of the header_buffer:

printf("header_buffer content:\n");
    for(int i = 0; i < sizeof(fct_header_t); i++)
        printf("%02x ",header_buffer[i]);

printf("\n");

Now we can use our buffer for many things, since it's structured and we can handle the information.

— MAKING A READ FUNCTION

Same as the write function, moving the read file instructions away from the main() function gives us more freedom later when scaling the program.

One key part in the reading function are to check if the data is valid for our program. We can achieve it by comparing the file header id against our known fct_id we created earlier using memcmp().

int read_from_file(char *filename){
    
    /* generate a temporal fct_file type to store read data */
    fct_file_t tmp_fct;
    
    /* open in read mode */
    FILE *in = fopen(filename, "r");
    
    /* check if there's a problem with the file before doing anything*/
    if(in != NULL) {
        fread(&tmp_fct, sizeof(fct_file_t), 1, in);
        fclose(in);
        
        /* verify it's a valid fct file comparing the header id */
        if (memcmp(&tmp_fct.file_header.id, fct_id, 12) != 0) {
            printf("%s\n", "Not a FCT V1 file or corrupted file. Identifier isn't valid");
            return 1;
        }
        
        /* print some file header info */
        printf("fct file version is %d\n", tmp_fct.file_header.version);
        printf("we can skip %d bytes to reach our data\n\n", tmp_fct.file_header.data_offset);

        /* test print some file data */
        printf("Average temp from day has been: %.2f\nAverage humidity from day has been: %.2f\n", tmp_fct.data.average_temp, tmp_fct.data.average_humidity);
        printf("Cloud type from day has been: %d\nPrecipitation type from day has been: %d\n", tmp_fct.data.cloud_type, tmp_fct.data.prec_type);
        
        return 0;
    } else {    
        printf("there's been an error with the file %s", filename);
        return 1;
    }
}

— MAKING A FLEXIBLE FILE FORMAT

At this point we can save forecast data for a day and read it back. Scaling the project, we could make our program ask the user to create a forecast session, initializing a data structure inside the program that can be stored in a file that understand days inside months, and months inside years (using the structs we proposed earlier in the article).

That way we would be able to give flexibility to the users, allowing them to work with data for an entire year, appending and/or modifying values over one file across time.

A practical example with existing file formats

Having to deal with existing file formats is another reality worth looking at. Usually when designing a program to interact with other programs' data, it's common to use file formats that actually exist.

Tasks like saving or reading pixel data for images and video, loading or writing samples data for audio have already a huge amount of available file formats to work with.

Unless we need a custom tailored data format to our software (because of encryption or efficiency), using existing file formats can give us some benefits like:

• Avoid to reinvent the wheel.
• Portability.

— We are going to check the TGA image file format for the example, so we need to find the file specification to know where to start.

TGA files store red, green and blue channels with 8 bit precision each. This leaves us with 24 bits per pixel.

TGA files also offer an additional 8 bit alpha channel that can be really useful. Assuming we are working with RGBA it should be 32 bit per pixel.

According to the format specification we have a header which contains the following data:

• Image ID length (1 byte) usually contains the date and time the file was created.
• Color map type (1 byte) handles whether a color map is included. Can be 0 or 1.
• Image type (1 byte) contains compression and color type information.
• Color map specification (5 bytes) describes the color map.
• Image specification (10 bytes) has Image dimensions and format.

—This gives us an 18 byte header where we know which data goes to each part. We can translate that info into a struct like this:

typedef struct tga_header_t {
    char id_size;
    
    char color_map_type;
    
    char image_data_type;
    
    short int color_map_origin;
    short int color_map_length;
    char  color_map_depth;
    
    short int x_origin;
    short int y_origin;
    short image_width;
    short image_height;
    char  bits_per_pixel; 
    char  image_descriptor;
} tga_header_t;  

Next we have to define how our pixels are constructed:

typedef struct pixel_t {
    unsigned char r;
    unsigned char g;
    unsigned char b;
    unsigned char a;
} pixel_t;

So a TGA file can be described as such struct:

typedef struct tga_file_t {
     tga_header_t header; /* a header struct defining the type of TGA */
     pixel_t *pixels;     /* the pixels forming the image */
} tga_file_t;

Now if we want to open a TGA file and check what info does it have inside the header we can create a read function like this:

int read_tga_file(char *filename) {
    FILE *fptr;
    fptr = fopen(filename, "r");
    tga_header_t tmp_header;
    pixel_t *pixels;
    
    tga_file_t tmp_file;
    
    if(fptr != NULL) {
        /* in order to get the image width and height we have to 
           jump into the desired position of the file.
           Since we know that they are 12bytes from the origin, we can use
           fseek() */
        fseek(fptr, 12, SEEK_CUR);
        
        fread(&tmp_file.header.image_width, 2, 1, fptr);
        fread(&tmp_file.header.image_height, 2, 1, fptr);
        printf("image %s has width: %d and height %d\n", filename, tmp_file.header.image_width, tmp_file.header.image_height);        
    } else {
        printf("there's been an error with the file %s\n", filename);
        return 1;
    }    
}

If we want to work with the pixel data, we need to create some space to handle their information.

We know that an image is constructed in two dimensions (x and y) given a size for each dimension (width and height). At each point of that 2D grid there's a pixel, which we assume contains four values as color information (RGBA).

tmp_file.pixels = malloc(tmp_file.header.image_width * tmp_file.header.image_height * sizeof(pixel_t));

To avoid having garbage data we can initialize each value to zeros:

for(int i=0; i<tmp_file.header.image_width * tmp_file.header.image_height; i++) {
     tmp_file.pixels[i].r = 0;
     tmp_file.pixels[i].g = 0;
     tmp_file.pixels[i].b = 0;
     tmp_file.pixels[i].a = 0;
}

Now we can read the image. Let's change some color values as a quick test. Here we have a function that stores the referenced value from an actual image pixel into our own pixel data, only leaving the green channel to its absolute value.

/* TGA is Little Endian encoded, so values in pixel are bgra instead of rgba. */
void change_pixel_color(pixel_t *pixel, unsigned char *p_value) {
      pixel->a = p_value[3];
      pixel->r = p_value[2];
      pixel->g = 255;        /* p_value[1]; */
      pixel->b = p_value[0];
}

So after allocating our pixel data from the file pointer we can run that function until we reach the last pixel in the image like this:

int n = 0;
char p_value[4];

while(n < (tmp_file.header.width * tmp_file.header.height) ) {
    fread(p, 1, 4, fptr);
    change_pixel_color(&(tmp_file.pixels[n]), p_value);
    n++;
}
fclose(fptr);

To write the data out into a file we can whether reopen our file in write mode and overwrite data there, or open a new file in write mode, and then store the new pixel data along with a TGA header.

Here's a result of a four pixel TGA Image passed through the previous change_pixel_color() function:

Summing up

Now that we've covered both ways of storing data from a program inside the computer, it's time for your creativity to flow in your next project.

Some data types and projects require more tailored byte buffers than just a bulk read and a bulk write of bytes. That topic will be covered in other series of articles.

Full working examples of the code shown above are coming to the unixworks repository. Stay tuned, and I'll see you in the next article (:

At some point when developing software no matter how big o small the program is going to be, we need to store some data in the computer, and read from other sources too. Let's take a look at how to work with external files in C.

Files in C programming don't have a predefined structure. They are meant to be a container for some sequence of bytes. That way the internal structure of a file is something that the program itself has to deal with.

As long as we know how a file structure is made, we can open, work and write with any file.

Opening files

Opening files in C can be achieved in two ways; using the stdio function fopen() or using the lower level one open().

The main difference between them is that open() is a system call while fopen() is a library call.

fopen() calls open() under the hood and uses buffering to improve execution timing. When timing is critical(eg. embedded systems), is better to use open() and take full control on when we want the data to be processed.

— The fopen() way

The fopen() function associates a file with a stream and initializes an object of the type FILE, which contains a structure with information to control the stream.

We can specify how we want to operate with the data by passing different modes into the mode parameter.

Possible modes are:

• r opens a file for reading
• w creates a file for writing. If it's not empty, it discards the previous content.
• a opens or creates a file (in case it doesn't exist) and writes at the end of it.

Adding a + sign after any of the letters make the file to work in update mode. That is, the mode allows both reading and writing.

FILE *file = fopen("path/to/file.type", "mode");

— The open() way

The open() function returns an int object called file descriptor. Every open file has a file descriptor number, which is used by the operating system to keep track of them.

Similar to fopen(), we can specify how we want to work with the opened file passing specific flags into the flags parameter.

Valid mandatory flags are:

• O_RDONLY which opens a file in read-only mode.
• O_WRONLY which opens a file in write-only mode.
• O_RDWR which opens a file in read/write mode.

Additional flags can be added in order to perform other operations such as O_APPEND to open a file in append mode, or O_ASYNC to use a pipe of a FIFO.

We can add a third optional parameter to specify permissions of the file, like:

• S_IRUSR user has read permissions.
• S_IWUSR user has write permissions.

int fileData = open("path/to/file.type", flags, mode);

Writing files

We can run a program that takes arguments from the user via the terminal emulator, and perform operations based on those arguments, print them back to the terminal, and ask for more operations if needed, but each time we close the program, that data is gone.

We can write data in binary files and in text files.

The standard library has two useful functions to help us in the task of saving that data we ask for and process during the program execution, into a file. These functions are fwrite() and fprintf().

— Using fwrite()

The function fwrite() writes a number of objects of a given size to a file. Is often used to write binary data.

The information we need to pass to fwrite() is the following:

• A memory buffer, or the address of the data to store.
• The size in bytes of each element of the data to store.
• The amount of elements to write.
• A pointer to a FILE object.

fwrite(&data, sizeof(data_type), strlen(data), file);

This is going to return us a binary file. We can check its content using a tool like hexdump(1).

typedef struct Car {
    int power;     //kW
    int torque;    //NM
    int wheels;    //[4, 5]
    int seats;     //up to 7
    int doors;     //[3, 5]
}

Car rallyCar {
    .power = 235,
    .torque = 384,
    .wheels = 5,
    .seats = 2,
    .doors = 3
};

FILE *file = fopen("cars.bin", "w");

fwrite(&rallyCar, sizeof(Car), 1, file);

fclose(file);

— We can however, write text files using fwrite() by making use of the function sprintf(), which writes its output as a string in the buffer referenced.

char buffer[40];

sprintf(buffer, "The actual engine torque is %f.\n", engine.torque);
fwrite(buffer, sizeof(char), strlen(buffer), file);

fclose(file);

— Using fprintf()

Similar to the printf() function, we have fprintf() in the standard library, with which we can write formatted outputs into a file, passing a character constant as a format parameter.

The information we need to pass to fprintf() is the following:

• A file pointer of type FILE.
• The desired output format, which is a const *char.
• The desired content to format.

fprintf(file_pointer, format, content);

This way we store text data by default in a file.

FILE *file = fopen("temp.log", "a");

if (file != null)
   fprintf(file, "%s\n", "Appending data to temp file.");

fclose(file);

At the end of the article we'll use this function to serialize some JSON data.

Other operations with files

Apart from opening and writing files the header file stdio.h has more functions required to work with I/O which we can use to rename, remove, and close files among other operations.

— Close a file

Once we are done working with a file, we can close the stream and free up the memory using the function fclose(). The function deletes any unwritten data for the stream and discards any unread buffered input, so be sure to write changes before.

fclose(file);

— Rename a file

We can rename a file using the function rename() by passing the name of the old file and a string (const *char) to use as the new one.

rename("old_file_name", "new_file_name");

— Remove a file

We can make a file unavailable using the remove() function, passing the file's filename. If the file has no other names linked, then the file is deleted. Depending on the mode used by the file, the function may or may not be able to perform the deletion.

remove("file_name");

— Create a temporary file

Using tmpfile() we can create a temporary file with a unique name in wb+ mode which is automatically removed once we close it or the program terminates.

If the function is unable to open a temporary file, it returns a NULL pointer, otherwise it returns a pointer to the temp file.

FILE *file = tmpfile(); //file is pointing to a tempfile.

How to map files in memory

There is a way to work more efficiently with files, that is allocating them in virtual memory with mmap.

Virtual memory helps when the processes ask for more memory than the system has. At that point the operating system's memory management takes memory from the RAM and places it into the swap, bringing it back to the RAM when requested. Is basically moving data from the RAM to the hard drive back and forward.

We can use that way of work to read and write files too.

Let's use mmap to request blocks of memory from a text file (it can be any other file too):

— Open a file

int fileData = open("text_file.txt", O_RDONLY, S_IRUSR | S_IWUSR);

If we want to also write content into the file we have to open it in a read-write mode using different flags in the open() function:

int fileData = open("text_file.txt", O_RDWR, S_IRUSR | S_IWUSR);

We can do the same using fopen(), but is a good thing not to mix high level I/O with low level operations. We would killing the performance.

If we use fopen() then we need to use the function fileno() to get the file descriptor from our opened file.

FILE *fileData = fopen("text_file.txt", "r");
int fileDescriptor = fileno(fileData);

— Get the size of the file

We need to include <sys/stat.h> and <unistd.h> to help:

#include <sys/stat.h>
#include <unistd.h>
...
struct stat sb;
if(fstat(file, &sb) == -1)
    printf("couldn't get file size\n");

— Allocate in memory using mmap()

We need to pass the following parameters to the function:

• The desired starting address, NULL in this case, letting the system to choose the address.
• The length of the file to map. We are using file status to get the total size in bytes with sb.st_size.
• The flag or flags representing how we want to operate with the memory page.

If we just want to read the file it's PROT_READ. If we want to read and write the file it needs to be PROT_READ | PROT_WRITE.

• The flag or flags representing if the mapping is going to be shared with other processes or not. In this case MAP_PRIVATE.

If we want to write the file we need to change MAP_PRIVATE to MAP_SHARED otherwise the program is not going to share the memory with the rest of the system, and it's not going to be able to write back to the file.

• The file descriptor from our opened file, fileData.
• The offset where to start mapping the file, in this case 0, which is the beginning.

char *fileInRAM = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fileData, 0);

— Operate with the data

Now that we have mapped our file we can start working freely with it.

for (int i = 0, i < sb.st_size; i++)
    printf("%c", fileInRAM[i];
printf("\n");

— Unmap memory and close the file

Once we're done working with the file, just by closing the file descriptor we don't unmap the data. The function munmap() takes mapped file and deletes its mappings in the specified address range.

After that we can close the file descriptor to finish.

munmap(fileInRAM, sb.st_size);
close(fileData);

A complete view of the code should look like this:

int main()
{
  int fileData = open("plain_text_file.txt");
  
  struct stat sb;
  
  char *fileInRAM = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fileData, 0);

  for (int i = 0, i < filesize; i++)
    printf("%c", fileInRAM[i];

  munmap(fileInRAM, filesize);
  close(fileData);
}

Structuring data

We know that the C programming language doesn't care about the type of file we use. Some applications may be fulfilled by storing data in plain text files, but even by being text files, they may need to follow a structure so we can interoperate later with the data inside them.

To achieve this we need to convert the abstract in-memory data into a series of bytes that record the data structure into a recoverable format. This is called serialization.

Our data structure can be a simple list or array, a complex group of nested arrays and structs, or whatever required.

Writing structured data to a file

— As an example, let's take a look at a program where the user can store information about a vehicle's engine.

• We should have a struct type that handles how an engine is defined.

/*simplified engine structure*/
typedef struct Engine {
    char model[10];                  //engine model
    char manufacturer[10];           //engine manufacturer
    int power;                       //kW
    int torque;                      //NM
    int cylinders;                   //total cylinders in engine
    int structure;                   //block structure [1, 2, 3] rows
    char fuelType[10];               //fuel type [gasoline, diesel]
} Engine;

• Once we are working in the program we can create an engine and assign values to it.

Engine engine {
    .model = "RB26DETT",
    .manufacturer = "nismo",        
    .power = 235,                  
    .torque = 384,                  
    .cylinders = 6,            
    .structure = 1,
    .fuelType = "gasoline"     
};

• Now it's time to define a constant to serialize the data into a file. Instead of reinventing the wheel, let's use an existing data-interchange format such as JSON (XML applies here too).

const char *ENGINE_EXPORT_FMT =
"{\n\t\"model\": \"%s\",\n\t\"manufacturer\": \"%s\",\n\t\"power\": %d,\n\t\"torque\": %d,\n\t\"cylinders\": %d,\n\t\"structure\": %d,\n\t\"fuel\": \"%s\"\n}\n";

Most of the "complexity" here is to correctly describe our object. As for this simple example, we can just go with this constant. For serious projects we would need to improve this in a header file and probably make some functions that warp the process.

• Moving on, we have to open a file to write the data to, or create a new one.

FILE *file = fopen("engine_data.json", "w+");

• Once we have our file opened, we need to print the content of our engine struct into it, using the function fprintf().

fprintf(file, ENGINE_EXPORT_FMT, engine.model, engine.manufacturer, engine.power, engine.torque, engine.cylinders, engine.structure, engine.fuelType);

Note that we have named our example file as .json but we could actually add the name and extension we'd want, and the result would be the same.

A complete view of the code should look like this:

#include<stdio.h>
#include<stdlib.h>

/*engine struct format data*/
const char *ENGINE_EXPORT_FMT = "{\n\t\"model\": \"%s\",\n\t\"manufacturer\": \"%s\",\n\t\"power\": %d,\n\t\"torque\": %d,\n\t\"cylinders\": %d,\n\t\"structure\": %d,\n\t\"fuel\": \"%s\"\n}\n";

/*simplified engine structure*/
typedef struct Engine {
    char model[10];                  //engine model
    char manufacturer[10];           //engine manufacturer
    int power;                       //kW
    int torque;                      //NM
    int cylinders;                   //total cylinders in engine
    int structure;                   //block structure [1, 2, 3] rows
    char fuelType[10];               //fuel type [gasoline, diesel]
} Engine;


int main()
{
    Engine engine {
        .model = "RB26DETT",
        .manufacturer = "nismo",        
        .power = 235,                  
        .torque = 384,                  
        .cylinders = 6,            
        .structure = 1,
        .fuelType = "gasoline"     
    };
    
    FILE *file = fopen("engine_data.json", "w+");
    
    fprintf(file, ENGINE_EXPORT_FMT, engine.model, engine.manufacturer, engine.power, engine.torque, engine.cylinders, engine.structure, engine.fuelType);
    
    fclose(file);
    
    return 0;
}

We should have a new file named engine_data.json in our directory with the engine struct parsed into it.

Parsing structured data from a file

If we want the saved data to be used back in the program, we have to kinda reverse engineering our constant to parse our object.

• Create a new constant char.

const char *ENGINE_IMPORT_FMT =
"{\n\t\"model\": \"%[^\"]\",\n\t\"manufacturer\": \"%[^\"]\",\n\t\"power\": %d,\n\t\"torque\": %d,\n\t\"cylinders\": %d,\n\t\"structure\": %d,\n\t\"fuel\": \"%[^\"]\"\n}";

• We need to specify where we want to start reading the data from the file.

fseek(file, 0, SEEK_SET);

• Finally we can assign the read data to a new variable using fscanf().

Engine iEngine;
fscanf(file, ENGINE_IMPORT_FMT, iEngine.model, iEngine.manufacturer, &iEngine.power, &iEngine.torque, &iEngine.cylinders, &iEngine.structure, iEngine.fuelType);

A complete view of the code should look like this:

#include<stdio.h>
#include<stdlib.h>

/*engine struct format data*/
const char *ENGINE_IMPORT_FMT = "{\n\t\"model\": \"%[^\"]\",\n\t\"manufacturer\": \"%[^\"]\",\n\t\"power\": %d,\n\t\"torque\": %d,\n\t\"cylinders\": %d,\n\t\"structure\": %d,\n\t\"fuel\": \"%[^\"]\"\n}";

/*simplified engine structure*/
typedef struct Engine {
    char model[10];                  //engine model
    char manufacturer[10];           //engine manufacturer
    int power;                       //kW
    int torque;                      //NM
    int cylinders;                   //total cylinders in engine
    int structure;                   //block structure [1, 2, 3] rows
    char fuelType[10];               //fuel type [gasoline, diesel]
} Engine;


int main()
{
    Engine engine;
    
    FILE *file = fopen("engine_data.json", "r");
    
    fseek(file, 0, SEEK_SET);
    
    fscanf(file, ENGINE_EXPORT_FMT, engine.model, engine.manufacturer, &engine.power, &engine.torque, &engine.cylinders, &engine.structure, engine.fuelType);
    
    fclose(file);
    
    return 0;
}

Summing up

Files play a really important role in software programs. We've seen how to work with operations that read, write and format text both from and into files, but the same can be achieved for binary files such as images or audio.

In addition to that, we can also implement ways to obfuscate how our program writes the data so not everyone can open our format back. This is kind of an unfriendly way to do the things, but corporate often makes this so the competition cannot just sneak into a company's new software and steal how they engineer things. But hey, we have reverse engineers to do so (:

A further discussion in this field will be present in a future article.

Accessing memory locations is one of the greatest features of the C programming language, although it requires some responsibility. The word pointer often scares programmers away, but it shouldn't.

Pointers give support for dynamic memory allocation, level-up flow control in a program and are closer to hardware which makes code more efficient.

What are pointers?

In C programming, variables hold values at a specific memory address. Pointers are variables that hold memory addresses and types of other variables and functions, giving direct access to physical memory locations anywhere inside the computer.

Given the variable var, &var is a pointer to var.

You can think about pointers and variables like license-plates and vehicles. While vehicles can seize too many types and forms, license-plates usually come in an unified form.

How pointers work

With pointers it's possible to access any memory location and change the data contained at that location.

A pointer is declared by adding an asterisk (*) in front of the variable name in the declaration statement.

It's heavily recommended to initialize pointers as NULL since when we create a pointer it isn't initialized and holds random data that can point to anywhere in the computer memory.

int variable_name = 8;         //define a variable
int *variable_pointer = NULL;  //define a pointer to a variable

NULL is a macro to address 0. In programming terms 0 is an invalid address. It can be defined like this:

#define NULL ((void*)0)

— In order to work with declared pointers, we have two basic pointer operators:

• & Address constant of an object in memory. Given a variable, point to it.

/* we pass a variable asking for its memory address */
printf("%d\n", &variable_name);

/* the program should return a memory address */
"0xfbee324b"

• * Content of a memory address. Given a pointer, get the value stored in it. This is usually called pointer dereferencing.

/* make our pointer point to the address of the given variable */
variable_pointer = &variable_name;

/* we pass a variable asking for its value */
printf("%d\n", *variable_pointer);

/* the program should return the content of the memory address */
"8"

Let's make a quick reminder of how to work with simple pointers:

int main() {
/* define a variable for a number */
  int num;
  
/* define a pointer to num */
  int *int_ptr = NULL;

/* add a value to num */
  num = 14;

/* now make the pointer point to num. This assigns num address to int_ptr */
  int_ptr = #

/* let's check what values contain each variable */
  printf("num = %d\n", num);
  printf("&num = %p\n", &num);
  printf("int_ptr = %p\n", int_ptr);
  printf("*int_ptr = %p\n", *int_ptr);
  printf("&int_ptr = %p\n", &int_ptr);

/* int_ptr points to num, changing int_ptr value modifies num too */
  *int_ptr = 8;
  printf("modified *int_ptr = %p modified num to num = %d\n", *int_ptr, num);

  return 0;
}

The result of that program should be similar to this:

num = 14
&num = 0x6fff86d087a5
int_ptr = 0x6fff86d087a5
*int_ptr = 14
&int_ptr = 0x6fff86d087a5
modified *int_ptr = 8 modified num to num = 8

Pointer utilities

We've seen a quick refresh of how pointers work. Now let's take a look at some options pointers give to us.

— We can have multiple pointers pointing to the same variable.

int main() {
  int num;
  int *first_ptr;
  int *second_ptr;
  
  num = 14;
  
  first_ptr = #
  second_ptr = first_ptr;

  return 0;
}

since first_ptr and second_ptr are both pointers we can reference them.

— We can pass pointers as function arguments.

Passing data using a pointer allows the function to modify the external data.

If we try to do the same with data passed as values instead of pointers then we only modify the function parameter, and not the original value since the addresses of the parameter and the variable in main are not the same.

void ModifyData(int *data);

int main() {
    
    int externalData = 10; 
    
    printf("\nExternal data value before modify is %d", externalData);
    
    ModifyData(&externalData);
    
    printf("\nExternal data value after been modified is %d", externalData);
} 

void ModifyData(int *data) {
    
    *data = 0;
}

The result should be:

External data value before modify is 10 
External data value after been modified is 0

However, we can't change the actual pointer to the data since passed a copy of the pointer.

A common practical example using pointers as function parameters is a swap function.

void SwapFloat( float *a, float *b) { 
    float tmp = *a; 
    *a = *b; 
    *b = tmp; 
}

— We can pass pointer to a pointer as a function argument.

This way we can modify the original pointer and not its copy. Similar to passing a variable in the previous example.

— We can return pointers.

This is pretty much straight forward. We have to declare the return type to be a pointer to the appropriate data type.

int *RoundFloat(float num);

An example implementing the function:

int *RoundFloat(float *num);

int main()
{
    float fnum = 5.23;
    int *frounded;
    
    frounded = RoundFloat(&fnum);
    
    printf("rounded value from %f is %d.", fnum, *frounded);
    return 0;
}

int *RoundFloat(float *num) {
    int *tmp;
    *tmp = ((*num + 0.5f) *1) /1;
    
    return tmp;
}

— We can create function pointers.

A function pointer is a variable that stores the address of a function to be used later on the program.

typedef float (*OperationsTable)(float, float);

When we call a function, we might need to pass the data for it to process along pointers to subroutines that determine how it processes the data.

typedef float (*OperationsTable)(float, float);

float Add( float x, float y) { 
    return x+y;
}
float Sub( float x, float y) { 
    return x-y;
}

float Operate(OperationsTable opTable, float x, float y) { 
    return opTable(x,y);
};

int main() {
    int a, b;
    int a = 5;
    int b = 10;

    Operate(Add, a, b);
}

Another option is to store function pointers in arrays and later call the functions using the array index notation.

float Add( float x, float y) { 
    return x+y;
}
float Sub( float x, float y) { 
    return x-y;
}

float (*OperationsTable[2])(float, float) = { Add, Sub};

int main() {
    int a, b;
    int a = 5;
    int b = 10;

    OperationsTable[0](a, b);
}

— We can use pointers with structs.

Normally we access struct components with a dot . but when a struct is marked as pointer, we access their values using the point-to operator ->.

Note that is possible to still use a dot . , but then the call to the component is as follows: (*foo).variable.

typedef struct Vector3 {
    int x;
    int y;
    int z;
} Vector3;

int main() {
    Vector3 origin = {3, 5, 10};
    
    Vector3 *point;
    
    point->x = origin.x;
    point->y = 0;
    point->z = origin.z;
    
    printf("\npoint values are x = %d | y = %d | z = %d", (*origin).x, (*origin).y, (*origin).z);
} 

— We can define strings.

There's no such thing recognized as a "string" in C. Strings in C are arrays of characters terminated with a NUL (represented as \0).

char *title = "unixworks";

The array way of creating a string literal would be:

char title[] = "unixworks";

Which is the equivalent to:

char title[] = {'u', 'n', 'i', 'x', 'w', 'o', 'r', 'k', 's', '\0'};

Note that using the pointer approach to create strings doesn't allow to modify the string later as it's supposed to be treated as a const.

However, we can work with the string pointer as an array. It will return the value of the first character, since the variable actually points exactly to the beginning of the string.

Pointers and arrays

Although pointers and arrays aren't the same thing, they can work hand to hand in C. In most of the cases, the name of the array is converted to a pointer to the first element.

• An array notation like array[index] can be achieved using pointers with *(array + index).
• The same way the array notation &array[index] can be achieved using pointer notation array + index.

Arrays in C programming need to have its size declared when we create them, or at least we are told to do so when learning. Other programming languages can perform dynamic arrays without declaring its size when created.

—The fact is that we can create dynamic arrays in C combining pointers and arrays. Managing memory in real-time is extremely useful to arrays that are generated at run-time.

The only prerequisite to create a dynamic array using pointers is to reserve memory for it. That is achieved calling malloc().

int *num_ptr;
num_ptr = malloc(MAX_NUMBERS * sizeof(int));

where:

• (int *) casts the data type.
• MAX_NUMBERS can be whatever value that determines the max elements in the array.
• sizeof(int) is the amount of bytes that each element in the array holds.

Dynamic array of void pointers

A useful utility mixing pointers and arrays we can create is a dynamic array of void pointers.

We start defining a struct as follows:

typedef struct Set {
    void **data;
    int capacity;
    int count;
} * Set_t;

Where we have:

• void **data that are void pointers stored as a dynamic arrays. When used in a pointer, void defines a generic pointer (pointer of any type).
• capacity which is the total allowed items.
• count which is the current amount of items. It acts as an index for the stored data.

We can initialize our List structure with the same criteria for dynamic arrays, using malloc():

Set_t set = malloc(sizeof(struct List)); 

*set = (struct Set_t) { 
    .count = 0, 
    .capacity = 1, 
    .data = malloc(sizeof(void *))
};

If we want to add data to our list, we can increase the count value of our struct:

set->count += 1;

and compare it against the capacity value.

if(set->count == set->capacity) {
    set->capacity *=2; 

    set->data = realloc(list->data, list->capacity * sizeof(void *));
}

As most of the data structures, this dynamic array of pointers becomes useful when we create some functions to work with it.

As an example we can make a function to get a value from an index of the set, and another one to check if a value is contained in an index of the set.

void *IndexValue(Set_t set, int index) { 
    if (index > set->count) {
        printf("Index is out of bounds.\n");
        exit(1);
    }
    return set->data[index];
}

void SetContains(Set_t set, void * value) {
    for (int i = 0; i < set->count; i++) { 
        if (Index(set, i) == value) 
            printf("Value is in the set.\n"); 
    }
    printf("Value is not in the set.\n");
}

Linked lists

Arrays are fine, but they can be inefficient depending the program to create and the target device architecture.

Linked lists are a data structure. Instead of asking a large contiguous block of memory in a request to store an array, ask for one data unit at a time, for one element at a time in a separate request.

Let's say we have some data we want to store as a list.

int x, y, z, w;

This makes the memory to allocate the data in non contiguous memory blocks. But we need to link the memory blocks in some way.

+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  x  |     |     |  y  |     |  z  |     |     |     |  w  |     |     |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

One common solution is to store next to each data value, the memory address of the next data block.

—This can be represented in C creating a struct, which we can name Node, where we store the data value, and the next node address (a pointer):

typedef struct Node {
  int data;
  node* next;
} Node_t;

This way we'll have something like this:

+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  x  |y_mem|     |  y  |z_mem|  z  |w_mem|     |  w  |  0  |     |     |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

instead of

arr[3] = {x, y, z, w};

+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|     | a[0]| a[1]| a[2]| a[3]|     |     |     |     |     |     |     |
+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+

We are using some extra memory in the list compared with the array method, but that gives us the ability to create and free nodes dynamically where we need, and when we need.

—Worth mention here is the last node of the list points to NULL or 0 as the next node address, indicating there's no more data in the list.

In the other hand, the address of the first node of the list gives us access to the complete linked list. Usually this first node is called head.

Node_t *head = NULL;

If we'd need to create another data link to our list, we first need to create a separate node, and then link the last node address to the newly created node instead of NULL.

Node_t *nodeA; 

nodeA = malloc(sizeof(node_t));
nodeA->data = 10;

head = nodeA;

We can also insert nodes anywhere in the list. The only thing to take care is to relocate the address values of each node.

Node_t *head = NULL;
Node_t *nodeA, *nodeB; 

nodeA = malloc(sizeof(node_t));
nodeB = malloc(sizeof(node_t));

nodeA->data = 10;
nodeB->data = 43;

head = nodeB;

nodeB->next = nodeA;
nodeA->next = NULL;

At this point the process starts to repeat itself a lot, and programming is intended to automate tasks. We can organize this a bit, creating a function that creates nodes for us.

Node_t *CreateNode(int data) { 
    Node_t *result = malloc(sizeof(node_t));
    result->data = data;
    result->next = NULL;
    return result;
}

This way we can start working dynamically and add nodes each time we need them.

Node_t *head = NULL;
Node_t *dummy; 

dummy = CreateNode(10);
head = dummy;

dummy = CreateNode(43);
dummy->next = head;
head = dummy;

This data structures are more useful if we implement functions to work with them. As an example, we can make a function to locate a node inside the list.

Node_t *LocateNode(Node_t *head, int data) {
    Node_t *tmp = head;
    while(tmp != NULL) {
        if (tmp->data == data)
            return tmp;
        tmp = tmp->next;
    }
    return NULL;
}

Another great feature in linked lists is the possibility to insert nodes at a certain point of the list. We can make a function for it too:

void InsertNodeAt(Node_t *insertPoint, Node_t *newNode) {
    newNode->next = insertPoint->next;
    insertPoint->next = newNode;
}

Summing up

Pointers open a huge field of possibilities in C but remember, with great power comes great responsibility.

We have to take care of the heap use in some way. Using pointers introduce us the power to dynamically allocate elements in memory and this can cause to out of memory errors.

When using malloc() to allocate memory, we know that it will return NULL if it runs out of memory, so a good practice is to check if we really got the memory needed when allocating.

pointerVar = malloc(sizeof(type_t)); 

if (pointerVar == NULL) {
    printf("\nOut of memory");
    exit(1);
}

Another good practice is to free up memory once we are done using it. A reminder on working with memory can be found here.

We need memory to perform processes and store values. C programs usually get their memory by calling the function malloc() and release that used memory calling the function free() when they're done.

Some programming languages have garbage collectors to take care of memory. C doesn't. It may be seen as a negative aspect, but it's in fact a really good one since we as programmers can have more specific control on how memory is managed.

— There are three basic ways to store memory. If we know the memory needed, it can be stored as static or as automatic, and if we don't know how much memory we are going to need, then it can be stored as dynamic.

• static memory allocation applies to global variables and variables marked with static. It is handled when the program starts and has a fixed size when the program is created.

int horsePower = 120;
static float pressure = 34.0f;

• Automatic memory allocation applies to variables defined inside functions that aren't marked as static.

function GetTorque(EngineType engine)
{
  float defaultTorque = 2.4f;
  ...
}

• dynamic memory allocation is performed at run-time to allocate an arbitrary amount of memory at an arbitrary point in the program. This operation is handled by the operating system the program is running on, and the memory itself it's allocated on the heap.

int *speed = malloc(sizeof(int);

— The memory assigned to a program in a common architecture can be divided in four blocks:

+-------------+-------------+-----------+---------------------------+
|    Code     |  Static     |   Stack   |           Heap            |
+-------------+-------------+-----------+---------------------------+

• Code stores the instructions to execute, the program code.
• Static/Global stores the variables declared outside functions (and the ones marked as static) that consequently are accessible anywhere while the program is running. This block is available until the program closes.
• Stack stores the information from function calls and local variables. If we exceed the amount reserved for this block, the program will crash. The point about the information inside the stack block is that once a process ends, it's automatically removed from the memory block (until it's needed again).
• Heap/Global stores large amounts of memory and maintains the variables in memory. Unlike the other blocks, the size of the Heap block is not fixed and we can control how much memory we want to use, and for how long we want to maintain data in the memory.

The way a heap block is implemented can vary between operating systems or compilers. When we work with dynamic memory allocation, we're always working with the heap memory block.

The only limit for the heap block is the available amount of memory that the system running the program has.

malloc, calloc, realloc & free

These are the four functions that generally deal with dynamic memory allocation in C. They are included in stdlib.h

— malloc

void* malloc(size_t size)

When we call malloc(), we are asking for a block of memory of a certain size in the heap memory block. malloc() returns a pointer to a block.

• If there's not more memory available, malloc() returns NULL.
• malloc() doesn't initialize the allocated memory.

int *speed = malloc(sizeof(int);

speed = 180;

— calloc

void* calloc( size_t num, size_t size)

If we know the number of elements that we want to store and the size of each element, we can use calloc().

• calloc() also initializes the bytes in the block to zeroes, which avoids random garbage. This is useful when debugging.

int *checkpoints;
checkpoints = calloc(sizeof(int), 2);

checkpoints[0] = 1;
checkpoints[1] = 2;

— realloc

void* realloc(void* pointer, size_t size)

If we allocated a block of memory but at a certain point of our program we need to change its size, we can call realloc().

We need to pass the memory block we want to change, and the new size (that can be bigger or smaller).

• The address that realloc() returns can be different from the one of the original memory block. Once we reallocate a block we need to point to the new address, otherwise the program would crash.

checkpoints = realloc(checkpoints, sizeof(int)*200);

— free

void free(void* memory)

When we are done using the memory, we can call free() to tell the program's memory that the specified block can be back to the operating system.

free(speed);
free(checkpoints);

If we don't call free() after using a particular memory block we will be making an unnecessary memory usage.

Getting the memory

We know that calling malloc() gets us memory to store dynamic values in the heap block, but what happens there is like a black box. Where does the memory come from?

malloc() calls a function named mmap() where the magic happens. mmap() requests memory from the kernel.

void mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

• *addr indicates where we want to allocate the memory. It can be NULL if we don't care where to store the values, otherwise we can specify a memory address (void*)0xFEEDB0000 and the system will try to satisfy the allocation.
• length determines the size of the memory block that we want to map. We can request sizes that aren't multiples of 4k, but the system is going to return 4k multiples either way, so we can set it to 4096 as a base and we are good to go. If we want to allocate more blocks we can multiply the initial value.

PAGESIZE 4096

• prot stands for protection and indicates which use do we want with the mapped memory. Common uses are PROT_READ and PROT_WRITE.
• flags tell the kernel how we want the memory to be managed. We can make memory available only for the ongoing process with MAP_PRIVATE, we can share the memory with external processes with MAP_SHARED, or use MAP_ANONYMOUS as common cases.
• fd stands for file descriptor and is used to access i/o resources. File descriptors are non-negative integers. If we don't want to store any file, we can pass a negative value -1.
• offset indicates where we want to start allocating memory so we can map only the parts we want.

We can un-map memory calling munmap():

int munmap(void *addr, size_t length);

— mmap() is useful when working with files since it allows us to handle them as memory buffers. A dedicated article on files will cover it.

Shared memory

Since mmap() allows having memory buffers, we can use them as shared memory in scenarios where we don't want to use pipes or signals and we want different processes to communicate each other.

When a program starts running, it becomes a process. A program may have multiple processes. We can identify each process by the id the system creates to differentiate them using the function getpid().

If we use the command-line with a tool like top, we can see all running processes and each one's ID.

*nix systems create processes using fork(), which clones a process creating a parent process and a child process.

int main()
{
  printf("A single process. ID: %d\n", getpid());
}

The example above will print the statement once.

int main()
{
  fork();
  printf("A single process. ID: %d\n", getpid());
}

If we call fork() in the main function we are cloning the process and we'll print twice the printf() call. We should see different values for each process ID.

— Usually we want to make multiple processes so we can have each one doing different things. Right now we have two processes but apart from the ID, it's not clear which one is the parent and which one is the child.

Luckily we know the returning values of each one:

• The parent returns the ID of the child.
• The child returns 0;
• On error, the parent returns -1 and no child process is created.

— At this point, changes made in either the parent or the child process are made locally so they can't see each other changes.

int nonShared = 4;
int main()
{
  //check if the process is the child
  if (fork() == 0)
    nonShared = 0;
  else
    //parent waits for child to complete before the next instruction
    wait(NULL); 

  printf("Parent not shared value: %d\n", nonShared);
  return 0;
}

If we want the parent and child processes to be able to communicate with each other, we can make use of mmap() to create a memory buffer that shares the information.

int nonShared = 4;
int main()
{
  uint8_t *sharedMem = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0);
  
  int pid = fork();

  //check if the process is the child
  if (pid == 0)
    *sharedMem = 1;
    nonShared = 0;
  else
    //parent waits for child to complete before the next instruction
    wait(NULL); 

  printf("not shared value: %d\n", nonShared);
  printf("shared value: %d\n", *sharedMem);
  return 0;
}

Now we can take some advantage on this and perform different operations for each process with shared values.

#include <stdio.h>
#include <unistd.h>   //fork()
#include <sys/mman.h> //mmap() 

int nonShared = 4;
int main()
{
  int *sharedMem = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0);

  int pid = fork();

  //check if the process is the child
  if (pid == 0)
    *sharedMem = nonShared + 2;
  else
    //parent waits for child to complete before the next instruction
    wait(NULL); 

  int result = *sharedMem / 2;
  if(pid != 0)
    printf("\nOperation result is: %d", result);
  return 0;
}

Eventually at a certain point in our development, we'll have a big single source file or we'll need to reuse code from a source file in another source. Header files are just source C files with a .h extension that have C code inside. They are designed to store function declarations and macros.

Header files are not mandatory but play a big game when sharing code between source files. They also help creating documentation and make code cleaner and more tidy.

Why we need header files

The way C is designed, requires the programmer to declare what functions he's going to use before defining them. This means that the compiler needs to know that there is some function Foo that takes parameters x and y before taking care of what Foo does inside.

There are two types of C header files:

• Built-in header files.
• Programmer-defined header files.

The first ones are provided by the C standard library, the GNU C library and similar.

The user-defined header files are the ones that we need to create manually and fill with our content.

— If we create a function CalcRadius that takes one float value for a circle's circumference and return the result, we have to declare it first:

//function declaration
float CalcRadius (float circumference);

//function definition
float CalcRadius (float circumference);
{
  float result;
  result = circumference / 2 * 3.14f;
  return result;
}

The function at the beginning is a declaration. It exists somewhere in the program but there's no memory allocated to it.

The detailed function below is the function definition.

— Header files describe what you can use from the outside module while the function definitions are stored in a source file with a .c extension.

When the compiler runs, it copies and pastes each header file included in a source file at the beginning of the code.

To implement a header in our calculation program we need to create two files:

• math.h
• math.c

Now we can cut and paste our function declaration inside math.h:

math.h

//function declaration
float CalcRadius (float circumference);

In order to link them together, inside the math.c file we need to add the math.h file at the beginning of the program, using the #include directive:

math.c

#include "math.h"

//function definition
float CalcRadius (float circumference)
{
  float result;
  result = circumference / 2 * 3.14f;
  return result;
}

This should be enough for the .c source file however, inside our .h file we have to perform some extra work in order to prevent some future errors that can happen when our program grows.

Header guards

We know when the compiler runs, it's going to copy-paste our header file in each #include directive. Since we can include the same header file in multiple .c source files we need a way to not copy-paste the same header multiple times.

— #ifdef's are pre-processor directives that are here needed to ensure the header file is only included once. Otherwise we would face an error similar to this when we run the compiler:

./math.h:4:7 error redefinition of CalcRadius function
./math.h:4:7 note previous definition is here

They work similar to if statements. The best way to protect our header for duplication is to name it with a unique identifier (usually its file name) and check if it's already defined or not. If not, then it's copied until the end of the condition.

Let's guard our header file:

math.h

#ifndef MATH_H /* This is our identifier */
#define MATH_H 

//function declaration
float CalcRadius (float circumference);

#endif /* MATH_H */

This way we can ensure that no matter how many times we need to include math.h in our program sources.

Compiler guards

Sometimes the C source code is compiled with a C++ compiler. This can be because of part of the program has been written in C++, or we have to use some modules that are created in C++. It also can be the case where we are including some C modules inside a C++ project.

— While C don't, C++ does name mangling due to function overloading. In C++ we can have two functions that have the same name with different arguments or return values, and the program can run without problems.

extern "C" {} ensures the compiler to treat the code inside the extern as C code.

math.h

#ifndef MATH_H 
#define MATH_H 

#ifdef __cplusplus
extern "C" {
#endif

//function declaration
float CalcRadius (float circumference);

#ifdef __cplusplus
}
#endif

#endif /* MATH_H */

Now we can work with the function CalcRadius in any other .c source file without worrying when our program grows.

Other types to include

We've seen how to declare functions in header files and define them later in source files. In header files we can also include structs and variables without giving them any values. In the .c source file we can access those variables and initialize them.

If we initialize variables with values in header files, the compiler is going to prompt an error when it runs.

math.h

...
float givenCircumference;
...

— There's an exception with declaring values in header files, which are constants. We know (in the example above) that 3.14~ is the PI value and that value never changes, it's a constant.

Let's add it into our header by defining the value:

math.h

#define PI 3.14159f
...
float givenCircumference;
...

math.c

...
result = circumference / 2 * PI;
...

Our final files should look like this:

math.h

#ifndef MATH_H /* This is our identifier */
#define MATH_H 

#define PI 3.14159f
float radius;
float circumference;


#ifdef __cplusplus
extern "C" {
#endif

//function declaration
float CalcRadius (float circumference);

#ifdef __cplusplus
}
#endif

#endif /* MATH_H */

math.c

#include "math.h"

//function definition
float CalcRadius (float circumference);
{
  float result;
  result = circumference / 2 * PI;
  return result;
}

Summing up

We can create a main.c file to perform the needed operations, and by including the math.h header, we should be able to access any function or value stored inside it, since the compiler is going to know where to find those values and functions when it runs.

main.c

#include <stdio.h>
#include "math.h"

int main()
{
  /* assign a value for our declared float */
  givenCircumference = 4.8f;

  /* execute our function */
  printf("Radius from circle is: %f \n", CalcRadius(givenCircumference);

  return 0;
}

Now you can start growing up your own math library and use it in every project that needs one (:

Programming in C is something that is often considered old-fashioned or complicated. The C programming language was created to be used in Unix systems as a high-level language. It's been almost fifty years since the creation of the Unix operating system, and the C programming language still plays a rock-solid role nowadays.

Programming software is not far away from programming operating systems and embedded devices. Not all software has to be mobile-first or web-ready.

Why C?

Apart from operating system kernels, code written in C can be found in database servers, embedded systems, cars, computational engines and real-time systems.

— C is an easy to learn programming language. Compared to other programming languages, it expects the programmer to take care of details like variable types or memory allocation but it's not that hard. We're working inside a machine that understands data that flows between memory directions, yet we're not asked to program in hex or binary code.

It's true that it takes longer than other high-level languages to get a finished program. But by programming in C you are going to learn a lot about how software communicates with the machine to achieve results.

What C offers

This are some -huge- advantages of developing software with C:

• C is simple (compared to C++, Java) yet powerful.
• No need to install any interpreter, virtual machine or library to start working with it.
• No need to install any specific IDE, a text editor is enough.
• Compilers are really easy to use and there is at least one C compiler for almost every existent architecture.
• You can bind scripting languages such as Lua or Python to your program.
• Direct access to memory. If done correctly, programs can be really memory efficient.
• Cross-platform is mostly ensured (depends if the software project in the making uses specific libraries like a GUI and which one we use in that case).

Wait, there is C++

C++ it's trendy, considered the go-to language to do heavy object oriented programming software projects.

— C++ is a superset of C, most of the people that program in C++ are using C programming features 99% of the time, and 1% using C++ features. Most software projects don't need object oriented programming.

— C is completely capable of making a good heavy-duty software, without over complicating methods. Take the Blender 3D authoring tool as an example.

What to expect

If you're new to programming in C, the basis can be learnt with the K&R C Programming language book which was written by the language creators.

Through these articles we're going to explore specific functionality and workflows in C that comes to a programmer's mind once the basis have been acquired. I'm not a PhD. nor a recognized teacher of any famous university, but I hope you can find the content useful and interesting (:

Here's an index for the content of the articles:

1. Working with headers
2. Working with memory
3. Working with pointers and arrays
4. Working with files
5. Working with files II
6. Threading basis
7. Minimal GUI applications

Working with languages like C/C++ requires running a process to compile our project. That process can look like a black box where magic happens, but it's not that complicated. Let's see how Makefiles work and how to write practical ones for your everyday hacks and projects.

You've maybe heard of Makefile generators like cmake. We're not using them here, neither a heavy IDE. The article is explained using a plain text editor and the command-line.

make is a tool used mostly to compile and build C/C++ source code. Makefiles just tell make how to compile and build a program. They consist in a series of instructions that perform automatically rather than having to manually type them in the command-line.

— Before we go further with make let's check what happens when we call the compiler so we know how to structure steps inside the Makefile later.

A simple compiling process takes four steps:

• The first step a compiler does is take our .c files and call the preprocessor, which handle the directives that start with a # like #include and #define and gets rid of the comments that may be present in our code.

At this point, all the code inside the header files that we have included using the directive #include "header.h" is copied and pasted in the program.

• The second step takes the source file and calls the compiler to translate C code into Assembly code, ending up with a file that has a .s extension.
• Once the compiler is done, it needs to translate the Assembly code into machine code, creating an object file, which is done via the assembler. The result file .o isn't an executable yet.
• The last step is bringing together all the object files to produce an executable. This part is done with the linker.

Flags

Each one of the steps needed to build a program can be invoked with a specific option to the compiler.

Flags give us the ability to enable or disable functionality for our building processes.

• -E calls the preprocessor only.
• -S runs the compiler and stops at the assembly file.
• -c is used to run the compiler up to the creation of an object file.
• -o generates the executable program from the object files.
• -g allows using gdb debugging.
• -Wall enables the compiler to print all warnings encountered while building the program.
• -I specifies a directory that contains prerequisites.

All the previous flags can be manually typed in a command-line environment:

$ cc -o demo_program main.c

but the idea here is to store those commands in a Makefile to automatically perform the build.

Basic structure

A Makefile (case sensitivity named) is a plain text file that can contain the following sections:

• Rules (that can be explicit or implicit)
• Macros (variable definitions)
• Comments

— Rules

A Makefile rule needs three basic items:

• A target, which is the name of the generated file.
• The dependencies needed to build the target.
• An action to realize in order to get the target.

Actions need to be indented by a tab character (not spaces) in order to work.

Note that we can have more than one action per target, and each one needs to be in a new line.

target: dependencies
    action

Let's pretend that we have a set of source files that we can compile into a program named calculator which depends on five independent source files.

calculator : main.c sum.h sub.h mult.h div.h
      cc -o calculator main.c

• Our target can be named as the program we want to create, so in this case is calculator.
• Our dependencies are five object files. Each of those files comes from it's own source.
• Our action is to execute the desired compiler, in this case cc to generate an output executable with the name calculator .

Compiled programming languages like C require us to recompile the program each time we change the source code. While the program keeps being simple there's no problem in rebuilding the whole program even if we only changed one file. But when we start to have a more complex program, compilation times increase, and recompiling everything just to update few changes is not effective.

The same way we create the target calculator we can make a target for each of the files that build it.

calculator: main.o sum.o sub.o mult.o div.o
      cc -o calculator main.o sum.o sub.o mult.o div.o

main.o: main.c main.h
      cc -c main.c
sum.o: sum.c sum.h
      cc -c sum.c
sub.o: sub.c sub.h
      cc -c sub.c
mult.o: mult.c mult.h
      cc -c main.c
div.o: div.c div.h
      cc -c div.c

This method forces us to write function declarations in separated .h header files and definitions in .c files to avoid multiple definitions. But we take the advantage of building only the objects that have modified dependencies.

make checks the timestamp of the files to keep track of modifications. If an object dependency gets a timestamp that is newer than the object's timestamp, it'll recompile that object when executed.

We can also create rules for steps that don't involve compiling or building the program, such as placing the built program in the correct directory, removing it (the same as uninstalling) or cleaning the compiled objects.

clean: 
      rm -f *.o calculator

Now instead of manually removing those files when we need a clean build, we can call make clean and the rule will perform the action.

To make an install rule we can follow the same procedure, just adding the binary as a dependency to the rule:

install: calculator
      mkdir  -p /opt/calc
      cp $< /opt/calc/calculator

and the uninstall rule is a simple recipe that removes the copied file:

uninstall:
      rm -f /opt/calc/calculator

The rules that don't involve compiling or building a program can get us in trouble if we ever meet the situation where an object is named like them (clean, install or uninstall in this case). To solve this, make has PHONY targets which are just a name for a recipe to be executed when you make an explicit request.

.PHONY: clean
clean: 
      rm -f *.o calculator

This way we avoid conflicts with other files.

— Macros

When programs start increasing the number of source files and library dependencies, the amount of objects and files to track increases. Luckily for us, make can handle this if we use macros (variables).

A macro has the following format:

name = data

where name is an identifier and data is the text that'll be substituted each time make sees ${name}.

Some predefined macros are:

• CC is used to store the name of the compiler which we want to use (cc, gcc, clang, etc).

CC = cc

• CFLAGS is used to list the flags we want the compiler to use.

CFLAGS = -c -g -Wall

• LDFLAGS is used to link libraries. Some header files like <math.h> are part of the system and aren't locally present in our code, but as any other header file, they contain just declarations and the compiler needs to check for the actual definitions somewhere.

LDFLAGS = -lm

Similarly we can make a macro for all our source files, dependencies and objects.

SRC = main.c sum.c sub.c mult.c div.c
OBJ = $(SRC:.c=.o)

We are storing all our source files in the SRC macro, and since the object files share names with the source files, we are transforming the content inside SRC by changing the .c suffix with .o and storing it in the OBJ macro.

Source files can be huge in number, and manually typing each source file name can end up being tedious and make the line hard to work with. We can take the advantage of wildcards:

SRC = $(wildcard *.c)

which will take every .c file inside the current directory.

Note that the value for SRC is encapsulated between brackets and includes the explicit wilcard word. If we just associate src to *.c it will store the literal set of characters and won't behave as expected.

Source files may happen to be in different directories. In that case we only need to repeat the wildcard process:

SRC = $(wildcard src/*.c) $(wildcard src/modules/*.c)

Macros don't need to be upper case, and can be used arbitrarily to simplify name repetitions like our program's name:

prog_name = calculator

Our Makefile can be transformed in something cleaner:

CC = cc
CFLAGS = -c -g -Wall
LDFLAGS = -lm
SRC = $(wildcard *.c)
OBJ = $(SRC:.c=.o)

prog_name = calculator

calculator: ${OBJ}
      ${CC} -o ${prog_name} ${OBJ} ${LDFLAGS}

main.o: main.c main.h
      ${CC} -c main.c
sum.o: sum.c sum.h
      ${CC} -c sum.c
sub.o: sub.c sub.h
      ${CC} -c sub.c
mult.o: mult.c mult.h
      ${CC} -c mult.c
div.o: div.c div.h
      ${CC} -c div.c

.PHONY: clean
clean: 
      rm -f *.o ${prog_name}

.PHONY: install
install: ${prog_name}
      mkdir  -p /opt/calc
      cp $< /opt/calc/calculator

.PHONY: uninstall
uninstall:
      rm -f /opt/calc/calculator

make can figure out that we want an object file from a source file as it has an implicit rule for updating an object .o file from a correspondingly named .c file using a cc -c command.

cc -c main.c -o main.o

The source .c file is automatically added to the dependencies, so we can reduce our rule:

main.o: main.c main.h
      ${CC} -c main.c

letting it appear as:

main.o: main.h 

Chances are that when building a program with make we get an error like this:

cannot find file "sum.h"

telling us that some required header isn't found. We can tell make where to look for prerequisites using the VPATH macro.

The value of VPATH specifies a list of directories that make should search expecting to find prerequisite files and rule targets that are not in the current directory.

VPATH = /inc /modules/inc

Note that VPATH will look through the directories list in the order we write them from left to right.

Another option to look for prerequisites is telling the compile where to look for them using the -I flag which indicates a directory where the requested code should be:

-I/src/inc

and should be included in the CFLAGS macro.

We can take our example and clean it with the new shown resources:

CC = cc
CFLAGS = -c -g -Wall -I/src/inc
LDFLAGS = -lm
SRC = $(wildcard *.c)
OBJ = $(SRC:.c=.o)

prog_name = calculator

calculator: ${OBJ}
      ${CC} -o ${prog_name} ${OBJ} ${LDFLAGS}

main.o: main.h
sum.o: sum.h
sub.o: sub.h
mult.o: mult.h
div.o: div.h

.PHONY: clean
clean: 
      rm -f *.o ${prog_name}

.PHONY: install
install: ${prog_name}
      mkdir  -p /opt/calc
      cp $< /opt/calc/calculator

.PHONY: uninstall
uninstall:
      rm -f /opt/calc/calculator

Now we only need to save the file and execute make calling the desired command. To build the calculator program it'd be:

$ make calculator

— Comments

Comments are pretty much self explanatory. They are lines of text that as in programming languages, do nothing but provide useful information or reminders.

We can place comments around our Makefile by using the hastag # symbol. Anything after a # will be ignored.

# An example comment

Summing up

In addition to compiling and building our own C/C++ code, working inside a BSD system involves being working close with its source code, and most of the times we have to compile and build packages from ports. That process works the same way so you can now start tweaking and inspecting source Makefiles each time you need to change or install a program. It'll grant you access to custom install instructions specific for your machine.

We can do more things with make like building install menus, compiling libraries and including Makefiles inside other Makefiles. All those topics need a dedicated article for each of them.

There's an official manual for GNU Make that you can read for advanced knowledge in the tool.

There are a lot of graphical interfaces to interact with Git with cozy buttons and windows. There's also a fast and powerful alternative: using Git directly from the command line.

There are different version control systems like Mercurial, Subversion or Bazaar. We are using Git in this guide.

What is Git?

Git is a powerful tool to maintain our code projects up to date and keep track of the changes we've made. Using it from the command-line is not that complicated as it could be seen.

To verify that we have git installed in our system, ask for it's version in the command line:

$ git --version

If the result is similar to git version 0.00.0 we are good to go. If not, just grab the package into your system:

$ doas pkg install git

— The first thing to perform after installing git is to set a username and an email address since git commit uses that information every single time.

$ git config --global user.name "Your Name"
$ git config --global user.email yourname@example.com

The --global flag is useful if we don't want to write the credentials each time we want to perform an action inside Git, since Git will always use that information for anything you do on that system.

In order to override global settings a different name or email address for specific projects, we can run the command without the --global option when working in that project.

— To check our actual settings we only need to ask git for them:

$ git config --list

Create a repository from a local folder

If we have a recently started project that starts growing up and we decide to upload it into a git hosting service we have two options:

— The first one is to create a repository in the hosting service, initialize it with a README.md, clone it in our local drive, and then move all our project inside the cloned repository. This is pretty much self explanatory.

— The second one is to tell git to get our actual project file and upload it into an empty repository hosted in our git service.

The second option is pretty easy to achieve:

• We need to create an empty repository in our git hosting service (github, gitlab, codeberg...) without initializing it.

Both names (the project directory name and the git repository name) have to match.

• In our local repository we have to initialize git.

$ git init

• After initializing git we have to add the content and commit our action

$ git add -A
$ git commit -m "commit message"

• Now we have to remotely add our git origin, which is our newly cloud created repository.

$ git remote add origin https://githost.com/username/repository.git

• The final step is pushing the content to the origin master branch.

$ git push -u origin master

We'll be asked for our git hosting credentials when pushing content.

Select what to upload

Chances are that we have come files inside our local project that we don't want to upload, like temporary files that the system creates or test builds that serve for debugging purposes.

— We can create a special file for git that allow us to specify which content to omit when pushing the project to the git service.

This file needs to be named .gitignore and is a good idea to create it in the top level directory of our repository.

Git uses globbing patterns to match against file names. We can construct our patterns using a set of symbols:

• ** A pattern with a double asterisk will match directories anywhere in the repository.

**/debug

• *. A patter with an asterisk will match zero or more characters anywhere in the repository.

*.o *.log

• ! Prepending an exclamation mark to a pattern negates it. A file will not be ignored if it matches a pattern, but also matches a negating pattern defined later in the file.

!important.log

Commit and Push our content

Once we've made changes locally to our code or project, we need to merge them into the hosted git repository.

First we need to tell git to add our changes.

$ git add -A

The following step is to commit the changes, usually with a comment.

$ git commit

If we want to make a one line comment we can add the flag -m after commit and write inside double quotes our message:

$ git commit -m "Updated foo.c -Changed boo function -Removed trash"

In the short run, we are most likely going to remember what we did in that commit. A lot of commit messages are similar to "update code" or "wrote function boo".

In the long run, you're going to love the time spent writing the commit messages with detail and common sense. Here's a short template of how can we structure a commit message:

Summarize the change in a few but meaningful words.

Additions:
- what you added

Fixes:
- what it fixes

Changes:
- what it changes


Longer explanation if needed goes here, along with additional notes, or relevant info.

When typing git commit without the -m flag, the shell will open the default text editor and will ask us to write the commit message.

Pull changes to our local folder

Every time we access the repository locally, we need to keep track of the cloud updates, so the work can flow seamlessly.

The first thing we have to do before start working inside the local repository, is to check for changes:

$ git status

if we have changes we can add them to our local repository via git pull:

$ git pull

Then we can start messing around.

— Things went nuts, the content inside the cloud repository had new changes but we were working locally without pulling them first!

Don't worry, there's a solution for that. You can stash (hide) your changes, pull and then apply your changes again:

$ git stash -u
$ git pull
$ git stash pop 

Copy a repository from the web

It's probably something you already know, but just for refreshing the memory, let's take a brief look at it.

When we want to get a repository from the web, we have an option to zip the entire repository and download it with just one click in the specified icon. This requires to manually unzip it later. But we are trying to use git from the command-line, and extra steps like unzip projects aren't part of the goal.

Every code repository has an https direction we can use to clone using git in the command-line in a very easy and quick way:

• First we need to create (or navigate to) a directory where we would like to store the git project.
• The second step is to copy that url and clone it via git clone:

$ git clone https://hosting.site/user/repo-name.git

Git hosting

Project tracking in git is great, but we need to keep our repository somewhere. One of the solutions is to create our own server and make our own repository with tools like Gogs.

— Another way is to go online and register in a git hosting service. The most popular out there is GitHub. Since Microsoft acquired it, is becoming a social hub where developers share code, follow each others, post updates and sponsor projects they like. There's nothing wrong going GitHub. Just be sure to read the terms & conditions carefully if you're concern about privacy, and don't forget to license your code.

Luckily there are alternatives to GitHub. All the following services provide options to store your code freely and the ability to decide whether your code is public or private.

• Gitea is based on Gogs. It's offered as a self-hosted service but you can use an already free hosted service at gitea.com
• Codeberg is great to store open source projects. It's based on Gitea.
• NotABug is based on Gogs and offers free code hosting for any project that is distributed under any free license.
• Gitlab is a commercial git service that offers enterprise ready tools and also a free plan where you can store your code and take advantage of a limited set of their tools.
• Bitbucket is another commercial git service aimed for teams and big projects. If you work solo or your project is less than five persons, you can opt for a free plan.

Summing up

Git is way more complex and powerful than just the content we read here. There's no point in copy-pasting complex workflows and custom needs in a guide that pretends to explain core common things and be useful in a hurry .

If you want to deep dive in Git, there is an official book available to read for free online in the official Git site.

Shell scripting covers almost every essential need to create automated command-line programs. But what about going beyond the standards and extending our arsenal with some external tools? Let's dive a bit inside awk and sed.

Write programs to handle text streams, because that is a universal interface. — Ken Thompson.

• Awk is a programming language that let us manipulate structured data.
• Sed is a stream editor to manipulate, filter, and transform text.

Both of them are stream-oriented; they read input from text files one line at a time and direct the result to the standard output, which means the input file itself is not changed if it's not specified to do so.

Although their syntax may look cryptic, awk and sed can solve a lot of complex tasks in a single line of code. Combining them with the use of regular expressions we have a Swiss army knife for anyone working with text files. Since we're working inside a *nix system this is perfect for us.

One of the most useful cases with awk and sed is parsing files and generating reports. It's a bit complicated to explain both tools without seeing them in action. To work through this post without searching too much for a file to parse, create a file named pieces-list and populate with some text inside:

Name= "Capacitor" ID= 3456 quant.= 204 Man.= "Bosch"
Name= "Battery" ID= 2760 quant.= 0 Man.= "Phillips"
Name= "Fan-Frame" ID= 7864 quant.= 131 Man.= "Mitsubishi"
Name= "Bluetooth-Emmiter" ID= 19085, quant.= 184 Man.= "Intel"
Name= "WiFi-Card", ID= 2941, quant.= 115, Man.= "Intel"
Name= "Fan" ID= 4512 quant.= 98 Man.= "OEM"

AWK

Awk is a full fledged programming language and a powerful file parser. It offers a more general computational model for processing a file, allowing us to replace an entire shell script with an awk single liner.

Awk programs consist of a series of rules. Rules generally consist of a pattern and a set of actions.

When a file is processed, awk reads the file line by line, then it checks to see if the line matches one or more of the patterns in the file and executes the actions associated to the matching pattern, taking the selected line as it's input.

If you've been reading the blog, you'll notice that we've used awk previously to configure our panel bar.

— The basic command-line syntax to invoke awk is:

$ awk [options] 'pattern {actions}' inputFile

We've seen how to get an output of a file before, using the cat command.

$ cat pieces-list

We've also seen how to split data to print only the parts we want using grep.

$ cat pieces-list | grep Intel

To start working with awk let's use it to print our pieces-list file:

$ awk '{ print }' pieces-list

We should have the same output result after running the program with both cat and the new awk method.

With awk we can use patterns too:

$ awk '/Intel/ { print }' pieces-list

patterns are declared between forward slashes.

This is useful but we still get a complete line containing the pattern we were looking for. One powerful feature of awk is that we can select pieces (named fields) of the line.

Named fields are represented with a dollar sign and the position number ($N).

$ awk '/Intel/ { print $2 }' pieces-list

Sometimes our pattern has to meet some conditions to be useful for us. We can use boolean statements to perform as patterns too:

In this example, the condition is that the sixth field has to be greater than one hundred:

$ awk '$6 > 1 { print $2 }' pieces-list

By default field separators are defined by spaces or tabs. If we want to use other pattern as a field separator we have to indicate so, changing the F variable:

-F=,

Awk allows us to use some internal functions to perform several actions.

• length() allows to get the number of characters for the specified named fields.

$ awk '{ print length($2) }' pieces-list

• printf formats the output of the specified named fields. We can align items both to the left and to the right using -% and % respectively.

$ awk '$6 > 1 { printf "%-19s", $2 }' pieces-list

— We can go further with awk and store all our commands inside a file so it's easier to apply the same line of commands for multiple files.

Awk command files can contain two special patterns:

• BEGIN{} is a pattern that is executed only once before running the main commands.
• END{} is a pattern that performs actions after all the instructions have been executed. It's only executed once.

Let's create a script to store our awk commands:

$ touch steps.awk

so now we can perform some awk examples into our pieces-list file.

— Format output

Our example pieces-list text is a bit messy. Wouldn't it be great to have each field ordered in nice columns?

First we need to define which character size our columns need. This value is given by our longest value in each field.

Using the builtin function length($N) we can get those values.

Let's define our main columns with the given values in our BEGIN pattern:

BEGIN{ printf "\n%-15s %-22s %-5s %9s\n", "MANUFACTURER ", "| PIECE NAME ","| ID ","|QUANTITY"}

In the main body we need a similar line for each one of the products in the list. This time we have to change our printed format in the fields that need to output a number:

{printf "%-16s %-22s %6d %9d\n", $8, $2, $4, $6}

In order to execute our stored awk commands, we simply need to indicate awk to read the file as follows:

$ awk -f steps.awk pieces-list

Our result should look similar to this:

MANUFACTURER   | PIECE NAME          | ID    | QUANTITY
-------------------------------------------------------

"Bosch"         "Capacitor"            3456         204
"Phillips"      "Battery"              2760           0
"Mitsubishi"    "Fan-Frame"            7864         131
"Intel"         "Bluetooth-Emmiter"   19085         184
"Intel"         "WiFi-Card"            2941         115
"OEM"           "Fan"                  4512          98

The same we used our messy example file we can use a web server ip traffic, username and password databases... endless possibilities to format.

— Process command-line arguments

We can take input from the user and pass it as a variable to perform actions with our data.

Let's say we want to ask the user for the product's ID and report them the manufacturer's name, the product's name and it available quantity.

Create a search.awk script we can perform the following instructions:

BEGIN{ print "Search results:\n" }

{if ( id == $4 ) print "Item ID " $4 "\n\t— Manufacturer: " $8 "\n\t— Piece Name: " $2, "\n\t—Stock Amount: " $6}

END{ print "\n---------------------------------\n"}

In this case we have created a variable named id to compare against our ID field. To make it work we should run the script addressing a value for the variable:

$ awk -v id=3456 -f search.awk pieces-list

Search results:

Item ID 3456
   — Manufacturer: "Bosch"
   — Piece Name: "Capacitor" 
   — Stock Amount: 204

---------------------------------

— Arithmetic and string operators

As in almost every programming programming language we can perform arithmetic operations inside awk passing named fields as values to operate with:

$ awk '{result += $6} END{printf "total amount of items: %d\n", result}' pieces-list 

total amount of items: 732

SED

Sed automates actions that seem a natural extension of interactive text editing. Most of these actions like replacing text, deleting lines, inserting new text, removing words... could be done manually from a text editor.

Automating all editing instructions inside one place and execute them in one pass can change hours of manual working in minutes of automated computing.

The command-line syntax to invoke sed is:

$ sed [options] instructions inputFile

If we run sed without any of these three parts we will have our file printed into our command line:

$ sed '' pieces-list

As you can see, the structure of calling sed is similar to calling awk.

This are a few instructions we can combine:

• / acts as a separator for numbers or patterns.

/patternA/patternB/

• s replace all the occurrences with a pattern.

s/orig/new/ 

We can indicate where to replace the matching pattern by adding the number of lines before the s character.

2s/orig/new/

This will replace orig with new in the second line of the file.

• g means everywhere.

s/orig/new/g

• w writes the contents of the pattern space into a file.

w /path/to/output_file

• d deletes a specified line Nd where N is the line number. This can be act just the opposite, deleting all non matching input by adding an exclamation point N!d.

1d inputfile
1!d inputfile

Running multiple commands with sed can be achieved by separating them inside the single quotes with semi colons ; in the command line, or by writing all the commands into a file with the extension .sed.

Let's use some sed power to work inside our pieces-list.

— Find and replace

$ sed 's/quant./Quantity/' pieces-list
$ sed 's/Man./Manufacturer/' pieces-list

This method will change any origPat match in worklist with newPat that occurs the first time on a line.

We can replace a pattern with an empty space too by leaving the new pattern blank:

$ sed 's/"//g' pieces-list

Now let's combine both instructions at once:

$ sed 's/quant./Quantity/'; 's/"//g' pieces-list

so our pieces-list looks like this:

Name= Capacitor ID= 3456 Quantity= 204 Manufacturer= Bosch
Name= Battery ID= 2760 Quantity= 0 Manufacturer= Phillips
Name= Fan-Frame ID= 7864 Quantity= 131 Manufacturer= Mitsubishi
Name= Bluetooth-Emmiter ID= 19085, Quantity= 184 Manufacturer= Intel
Name= WiFi-Card, ID= 2941, Quantity= 115, Manufacturer= Intel
Name= Fan ID= 4512 Quantity= 98 Manufacturer= OEM

— Extract and edit

Another powerful option that we have the ability to perform within sed is to extract information from a file, edit that information in memory and put the new edited data inside another file, without using pipelines.

Let's use a file for storing the instructions for sed.

$ vim extract.sed

We want to inspect a whole file, and we're not going to know the number of lines. We need to search from pattern one through to pattern two:

/Name=/,/Man.=/ 

so we work with the text contained between the start and the end pattern.

Working on that pattern space we can open a curly brackets section, just like a function so we can store the commands to execute in.

/Name=/,/Man.=/ {
s/"//g
s/.*Man.=//g
w manufacturer_list
}

Now we can run sed with this file to create our output file.

$ sed -f extract.sed pieces-list

Bosch
Phillips
Mitsubishi
Intel
Intel
OEM

Of course all of this can be scripted through pipelines but using just sed we've achieved the same in fewer lines and less time.

Combining Awk and Sed

We've seen that we can take the advantage of clean our text data with sed, and format it with awk. Let's go a step further and combine both powers to get a better report.

$ sed 's/"//g' pieces-list | awk -f steps.awk

This way we remove the double quotes from all names and get a clean result.

We can sort results taking any desired field as an index base. In this case we are going to use the Manufacturer's name to perform a sorted list at the items:

$ sed 's/"//g' pieces-list | awk '{ print $8 " " $0 }' | sort | awk -f steps.awk

We know what the sed line does. Let's analyze the awk one:

1. After the first pipe, we call awk to print the eighth value of the list with print $8.
2. Next, we add a blank space with " ". This acts as our separator. Since the file is using spaces, we keep the method.
3. Lastly we print the whole corresponding line so the next program in the pipe can read the information correctly.

Our result is going to be something weird. The formatted list maybe looks like this:

MANUFACTURER   | PIECE NAME          | ID    | QUANTITY
-------------------------------------------------------

Man.=           Name=                     0           0
Man.=           Name=                     0           0
Man.=           Name=                     0           0
Man.=           Name=                     0           0
Man.=           Name=                     0           0
Man.=           Name=                     0           0

---- End of report. Time: 06:44 | Date: 2020-04-01 ----

Since we are adding the eighth field as an index, we have increased the length of the lines and we need to increase the field to print inside our steps.awk file.

Having to track all this steps individually and in different files is not useful at all, that's why writing shell scripts for multiple tasks is so handy (yes, we can call sed and awk from within a shell script!).

— Create a script named format-report.sh and open it.

• Remember this is a Shell script so indicate it at the beginning of the file.

#!/bin/sh

• First we need to order our list based on the manufacturer's name.

awk '{print $8" " $0 }' $* | sort | 

• We have to add a header for our report using the BEGIN pattern from awk.

awk 'BEGIN{ printf "\n%-15s %-22s %-5s %9s\n", "MANUFACTURER ", "| PIECE NAME ","| ID ","| QUANTITY"
print "-------------------------------------------------------\n"}

• Next we execute the main loop of awk to print the formatted list.

{printf "%-18s %-23s %6d %9d\n", $9, $3, $5, $7}

• And we can add some condition to check if an item is out of stock.

{ if ($7 < 1) printf "\nWarning! Item %d is out of stock.(%s from %s)\n", $5, $3, $9}

• Once the main loop is done we can print a footer for our report using the END pattern, indicating time and date.

END {"date +'%Y-%m-%d'"|getline d; "date +'%H:%M'"|getline t; print "\n---- End of report. Time: " t " | Date: " d " ----"}' | 

• Lastly we call sed to get rid of the double quotes that the names inside the list have.

sed 's/"//g'

In order to run the script, save it, change its permissions to make it executable, and pass the pieces-list as the first argument:

$ ./format-report.sh pieces-list

We should see something similar to this:

MANUFACTURER   | PIECE NAME          | ID |    QUANTITY
-------------------------------------------------------

Bosch           Capacitor              3456         204
Intel           Bluetooth-Emmiter     19085         184
Intel           WiFi-Card              2941         115
Mitsubishi      Fan-Frame              7864         131
OEM             Fan                    4512          98
Phillips        Battery                2760           0

Warning! Item 2760 is out of stock.(Battery from Phillips)

---- End of report. Time: 07:40 | Date: 2020-04-01 ----

Summing up

A fundamental part of the power of *nix systems are pipes and the ability to use them to combine programs as building blocks in many ways to create automated workflows.

We've seen how to manage text data without touching a manual text editor in several ways, so now we can introduce these techniques using awk and sed to our pipe workflow with a new level of flexibility.