Semaphores on Linux - sem_init() vs sem_open()

3 minute read

"Semaphore" Regular readers will know that I’m working on a Linux server daemon that, amongst other things, moves data back and forth between sockets and files without it appearing in user space, and even ‘tees’ that data to a second destination, again without a copy to a user space buffer. Now I have multiple instances of my server running, and they need to synchronize access to shared data structures. The standard mechanism for this is the semaphore. I won’t get into a deep discussion of semaphores here, the Wikipedia article linked in the preceding sentence gives a good description. Basically, if you want to ensure that no more than one thread (ok, ‘n’ threads in the general case) has access to some resource concurrently, you use a semaphore. Looking for an example of semaphores on Linux, I found the aptly named Semaphores in Linux, by Vikram Shukla, on the O’Reilly Linux DevCenter. This is a very useful article, explaining the general semaphore concept and comparing the System V and POSIX semaphore implementations. Guided by the article, in particular, the ‘Related Process’ example, which closely matched my use case, I wrote a quick test program using the POSIX sem_init() call to initialize a semaphore and sem_wait()/sem_post() to decrement/increment the semaphore respectively. Only one problem. It didn’t work - my processes had concurrent access to the shared resource! Going back to Vikram’s example, and reading the sem_init() man page very carefully, the issue seems to be that the semaphore is created on the stack of the parent process. When the child is forked, it gets a copy of the semaphore, not a reference to the parent’s semaphore. Adding a few sleep()’s and printf()’s to the example highlights the problem:

#include <semaphore.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>

#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

int main(int argc, char **argv)
{
  int fd, i,count=0,nloop=10,zero=0,*ptr;
  sem_t mutex;

  //open a file and map it into memory

  fd = open("log.txt",O_RDWR|O_CREAT,S_IRWXU);
  write(fd,&zero,sizeof(int));
  ptr = mmap(NULL,sizeof(int),PROT_READ |PROT_WRITE,MAP_SHARED,fd,0);
  close(fd);

  /* create, initialize semaphore */
  if( sem_init(&mutex,1,1) < 0)
    {
      perror("semaphore initilization");
      exit(0);
    }
  if (fork() == 0) { /* child process*/
    for (i = 0; i < nloop; i++) {
      sem_wait(&mutex);
      printf("child entered crititical section: %d\n", (*ptr)++);
      sleep(2);
      printf("child leaving critical section\n");
      sem_post(&mutex);
      sleep(1);
    }
    exit(0);
  }
  /* back to parent process */
  for (i = 0; i < nloop; i++) {
    sem_wait(&mutex);
    printf("parent entered critical section: %d\n", (*ptr)++);
    sleep(2);
    printf("parent leaving critical section\n");
    sem_post(&mutex);
    sleep(1);
  }
  exit(0);
}

Running this shows that both the parent and the child are in the critical section at the same time:

child entered critical section: 0
parent entered critical section: 1
parent leaving critical section
child leaving critical section
parent entered critical section: 2
child entered critical section: 3
...

The explanation is in the sem_init() man page:

If pshared is non-zero, then the semaphore is shared between processes, and should be located in a region of shared memory (see shm_open(3), mmap(2), and shmget(2)). (Since a child created by fork(2) inherits its parent’s memory mappings, it can also access the semaphore.) Any process that can access the shared memory region can operate on the semaphore using sem_post(3), sem_wait(3), etc.

The key here is that the semaphore must be in a region of shared memory, even if you’re accessing it from related processes such as a parent and its child. There are two ways of fixing the problem. The first is to use shm_open(), ftruncate() and mmap() to create a shared memory region and obtain a pointer to it:

  int shm;
  sem_t * mutex;

  ...

  if ((shm = shm_open("myshm", O_RDWR | O_CREAT, S_IRWXU))   0) {
    perror("shm_open");
    exit(1);
  }

  if ( ftruncate(shm, sizeof(sem_t)) < 0 ) {
    perror("ftruncate");
    exit(1);
  }

  if ((mutex = mmap(NULL, sizeof(sem_t), PROT_READ | PROT_WRITE,
      MAP_SHARED, shm, 0)) == MAP_FAILED) {
    perror("mmap");
    exit(1);
  }

  if (sem_init(mutex, 1, 1) < 0) {
    perror("semaphore initialization");
    exit(1);
  }

  ...

The other, simpler, solution is to just use sem_open(), which Vikram describes in the next section of the article:

  if ((mutex = sem_open("mysemaphore", O_CREAT, 0644, 1)) == SEM_FAILED) {
    perror("semaphore initilization");
    exit(1);
  }

Either of these approaches gives the desired result:

child entered crit section: 0
child leaving crit section
parent entered crit section: 1
parent leaving crit section
child entered crit section: 2
child leaving crit section
parent entered crit section: 3
...

Postscript: this is a minor flaw in an otherwise excellent and very useful article. I address it here, rather than in a comment on the article, due to the amount of space required for a full explanation.

Updated:

Comments

Andrea

Nice post, thanks.

Another way is to create a region of anonymous memory and set it as shared via mmap():

sem_t *mutex = mmap(NULL, sizeof(sem_t), PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, 0); if (!mutex) { perror(“out of memory\n”); exit(1); }

Nisheeth

One question is ..even if you share the memory and get the shm descriptor for shared memory how will you pass on to other unrelated processes (processes which are not related by parent child relationship)…

What I think of is message queues of System IPC would work…correct me if i am wrong…

Pat Patterson

Hi Nisheeth - shm_open takes a name for the shared memory region, so unrelated processes just need to know the same name. Yes - I think System V IPC message queues can do all this, they just looked quite a bit more complex to deal with when I was working on this.

Nilesh

Thanks for this explanation. I tried using sem_open this way:

if ((mutex = sem_open(“mysemaphore”, O_CREAT, 0644, 0)) == SEM_FAILED) { cerr << "semaphore initilization error" << endl; return 1; }

pid_t pid = fork();

if (0 == pid) { /* if ((mutex = sem_open("mysemaphore", 0, 0644, 0)) == SEM_FAILED) { cerr << "semaphore initilization error" << endl; return 1; } */ cout << "here1" < 0) { sem_wait(mutex); cout << "here2" << endl; wait(&retval); }

The output is: here2 here1

that's not correct. Am I missing something?

Pat Patterson

Hi Nilesh, it looks like your code is missing a couple of lines, but I think I understand your question. The answer is that the order in which the two processes enter the critical section is not guaranteed; only the fact that they won’t be there at the same time. What are you trying to do?

BTW - it’s better to create a gist and post the URL here - then your code won’t get eaten by WordPress - https://gist.github.com/

zac

Dear Pat,thank u 4 that wonderful post. Cud you plz suggest something 4 thhis problem i am facing: I am testing a custom inter-process semaphore implementation where semaphore is called as a function, ie, one function to create sem,another function to take sem and so on. I am forking a process and i want the sem to be created in the shared memory. I cannot pass the sem_t *sem value to the custom function(it has only one argument,which is semaphore name). What can i do to make this function run in the shared memory space?Or how can i get around this problem?

Pat Patterson

abhishek - sem_open is mentioned towards the bottom of the blog entry as a simpler alternative to sem_init. See also Vikram’s original article: http://linuxdevcenter.com/pub/a/linux/2007/05/24/semaphores-in-linux.html?page=4

Shashank

Hi Pat, My semaphores are not accessable by other processes. Below is the snippet

void initStudentStuff(void) {

CHK(fdnc = open("numCountFile",O_RDWR|O_CREAT|O_TRUNC,S_IRUSR|S_IWUSR));
CHK(fdlc = open("lineCountFile",O_RDWR|O_CREAT|O_TRUNC,S_IRUSR|S_IWUSR));

CHK(lseek(fdnc,0,SEEK_SET));
assert(sizeof(numCount) == write(fdnc,&amp;numCount,sizeof(numCount)));
CHK(lseek(fdlc,0,SEEK_SET));
assert(sizeof(lineCount) == write(fdlc,&amp;lineCount,sizeof(lineCount)));

sprintf(semName,"/%s%ldsem",COURSEID,(long)getuid());
sem = sem_open(semName,O_RDWR|O_CREAT|S_IRUSR|S_IWUSR,1);

}

/* In this braindamaged version of placeWidget, the widget builders don’t coordinate at all, and merely print a random pattern. You should replace this code with something that fully follows the p3 specification. */ void placeWidget(int n) {
CHK(sem_wait(sem));

CHK(lseek(fdnc,0,SEEK_SET));
assert(sizeof(numCount) == read(fdnc,&amp;numCount,sizeof(numCount)));
CHK(lseek(fdlc,0,SEEK_SET));
assert(sizeof(lineCount) == read(fdlc,&amp;lineCount,sizeof(lineCount)));
  
printeger(n);    // printf ("for pid %d line count = %d , num count = %d\n", n, lineCount, numCount);
if (lineCount == numCount)
{
    ++lineCount;
    numCount = 0;
    printf("N\n");
    fflush(stdout);
}

++numCount;

CHK(lseek(fdnc,0,SEEK_SET));
assert(sizeof(numCount) == write(fdnc,&amp;numCount,sizeof(numCount)));
CHK(lseek(fdlc,0,SEEK_SET));
assert(sizeof(lineCount) == write(fdlc,&amp;lineCount,sizeof(lineCount)));

CHK(sem_post(sem)); }

Leave a Comment

Your email address will not be published. Required fields are marked *

Markdown is supported.

You May Also Enjoy