This new terminology will become more easily accessible via a hello world program:

/* hello_pthread.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <pthread.h>


void * hello_fun(void * args){

  printf("Hello World!\n");

  return NULL;
}

int main(int argc, char * argv[]){

  pthread_t thread;  //thread identifier

  //create a new thread have it run the function hello_fun
  pthread_create(&thread, NULL, hello_fun, NULL);

  //wait until the thread completes
  pthread_join(thread, NULL);

  return 0;
}

The way to follow this program is that the thread is created with pthread_create() and is set to run the function hello_fun(). The main thread, e.g., the original program, then attempts to join the new thread with the main thread with pthread_join(), which blocks until successful. In the meanwhile, the new thread running the function prints the all important "Hello World!" and returns, joining with the main thread and then exiting.

They key function to consider:

int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
                         void *(*start_routine) (void *), void *arg);

This creates a new thread in the calling process. The thread is identified by the type pthread_t, and can have a set of attributes. We will not use the attribute field for our threads. Following is a function pointer start_routine which takes a pointer argument and returns a pointer. This is the function that gets called when the program starts. The next argument arg is a reference to the argument to start_routine.

In the hello world example, we did not pass arguments, but let's look at a slightly more advanced example:

/* hello_args_pthread.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <pthread.h>


void * hello_arg(void * args){

  char * str = (char *) args;

  printf("%s", str);

  return NULL;
}

int main(int argc, char * argv[]){

  char hello[] = "Hello World!\n";

  pthread_t thread;  //thread identifier

  //create a new thread that runs hello_arg with argument hello
  pthread_create(&thread, NULL, hello_arg, hello);

  //wait until the thread completes
  pthread_join(thread, NULL);

  return 0;
}

This time we set up the function hello_arg to take the argument hello, a string containing the phrase "Hello World!". Note that in the startup routine, it must take a void * argument, so we have to cast

char * str = (char *) args;

but we know the argument was a string, so casting to a char * is safe here. The result of calling pthread_create() is that the new thread executes:

hello_arg(hello);

Just like with processes, it is often useful to be able to identify when a thread has completed or exited. The method for doing this is to join the thread, which is a lot like the wait() call for processes. Joining is a blocking operation, and the calling thread will not continue until the thread identified has changed states.

Typically, only the main thread calls join, but other threads can also join each other. All threads are automatically joined when the main thread terminates. For example the following code produces no output:

/* hello_pthread_bad.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <pthread.h>


void * hello_fun(){

  printf("Hello World!\n");

  return NULL;
}

int main(int argc, char * argv[]){

  pthread_t thread;

  pthread_create(&thread, NULL, hello_fun, NULL);

  return 0;
}

The program fails to join the new thread before the main thread terminated. As a result, the thread was automatically joined and did not have a chance to print "Hello World".

The fact that you do not have to join threads is actually an advantage because once the main thread dies the entire process dies. There are no zombies, no wasted resources. It all just comes to a hault.

If we look more closely at the join function we see that a thread can pass a return value, much like an exit status for processes, except it can be of any type instead of just an integer.

int pthread_join(pthread_t thread, void **retval);

The retval is a reference to a reference which will be set to the return value. Let's see how it's used in a hello world program:

/* hello_return_pthread.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <pthread.h>


void * hello_return(void * args){
  //allocate a new string on the heap with "Hello World!"
  char * hello = strdup("Hello World!\n");

  return (void *) hello;
}

int main(int argc, char * argv[]){

  char * str;

  pthread_t thread;  //thread identifier

  //create a new thread that runs hello_return without arugments
  pthread_create(&thread, NULL, hello_return, NULL);

  //wait until the thread completes, assign return value to str
  pthread_join(thread, (void **) &str);

  printf("%s", str);

  return 0;
}

The start up routine returned a void * which is really a reference to the string containing "Hello World!". The reference to the string is stored at str and printed to the string.

If you look more closely at this program, you start to notice that the fact that this is possible is why threads and process are so different. The string hello in the thread is allocated on the heap and the reference to that string is provided in the main program. That means the thread and the main thread are sharing resources, namely memory. While two processes can share some memory, it occurs naturally for threads and enables a lot of powerful programming paradigms — it also creates new challenges. What happens when two threads try and write to the same memory at the same time? We'll explore that in a following lesson.

To compile a program with POSIX thread, first you must include the header file:

#include <pthread.h>

This provides access to the underlying data types, like pthread_t, and function declarations. However, this is not enough because pthreads are not part of the standard C library. Instead, you must also link the pthreads library at compilation, much the same way we lined the readline library:

#> clang -lpthread hello_thread.c -o hello_thread

Where the -lpthread indicates to the compile to link the POSIX thread library.

When identifying a process, we use its process id, or pid. So far, we've seen POSIX threads identified by their pthread_t which is a transparent identifier used by the POSIX overlay of the clone() system call. While pthread_t identifiers are necessary for working with the pthread library they are not that human usable. Instead, we will often assign a thread a user level identifier, like a number – thread 1, thread 2, and etc.

Interestingly, a thread has a kernel level identifier much like a process id call the thread id. The traditional method for retrieving this is using the gettid() system call. Usually, this system call is not implemented in the standard C library, and instead we have to use the syscall() interface to gain access. Let's look at a sample program:

/* hello_id_pthread.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <unistd.h>
#include <sys/types.h>
#include <sys/syscall.h>

#include <pthread.h>

//have to call syscall directly, no libc wrapper
pid_t gettid(){
  return (pid_t) syscall (SYS_gettid);
}

void * hello_fun(void * args){

  //retrieve the thread_id
  pthread_t thread = pthread_self();

  //print all identifying information
  printf("THREAD: TID:%d PID:%d PthreadID:%lu\n", gettid(), getpid(), thread);

  return NULL;
}

int main(int argc, char * argv[]){

  pthread_t thread;  //thread identifier


  //create a new thread have it run the function hello_fun
  pthread_create(&thread, NULL, hello_fun, NULL);

  //print all identifying information
  printf("MAIN: TID:%d PID:%d \n", gettid(), getpid());

  //wait until the thread completes
  pthread_join(thread, NULL);

  return 0;
}

The output of this program is:

#> ./hello_id_pthread 
MAIN: TID:21301 PID:21301 
THREAD: TID:21302 PID:21301 PthreadID:140378868139776

You'll notice that a thread id is a LOT like a process id. In fact, for the main program, the thread id is process id. For the new thread, the thread id is different, but it retains the process id of the main program. This idea of process id's being like thread id's is why threads are schedule like normal programs.

Let's a look at an example program to start this conversation:

/* busy_thread.c*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <pthread.h>


void * hello_fun(void * args){

  while(1){
    //busy wait
  }

  return NULL;
}

int main(int argc, char * argv[]){

  pthread_t thread[4];  //thread identifier
  int i;

  //create 4 threads
  for(i = 0 ; i < 4; i++){
    pthread_create(&thread[i], NULL, hello_fun, NULL);
  }

  //wait for all threads to finish
  for(i = 0 ; i < 4; i++){
    pthread_join(thread[i], NULL);
  }

  return 0;
}

This program creates 4 threads, all of which busy wait, and the main thread waits for the rest of the threads to complete. The question is, how much resources does this use? Let's run this program to inspect its resources usage:

#> ./busy_pthreads &
[1] 21322
#> ps
  PID TTY          TIME CMD
18344 pts/5    00:00:00 bash
21322 pts/5    00:00:06 busy_pthreads
21327 pts/5    00:00:00 ps

If we just run the program in the background and look at the ps output, we see that the program is singular. No information about the threads is provided. But, let's look at the top output instead:

Tasks: 204 total,   1 running, 198 sleeping,   5 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7862628k total,  4490728k used,  3371900k free,   194504k buffers
Swap: 12753916k total,       12k used, 12753904k free,  3641808k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                     
21322 aviv      20   0 39240  380  296 S  394  0.0   4:06.83 busy_pthreads
    1 root      20   0 24604 2548 1352 S    0  0.0   0:01.81 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.19 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:08.67 ksoftirqd/0       
(...)

Look at the column for %CPU. You're reading that right 394%!!! That's because each of the threads is scheduled individually and is using resources like a process. The machine running the program is multi-core, so each thread can run at the same time, and thus, one program is using all 4 cores at 100%.

This might seem nuts, but we can see this a bit better when we expand the program into its constituent threads.

#> ps -L
  PID   LWP TTY          TIME CMD
18344 18344 pts/5    00:00:00 bash
21322 21322 pts/5    00:00:00 busy_pthreads
21322 21323 pts/5    00:03:50 busy_pthreads
21322 21324 pts/5    00:03:50 busy_pthreads
21322 21325 pts/5    00:03:50 busy_pthreads
21322 21326 pts/5    00:03:50 busy_pthreads
21333 21333 pts/5    00:00:00 ps

The -L option for ps will organize by thread id (LWP) and process id (PID), and now you can start to see that the program is actually running as 5, the 4 threads and the 1 more for the main program. If we look at the top output using the H option (hit H when top is up).

Cpu(s):  0.1%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7862628k total,  4490620k used,  3372008k free,   194552k buffers
Swap: 12753916k total,       12k used, 12753904k free,  3641820k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                     
21323 aviv      20   0 39240  380  296 R  100  0.0   6:34.12 busy_pthreads
21324 aviv      20   0 39240  380  296 R  100  0.0   6:34.13 busy_pthreads
21325 aviv      20   0 39240  380  296 R  100  0.0   6:34.10 busy_pthreads
21326 aviv      20   0 39240  380  296 R  100  0.0   6:34.10 busy_pthreads
21322 aviv      20   0 39240  380  296 S    0  0.0   0:00.00 busy_pthreads

We can see that each of the reads is using 100% of single core and the main thread is blocking (S) waiting to join each of the other threads. top even goes so far as using the tid as the pid for each thread.

Before we get into the details of programming with locks. Let's look at an example of why we need locks. Consider the program below which is simulating a bank. The user selects the number of threads from the command line and then each thread will manipulate an account balance by randomly choosing an amount to credit or debit the account while tracking the total number of debits and credits.

/* bank.c*/
int balance = INIT_BALANCE;


int credits = 0;
int debits = 0;

void * transactions(void * args){
  int i,v;

  for(i=0;i<NUM_TRANS;i++){

    //choose a random value
    v = (int) random() % NUM_TRANS;

    //randomnly choose to credit or debit
    if( random()% 2){
      //credit

      balance = balance + v;
      credits = credits + v;

    }else{
      //debit

      balance = balance - v;
      debits = debits + v;

    }

  }

  return 0;
}

int main(int argc, char * argv[]){

  int n_threads,i;
  pthread_t * threads;

  //error check
  if(argc < 2){
    fprintf(stderr, "ERROR: Require number of threads\n");
    exit(1);
  }

  //convert string to int
  n_threads = atol(argv[1]);

  //error check
  if(n_threads <= 0){
    fprintf(stderr, "ERROR: Invalivd value for number of threads\n");
    exit(1);
  }

  //allocate array of thread identifiers
  threads = calloc(n_threads, sizeof(pthread_t));

  //start all threads
  for(i=0;i<n_threads;i++){
    pthread_create(&threads[i], NULL, transactions, NULL);
  }

  //join all threads
  for(i=0;i<n_threads;i++){
    pthread_join(threads[i], NULL);
  }

  printf("\tCredits:\t%d\n", credits);
  printf("\t Debits:\t%d\n\n", debits);
  printf("%d+%d-%d=   \t%d\n", INIT_BALANCE,credits,debits,
         INIT_BALANCE+credits-debits);
  printf("\t Balance:\t%d\n", balance);

  //free array
  free(threads);

  return 0;
}

Once the program completes, it prints out the total amount credited and debited and the current balance. The program also prints out what the balance should be if we consider credits, debits and initial balance. Here is two runs of the program with 2 worker threads:

#> ./bank 2
	Credits:	4707
	 Debits:	4779

1000+4707-4779=   	928
	 Balance:	942

#> ./bank 2
	Credits:	4868
	 Debits:	4465

1000+4868-4465=   	1403
	 Balance:	1587

What happened? In neither case did the balance at the end match the expectation based on the total debits and credits. How could this be?

Each thread is scheduled independently and can be interrupted and preempted arbitrarily by the OS. This can even occur in the middle of an operation. Consider just the simple bit of code to add numbers:

balance = balance + v

To assign the final balance, two things must happen:

1. balance + v must be computed and stored in a temp variable
2. The temp variable must be assigned to balance

What happens if the thread did step (1) but was preempted by the other thread before completing step (2). In the meanwhile, the other thread completes (1) and (2), and finally the original thread is scheduled back to do step (2). Consider the diagram below:

      Thread 1                 Thread 2        balance     v     tmp

(1) bal + v -> tmp                               1000      5     1005

                           (1) bal + v -> tmp    1000      3     1003
                           (2) bal <- tmp        1003
(2) bal <- tmp                                   1005            1005

Since the incremental work of Thread 1 was assigned after the work of Thread 2, it is if Thread 2 never did the work in the first place. This is why we get such inconsistent results with the balance, and further, it's more than just the balance value, it not clear that even credit or debit counters are consisten either.

To solve this problem, we have to identify the critical sections of the program, that is, sections of code that only one thread can execute at a time. Once a critical section is identified, we use a shared variables to lock that section. Only one thread can hold the lock, so only one thread executes the critical section at the time.

While it may simple at first creating a lock is no small task. The locking procedure must be an atomic operation where the whole process of testing if the lock is available and acquiring the lock occur singularly. To demonstrate why this must be the case, consider the following code where we implement a naive lock, which is just a variable integer, either set to 0 for available or 1 for locked.

char USAGE[] = "naive_lock n_threads\n"
  "USAGE: run n threads with a naive lock\n";

int lock = 0; //0 for unlocked, 1 for locked

int shared = 0; //shared variable

void * incrementer(void * args){
  int i;

  for(i=0;i<100;i++){

    //check lock
    while(lock > 0);  //spin until unlocked

    lock = 1; //set lock

    shared++; //increment

    lock = 0; //unlock
  }

  return NULL;
}


int main(int argc, char * argv[]){
  pthread_t * threads;
  int n,i;

  if(argc < 2){
    fprintf(stderr, "ERROR: Invalid number of threads\n");
    exit(1);
  }

  //convert argv[1] to a long
  if((n = atol(argv[1])) == 0){
    fprintf(stderr, "ERROR: Invalid number of threads\n");
    exit(1);
  }

  //allocate array of pthread_t identifiers
  threads = calloc(n,sizeof(pthread_t));

  //create n threads
  for(i=0;i<n;i++){
    pthread_create(&threads[i], NULL, incrementer, NULL);
  }

  //join all threads
  for(i=0;i<n;i++){
    pthread_join(threads[i], NULL);
  }

  //print shared value and result
  printf("Shared: %d\n",shared);
  printf("Expect: %d\n",n*100);

  return 0;
}

If you follow the code, you see that the variable lock is set to 0 for unlocked and 1 for locked. Then each thread will spin until the variable is set to 0 for unlocked, that is while(lock > 0);

If we run this program a few times with 2 threads, it might seem to work …

#> ./naive_lock 2
Shared: 200
Expect: 200
#> ./naive_lock 2
Shared: 200
Expect: 200
#> ./naive_lock 2
Shared: 200
Expect: 200
#> ./naive_lock 2
Shared: 200
Expect: 200
#> ./naive_lock 2
Shared: 200
Expect: 200
#> ./naive_lock 2
Shared: 172
Expect: 200

… until all of a sudden it does not work. What happened?

The problem is the testing of the lock and the acquisition of the lock are not atomic operations. If we look at that sequence of code:

//check lock
while(lock > 0);  //spin until unlocked
                                         <----------- What if threads swap here?
lock = 1; //set lock                                  Then two (or more) threads could have the lock.

What we need is a specialized operation for testing and setting that all happens at once, and fortunately, this is provided to us by pthread library.

The term mutex stands for a mutually exclusion, which is a fancy name for lock. A mutex is not a standard variable, instead, it is guaranteed to be atomic in operation. The act of acquiring a lock cannot be interrupted.

In the pthread library the type of a mutex is:

pthread_mutext_t mutex;

You must first initialize a mutex before you can use it:

 pthread_mutex_init(&mutex, NULL);
 //                          ^
//                      will not use second arg of init

You then can acquire and unlock a mutex:

pthread_mutex_lock(&mutex);

/* critical section */

pthread_mutex_unlock(&mutex);

Both of these operations are atomic, and the locking function is blocking. That is, once a thread tries to acquire a lock, it will be suspended execution until the lock is available. Further, locks are ordered in their acquisition; the order in which threads try to acquire the lock determine which thread gets the lock next. How that works is a discussion for your operating systems class, and more generally, the locking functionality implantation details are heavily supported by the operating system and the hardware.

Finally, creating a mutex allocates memory. So we have to deallocate the mutex, or destroy it:

pthread_mutex_destroy(&mutex);

One locking strategy for locking a program is to use a single lock for the critical section to protect the entire critical section. This is called coarse locking because it is a large granularity. In code, this might look like this for the critical section:

/*bank_coarse_lock.c*/
int balance = INIT_BALANCE;


int credits = 0;
int debits = 0;

pthread_mutex_t lock;

void * transactions(void * args){
  int i,v;

  for(i=0;i<NUM_TRANS;i++){

    pthread_mutex_lock(&lock); //acquire lock

    //choose a random value
    v = (int) random() % NUM_TRANS;

    //randomnly choose to credit or debit
    if( random()% 2){
      //credit

      balance = balance + v;
      credits = credits + v;

    }else{
      //debit

      balance = balance - v;
      debits = debits + v;

    }

    pthread_mutex_unlock(&lock); //release lock
  }

  return 0;
}

With this locking strategy, consistency is achieved. All the critical code executes atomically and the output is consistent:

#> ./bank_coarse_lock 3
	Credits:	7247
	 Debits:	7462

1000+7247-7462=   	785
	 Balance:	785

While coarse locking is a reasonable choice, it is inefficient. We looks some paralleism because not all parts of the critical section are critical to each other. For example, consider that the variable credits and debits are used exclusively of each other; each thread only performs a credit or debit but not both. Maybe it would be worth while to do more fine grain locking.

Consider the following changes:

int balance = INIT_BALANCE;


int credits = 0;
int debits = 0;

pthread_mutex_t b_lock,c_lock,d_lock;

void * transactions(void * args){
  int i,v;

  for(i=0;i<NUM_TRANS;i++){

    //choose a random value
    v = (int) random() % NUM_TRANS;

    //randomnly choose to credit or debit
    if( random()% 2){
      //credit

      pthread_mutex_lock(&b_lock);
      balance = balance + v;
      pthread_mutex_unlock(&b_lock);


      pthread_mutex_lock(&c_lock);
      credits = credits + v;
      pthread_mutex_unlock(&c_lock);

    }else{
      //debit

      pthread_mutex_lock(&b_lock);
      balance = balance - v;
      pthread_mutex_unlock(&b_lock);

      pthread_mutex_lock(&d_lock);
      debits = debits + v;
      pthread_mutex_unlock(&d_lock);

    }
  }

  return 0;
}

In this example, we use fine locking because each of the shared variables are independently locked only when used. This allows for more parallelism at the cost of complexity. If we compare the run times of the two strategies we see that, yes, fine grain locking is marginally faster and much faster on larger load sets.

One consequence of fine grain locking is the potential for deadlocks. A deadlock occurs when two threads each hold a resource the other is waiting on. Let's look at a naive example:

#include <stdio.h>
#include <stdlib.h>

#include <pthread.h>

pthread_mutex_t lock_a, lock_b;

void * fun_1(void * arg){
  int i;
  for (i = 0 ; i< 10000 ; i++){

    pthread_mutex_lock(&lock_a); //lock a then b
    pthread_mutex_lock(&lock_b);

    //CRITICAL SECTION

    pthread_mutex_unlock(&lock_a);
    pthread_mutex_unlock(&lock_b);

  }

  return NULL;
}

void * fun_2(void * arg){
  int i;
  for (i = 0 ; i< 10000 ; i++){

    pthread_mutex_lock(&lock_b); //lock b then a
    pthread_mutex_lock(&lock_a);

    //CRITICAL SECTION

    pthread_mutex_unlock(&lock_b);
    pthread_mutex_unlock(&lock_a);

  }

  return NULL;
}

int main(){

  pthread_t thread_1,thread_2;

  pthread_mutex_init(&lock_a, NULL);
  pthread_mutex_init(&lock_b, NULL);

  pthread_create(&thread_1, NULL, fun_1, NULL);
  pthread_create(&thread_2, NULL, fun_2, NULL);

  pthread_join(thread_1, NULL);
  pthread_join(thread_2, NULL);

  return 0;
}

In this code we have two threads and two locks. One thread acquires lock_a and then lock_b while the other thread acquires lock_b and then lock_a. If we run this program we find that nothing happens; the program hangs. This is because of a deadlock.

Consider what happens when one each thread run such that one holds lock_a and the other holds lock_b, then one thread is waiting on the lock the other thread holds. Both threads are now blocking on acquiring the resources, and the main thread is blocking trying to join the worker threads. Nothing is running: deadlock.

By adding a lock to each account, we now require a worker thread to acquire the account lock on both accounts before executing a transfer:

//create an account
struct account{
  int balance;
  int credits;
  int debits;
  pthread_mutex_t lock; //lock for the account
};

//array of accounts
struct account accounts[NUM_ACCOUNTS];


void * transactions(void * args){
  int i,v;
  int a1,a2;

  for(i=0;i<NUM_TRANS;i++){

    //choose a random value
    v = (int) random() % NUM_TRANS;

    //choose two random accounts
    a1 = (int) random() % NUM_ACCOUNTS;
    while((a2 = (int) random() % NUM_ACCOUNTS) == a1);

    //transfer from a1 to a2
    //    printf("Transfer %d->%d : %d \n", a1,a2,v);

    //acquire lock for a1 then a2
    pthread_mutex_lock(&accounts[a1].lock);
    pthread_mutex_lock(&accounts[a2].lock);


    accounts[a1].balance += v;
    accounts[a1].credits  += v;

    accounts[a2].balance -= v;
    accounts[a2].debits += v;

    pthread_mutex_unlock(&accounts[a1].lock);    
    pthread_mutex_unlock(&accounts[a2].lock);

  }

  return 0;
}

This might seem to work, but this locking strategy actually will form a deadlock. Why? The locking strategy is to lock a1 then a2 always in that order, how could a deadlock occur?

Consider that the accounts are chosen randomly from all accounts and that there are multiple threads running. The variables a1 and a2 are standin's for account numbers or indexes into the array of accounts. What if in two threads the following happens?

Thread 1        Thread 2
---------      ----------
a1 = 5;          a1 = 3;
a2 = 3;          a2 = 5;

Then when we look at lock ordering, we find the following situation and deadlock might occur:

Thread 1        Thread 2
---------      ----------
lock( 5 )       
                lock( 3 )
lock( 3 )       
                lock( 5 )


        DEADLOCK

We find ourselves back where we started, each thread is now waiting on a lock held by another thread. Progress cannot be made and a deadlock occurs. This happens because we did not apply a proper order to locking: How do we apply a proper ordering?

One way to apply a proper ordering is to leverage the fact that the account indexes are already ordered. What if we always lock accounts in account order? Then in above example, both threads would do lock(3) then lock(5). In code, this would look like this:

void * transactions(void * args){
  int i,v;
  int a1,a2;

  for(i=0;i<NUM_TRANS;i++){

    //choose a random value
    v = (int) random() % NUM_TRANS;

    //choose two random accounts
    a1 = (int) random() % NUM_ACCOUNTS;
    while((a2 = (int) random() % NUM_ACCOUNTS) == a1);

    //transfer from a1 to a2

    if(a1 < a2){
      pthread_mutex_lock(&accounts[a1].lock);
      pthread_mutex_lock(&accounts[a2].lock);
    }else{
      pthread_mutex_lock(&accounts[a2].lock);
      pthread_mutex_lock(&accounts[a1].lock);
    }

    accounts[a1].balance += v;
    accounts[a1].credits  += v;

    accounts[a2].balance -= v;
    accounts[a2].debits += v;

    if(a1 < a2){
      pthread_mutex_unlock(&accounts[a2].lock);
      pthread_mutex_unlock(&accounts[a1].lock);
    }else{
      pthread_mutex_unlock(&accounts[a1].lock);
      pthread_mutex_unlock(&accounts[a2].lock);
    }
  }

  return 0;
}

Now, everything is consistent and deadlock is avoided.