IPC payload does not arrive at receiver

Problem

I implemented an L4Re client/server application similar to the clntsrv example. However, when sending a message from client to server, the server receives something weird instead of the correct message payload.

Example code:

Server

 class MyServer : public L4::Server_object
 {
    public:
       int dispatch(l4_umword_t obj, L4::Ipc::Iostream& ios)
       {
          l4_umword_t val;
          l4_msgtag_t t;
          ios >> t;
          
          if (t.label() != MY_PROTOCOL_NUMBER) {
             return -L4_EBADPROTO;
          }
          
          notify_my_other_server();
          
          L4::Opcode op;
          ios >> op;
          
          switch(op) {
          case MY_OPCODE:
             ios >> val;
             printf("Value: %d\n", val);
             return L4_EOK;
          default:
             return -L4_ENOSYS;
          }
       }
 };

Client

 L4::Cap<void> server;
 
 int send(l4_umword_t value, l4_umword_t *result)
 {
    L4::Ipc::Iostream s(l4_utcb());
    s << MY_OPCODE << value;
    l4_msgtag_t res = s.call(server.cap(), MY_PROTOCOL_NUMBER);
    if (l4_ipc_error(res, l4_utcb())) {
       return -1;
    }
    s >> *result;
    return 0;
 }

Explanation

1. Whenever you do an IPC (or any other system call), your parameter(s) are transmitted through the current thread's user-level thread control block (UTCB).
2. For the case of IPC, the parameteres comprise the message payload. The kernel copies them from the sender's UTCB into the receiver's UTCB.
3. The IPC server framework (L4::Ipc::* namespace in the above example) is merely a set of C++ wrappers around packing and unpacking data into/from the UTCB and performing an invocation of a communication channel.

Now notice the server's call to notify_my_other_server() right within the code for unpacking the parameters from the L4::Ipc::Iostream. As the name suggests, this function leads to another IPC being sent to another server. This means, the current thread's UTCB will be overwritten with the new IPC call's parameters and hence, after returning from this function the UTCB's content is undefined. Therefore, you must not unpack data from the UTCB afterwards.

Lesson learned

• Do not perform any IPC or other system call within a server's dispatch() method until you are completely done reading the server call's parameters from the UTCB.
• If you need to print data, consider using outhex32() and outstring() from l4/sys/kdebug.h

Real-life issues

• In real life, the function you call will not have such an intuitive name as in the example above.
• Similar issues arise when writing rather low-level code, for instance for handling exceptions or page faults. These involve setting up a reply UTCB and you need to be careful not to perform a system call within this setup phase.