User Tools

Site Tools


ipv6_changes_to_iotvity

IPv6 additions to IoTivity

John Light and Juliet Cai Intel Opensource Technology Center, OIC Development 3/14/2015

The first step in adding IPv6 to IoTivity is to provide a complete implementation for a subset of the code. Specifically:

  • ethernet adapter
  • non-DTLS
  • multi-threaded
  • Presence is not handled
  • No interface binding

The current code has been demonstrated to perform IPv6 communication between and OIC client and server in those narrow circumstances. Further work will involve generalizing those four aspects. Here is a summary of the contents of this document.

  1. Assumptions about how IoTivity should handle IPv6.
  2. Approach to adding IPv6.
  3. Types of changes made in ca-ipv6 branch.
    1. changes to APIs.
    2. changes to implement IPv6 capability.
    3. changes to make IoTivity work properly.
    4. changes to make the code more elegant.
  4. Descriptions of major changes.
    1. Changes to API.
      1. C++ API
      2. csdk API
      3. CA API
    2. Changes to ethernet adapter.
    3. Changes to connectivity abstraction.
    4. Changes to csdk.
    5. Changes to C++ sdk.
  5. What works at this point

IPv6 tidbits

An IPv6 address is 128 bits long, usually displayed as 8 16-bit hexadecimal items. So my IP address is

  • fe80:0:0:0:20c:29ff:fe1b:9c5

The canonical form of an IPv6 address removes the longest run of zeros, leaving a double colon in its place. The double colon can appear anywhere.

  • fe80::20c:29ff:fe1b:9c5 same as above * :: zero or IN6ADDR_ANY

Because colon is overloaded, when an IPv6 address appears in a URL it is enclosed by brackets:

  • http://[fe80::20c:29ff:fe1b:9c5]:80

Typically, the first 64 bits identify a network, and the remaining 64 bits identify a node on that network. So in your ifconfig output, you will probably see /64 appended to the address. It is not part of the address.

The most common type of IPv6 address type is ‘link local’, automatically generated from the interface MAC address. It will always start with fe80. It is only valid on the interface it is attached to, so in a multi-interface system, its interface must be attached to the 128 bit IP address thusly:

  • fe80::20c:29ff:fe1b:9c5%eth0

Some Linux systems add the %eth0 to the address in an ifconfig list, some do not. Some additional qualifying information may be added to the qualifier:

  • fe80::20c:29ff:fe1b:9c5%enp0s20u2u1

I only mention these additions because we will be seeing more of them in the future. They are not needed to follow this document.

Assumptions about how IoTivity should handle IPv6

Since IPv4 and IPv6 don’t interact, several approaches to transitioning from v4 to v6 have been invented. We have chosen the Dual Stack approach because it is most general, allowing running on either protocol family or both at the same time. By default, an IoTivity client will discover a server using both IPv4 and IPv6. An IoTivity server can listen on both v4 and v6. The datagram using one technology will arrive first, and that's the one that will be processed. The second one arriving will be dropped as long as it arrives before the first one’s token disappears. (If a second datagram arrives with a duplicate token for any other reason, it will be passed up to the app, just as before this change.)

Of course, an application can choose to run on either IPv4 or IPv6. The given approach will allow most users and applications to run over either, transparently, with a bias toward IPv6 when it is available. A likely common approach will be for clients to use both address families while servers use one or the other.

Approach to adding IPv6

First, we decided to do a deep, narrow, agile dive. Bypassing DTLS, single-thread, WiFi, and Presence allowed us to more quickly get to an holistic understanding of the issues and show something working. It took longer to reach this point than we had hoped because of the discovered scale of the changes. The original intention was to change only the lowest (socket) levels of Connectivity Abstraction, supported by adding a simple selection method to the API. We soon found that every level of IoTivity made assumptions about the transport addressing, and the discovery of these places in the code were made in a serial fashion. (When one was discovered, we found the next one.)

Types of changes made in ca-ipv6 branch

changes to APIs

The only change originally envisioned to the APIs was putting Address Family selection into the OCConnectivityType (C++ FindResource), and CAConnectivityType (csdk OCDoResource). We later found that the OCDevAddr structure used in the csdk callback to deliver addressing information was entirely incapable of handling an IPv6 address, so that was changed as well.

A needed change to the APIs is the ability for a server application to indicate what address family it wishes to use (currently it responds to both.) This addition is the server equivalent of ConnectivityType in the client.

changes to implement IPv6 capability

The initial changes involved plumbing the ConnectivityType additions from application to the bottom layers (caethernetclient.c and caethernetserver.c). Second came changing the bottom layers, where the socket programming is done, to do the right thing for IPv6 and IPv4. The client changes involved separate functions for each, selected based on the ConnectivityType. The server changes were less intrusive, so special case code in CAReceiveHander was sufficient. Those changes were done in the first couple of weeks while finding our way about the code. Once we started testing those changes, it became clear that the next and likely most substantial issue was address handling in the upper layers. IoTivity and the CA extension choose to handle network addresses as ASCII strings, a carryover from previously using CoAP, resulting in conversions back and forth between binary and text forms. For example, IP addresses are represented in URIs like so: coap://[fe80::20c:29ff:fe1b:9c5]:5300. Consider how many assumptions are made by networking software in formatting that address. Much of the later stages of recent development have dealt with reconciling those issues with the naive handling of IPv4 addresses throughout the stack.

changes to make IoTivity work properly

We ran across several issues in testing IPv6 that weren’t directly related to the address family itself.

  • The call from InprocClientWrapper::ListenForResource (C++) to OCDoResource (csdk) gives no indication that a multicast is expected. This leave OCDoResource to do something silly like assume that only an OCConnectivityType of OC_ALL means multicast. Since I wanted to multicast on ethernet (and others might), I enlarged the method enum to include OC_REST_GETALL, which unequivocally means multicast independent of the ConnectivityType.

changes to make the code more elegant

  • I got lost in the maze of ‘char *’ casts in stack.c (the top layer of csdk). Most of them were casts between ‘char *’ and ‘unsigned char *’, and some involved ‘const’ as well. When I discovered a cast that removed const from ‘char *’, I eliminated most of the casts by changing the ‘unsigned char *’ strings to ‘char *’, a change that has no effect on running code, but is much easier on the eyes.
  • In several places I ran across code like this:
    if (e == a && r)
    {}
    else if (e == b && s)
    {}

    \\where a and b are part of an enumeration. While this code works, it unnecessarily burdens the second ‘if’ when (e == a) is true but r is false. It can more properly be written

    switch (e)
    {
    case a:
            if (r)
            {}
    	break;
    case b:
    	if (s)
    	{}
    	break;
    default:
    	break;
    }

    \\I changed a couple of these where I thought they would benefit. I made the changes because I found the switch statement easier to read.

Descriptions of major changes

Changes to APIs

C++ API

The original OCConnectivityType enum was

typedef enum {
    OC_ETHERNET = 0,
    OC_WIFI,
    OC_EDR,
    OC_LE,
    OC_ALL //Multicast message: send over all the interfaces.
} OCConnectivityType;

This has been changed to

typedef enum {
    OC_ETHERNET = 0, // IPv4 and IPv6 (dual stack)
    OC_ETHERNET_IPv6,
    OC_ETHERNET_IPv4,
    OC_WIFI,         // IPv4 and IPv6 (dual stack)
    OC_WIFI_IPv6,
    OC_WIFI_IPv4,
    OC_EDR,
    OC_LE,
    OC_ALL // send over all the interfaces.
} OCConnectivityType;

The intention is that we will use both IPv4 and IPv6 by default, but the user can specify which one to use. csdk API The primary C SDK entry point (OCDoResource) uses OCConnectivityType, but quickly turns it into CAConnectivityType, described in the CA API section below.

OCDoResource takes an argument of type OCMethod. I updated it to this:

typedef enum {
    OC_REST_NOMETHOD    = 0,
    OC_REST_GET         = (1 << 0),     // Read
    OC_REST_PUT         = (1 << 1),     // Write
    OC_REST_POST        = (1 << 2),     // Update
    OC_REST_DELETE      = (1 << 3),     // Delete
    OC_REST_GETALL      = (1 << 4),	// multicast read
    // Register observe request for most up date notifications ONLY.
    OC_REST_OBSERVE     = (1 << 5),
    // Register observe request for all notifications, including stale notifications.
    OC_REST_OBSERVE_ALL = (1 << 6),
    // Deregister observation, intended for internal use
    OC_REST_CANCEL_OBSERVE = (1 << 7),
    #ifdef WITH_PRESENCE
    // Subscribe for all presence notifications of a particular resource.
    OC_REST_PRESENCE    = (1 << 8)
    #endif
} OCMethod;

The only change I made to it is highlighted in bold. Previously, the C++ ListenForResource and ListenForDevice methods would provide OC_REST_GET to OCDoResource, and OCDoResource would infer multicast from the OCConnectivityType argument. The inference was that only the value OC_ALL would multicast. Since I wanted to test IPv6 multicast only on ethernet, I couldn’t with the old regime, and I suspect others would want that control as well. So I added OC_REST_GETALL value, which is independent of the connectivity type. It is used by both ListenForResource and ListenForDevice.

Responses and requests are reported back to C applications and the C++ layer with a callback (OCClientResponseHandler). The third argument to the callback is a OCClientResponse structure, which has a OCDevAddr structure. The OCDevAddr structure carries back the identification of the sender of the response. The structure used to look like this:

typedef struct OCDevAddr {
    uint32_t     size;      /**< length of the address stored in addr field. */
    uint8_t      addr[DEV_ADDR_SIZE_MAX]; /**< device address. */
} OCDevAddr;

This structure has several problems. First, DEV_ADDR_SIZE_MAX is 16, which isn’t enough room to hold an IPv6 IP address and port number. Second, the addr array is abused, meaning that the four bytes of an IPv4 address are stored in the first four bytes, and the port (16 bits) is stored with a cast over the next two bytes. (IMHO, that’s an API no-no). Third, creating the array of bytes takes time in ocstack.c, and the only reasonable use of the array of bytes is to reconstruct the original string, which the C++ layer promptly does. Fourth, reconstructing an IPv6 ASCII address is non-trivial because of the preferred use of the ‘::’ (double colon) separator.

For all these reasons, I redefined the OCDevAddr structure thusly:

typedef struct OCDevAddr {
    #define provide_old_address_info
    #ifdef provide_old_address_info
    // old format: used by IPv4 in old applications
    uint32_t     size;      /**< length of the address stored in addr field. */
    uint8_t      addr[DEV_ADDR_SIZE_MAX]; /**< device address. */
    #endif // provide_old_address_info
 
    // new format: used by all types in new applications
    char         string[DEV_STRING_SIZE_MAX]; // holds IPv6 address
    OCConnectivityType type;
    uint16_t     port;
} OCDevAddr;

Comments on this change:

  • I kept the old elements for compatibility with old code and added the new items to handle the function more cleanly.
  • DEV_STRING_SIZE_MAX is 40, long enough to hold a fully expanded IPv6 address.
  • The port is stored explicitly as a short.
  • The connectivity type is available in the structure, which is both appropriate given the other content, and convenient for the C++ code that consumes the structure.
  • The samples and test that use the callback use helper functions in C SDK to reconstruct the ASCII strings that were torn apart, so I modified the helper functions to use the new information.

CA API

CAConnectivityType carries similar information as OCConnecitivityType, but in a much different manner. OCConnectivity type from the C API is quickly turned into CAConnectivityType, which is used throughout the lower levels of the stack. Here is what I found when I started:

typedef enum
{
    CA_ETHERNET = (1 << 0), /**< Ethernet Connection */
    CA_WIFI = (1 << 1),     /**< WIFI Connection */
    CA_EDR = (1 << 2),      /**< EDR Connection */
    CA_LE = (1 << 3),       /**< LE Connection */
} CAConnectivityType_t;

This structure supports much more flexibility than OCConnectivityType, and I believe OCConnecitivityType should be reworked to match it. To support IPv6, I changed it to the following:

typedef enum
{
    CA_ETHERNET = (1 << 0), /**< Ethernet Connection */
    CA_WIFI = (1 << 1),     /**< WIFI Connection */
    CA_EDR = (1 << 2),      /**< EDR Connection */
    CA_LE = (1 << 3),       /**< LE Connection */
    **CA_IPv6 = (1 << 30),    /**< Use IPv6 */
    CA_IPv4 = (1 << 29)     /**< Use IPv4 */**
    /**< If neither IPv6 or IPv4 is set, or if both are set, use both */
} CAConnectivityType_t;
 
#define CONNECTION_TYPES_MASK 0x0fff
#define CONNECTION_TYPE_MODS_MASK 0xf0000000
#define ADDR_FAMILY_MODS_MASK (CA_IPv6|CA_IPv4)

Here, independent bits control whether IPv6 or IPV4 is used. I also grouped them and provided masks to manage them.

Changes to Ethernet adapter

A major change to the ethernet adapter is the proliferation of the CAConnectivityType structure through all the levels down to the bottom, since the IPvX bits need to be available all the way at the bottom of the stack. I didn’t modify or use caethernetmonitor. When I realized we were going to great lengths to only use one interface of each adapter type, I reported this shortcoming to the IoTivity DEV list, and put off handling interfaces until some resolution is reached on the issue. This means that the IPv6-capable ethernet adapter currently runs “promiscuously” and handles all interfaces (including WiFi if it is available.)

caethernetclient.c

I found the name of this file confusing. It contains the sending code for both client and server.

Separate functions (CASendDataIPv6 and CASendDataIPv4) handle sending. Before, the port addresses were held in caethernetadapter.c and passed in as arguments. Since the ports are either constants or configurable along with the interface, I chose to move them down to this level, allowing more flexibility.

Address conversion was previously handled with the deprecated inet_addr function. We now use inet_aton for IPv4 and getaddrinfo for IPv6.

caethernetserver.c

I found the naming of this file confusing. It contains the receiving code for both client and server.

Here, all the changes are in the CAReceiveHandler function. Instead of listening to only a single IPv4 socket per context, the select now listens to two sockets for each context, identifies the type when a message is received, and formats the remote IP address appropriately before sending the message up the stack.

This file also includes the functions that create both the listening and sending sockets. Again, the deprecated inet_addr function is replaced by getaddrinfo, which in this case is used for both IPv6 and IPv4.

caethernetadapter.c

First, I added isMulticast to the CAEthernetData structure used for the output queue, in order to make multicast explicit rather than implicit. I also eliminated passing IP addresses through the layers: they are either already in the CARemoteEndpoint structure that comes down from above, or they are multicast addresses, which are better handled in the lower levels.

Again, I eliminated references to the network monitor, expecting to include them in the bottom layer once an approach to handling interfaces is identified.

Changes to connectivity abstraction

Most of the changes in the connectivity abstraction involve the plumbing of additional information up and down the stack.

caremotehandler.c::getCAAddress()

This function converts a packed address (IP or MAC) from the URI of an outgoing request to separate address string and port value. This complicated in the IPv6 case by the standard URI representation of an IPv6 address:

coap://[fe80::20c:29ff:fe1b:9c5]:5445

which is quite different from IPv4 format. The brackets need to be removed for getaddrinfo later.

In CACreateRemoteEndpointInternal() is the code that provides default behavior:

if ((type & (CA_IPv6|CA_IPv4)) == 0) // If neither IPv6 or IPv4 is given,
        type |= CA_IPv6|CA_IPv4;         // assume both are desired.

cainterfacecontroller.c::CAGetAdapterIndex()

The CONNECTION_TYPES_MASK (shown above) is used to isolate the connection types from the IP modifiers:

switch (cType & CONNECTION_TYPES_MASK)

camessagehandler.c::CADetachRequestMessage()

I partially consolidated the unicast and multicast paths here. Because the multicast path “knew” that a multicast address address would be generated later, it didn’t bother passing the CAConnectivityType down. Once I added the CAConnectivityType to the CAGroupEndpoint_t structure, I realized that it is a proper subset of the CARemoteEndpoint_t structure, and the latter could better be used for both. This simplified several aspects of the code. Here, the previously mentioned CA_GETALL method is turned in SEND_TYPE_MULTICAST in CADetachRequestMessage(), eliminating the need for the duplication of code in CADetachRequestToAllMessage().

Changes to csdk

ocstack.c::OCToCAConnectivityType() and ocstack.c::CAToOCConnectivityType

Both of these became much more complicated because of the different styles of the OC and CA connectivity type enums. Changing OCConnectivityType to be more like CAConnectivityType would simplify this code and give the user more flexibility.

ocstack.c::UpdateResponseAddr()

Here is the code that fills in the OCDevAddr structure described above. The deprecated code is contained in an enabled #ifdef, so it still provides the information expected. Note that the deprecated code is quite ugly. First, it copies the input address string so it can then safely damage it with strtok_r. Then, it abuses the output array with the following line:

memcpy(&address[4], &endPoint->addressInfo.IP.port, sizeof(uint32_t));

The new code replaces that abuse with these two lines:

strcpy(response->addr->string, endPoint->addressInfo.IP.ipAddress);
response->addr->port = endPoint->addressInfo.IP.port;

ocstack.c::HandlePresenceResponse()

Even though I don’t yet handle the Presence case, I added the following comment to a problematic line of code for later consideration:

// building URIs here is problematical on several levels.
// 1. They probably don't work on non-IP adapters.
// 2. This level doesn't have a good way to distinguish v4/v6 multicast.
// 3. Formatting of IPv6 addresses in URIs needs work (uses [ ])
snprintf(fullUri, MAX_URI_LENGTH, "coap://%s:%u%s", ipAddress,    endPoint->addressInfo.IP.port, OC_PRESENCE_URI);

ocstack.c::HandleStackRequests

Here is where the dual IP addresses are resolved. Remember that by default the client sends both IPv6 and IPv4 multicasts (unless the client picks), in that order. When the request gets here in a server, it creates an ‘database’ entry keyed on a token, and both the IPv6 and IPv4 messages will have the same token. The first request creates an entry, and second one finds that the token already exists. Of course, this could happen already for other reasons, resulting in multiple request packets being delivered upward. The addition is that if the only difference between the first and second messages is its IP type, the second one is discarded. The code is carefully constructed to not affect the previous behavior when duplicate messages are delivered for other reasons. The line that accomplishes this is

if ((protocolRequest->connectivityType ^ request->connectivityType) == (CA_IPv6|CA_IPv4))

ocstack.c::OCDoResource()

I largely rewrote/reorganized this function. It stretched on for pages and pages, but much of the stretching was not functional. And it had some barriers to IPv6.

The arguments to OCDoResource were problematical. As mentioned earlier, I extended OCMethod to more precisely describe what is going on. Then I had to understand that the referenceUri is not used at all. The code wants you to think it will be used, but I can’t imagine what for. The OCConnectivityType was modified, as described earlier. And an important part of the OCCallbackData structure (OCDevAddr) was modified as discussed earlier.

One thing that stretched the function was two switch statements on the same variable (method). Once to qualify the given method; again to convert the OC method to a CA method. I combined them, saving me a lot of scrolling time.

I moved several one-time checks to the front of the function: this reduced scrolling in the meat of the function.

The URI handling was quite ad hoc, requiring lines at the end like

if(newUri != requiredUri)
{
    OCFree(newUri);
}

This kind of thing reduces maintainability and makes it harder to learn the code. The changed code allows a simple free(). [ newUri → requestUri ]

An OCRequestData object is embedded in OCRequestInfo (not a pointer to OCRequestData), but OCDoResource filled in a separate OCRequestData struct and then copied the separate structure into OCResourceInfo. I got rid of the separate OCRequestData structure and put all the values into structure embeded in OCRequestInfo, saving the data copy and making what’s going on clearer.

As discussed earlier, a connection type of OC_ALL used to determine whether multicast was used. In that case, a CAGroupEndpoint_t was created and filled in. By using CARemoteEndpoint_t for both, the decision and extra code are eliminated. Now both cases call CASendRequest, eliminating 20 lines of code.

Previously, the code sent the request, then registered the callback. I changed that around to first register the callback, then send the request. The old way got the request sent a bit earlier, perhaps overlapped with adding the callback. I judged that the overlap wasn’t worth the (unlikely) risk of a failure in AddClientCB(). Also, the SendRequest kicks off another thread, interfering with debugging AddClientCB().

The following two helper functions are used by C++ to access the OCDevAddr structure, making the change there transparent. I suspect all the sample and test code does the same.

ocstack.c::OCDevAddrToString

This is a helper function that converts the OCDevAddr (discussed above) to a string. Now instead of using the byte array to reconstruct the IPv4 address (which all it could do because it has no address type info available), it just returns a copy of the string of the new struct, a string that is already properly formatted.

ocstack.c::OCDevAddrToPort

This is a helper function that pulls the port number out of OCDevAddr. It uses the new data, so it’s cleaner.

ocserverrequest.c::HandleSingleResponse

When the request was being turned into a response, the address family bits were lost through some obscure masking code. It was simplified to

connType = responseEndpoint.connectivityType & CONNECTION_TYPES_MASK;

Changes to C++ sdk

OCSerialization.h::OCSecureType

Was

enum class OCSecureType
{
    IPv4Secure,
    IPv4
};

Now

enum class OCSecureType
{
    Normal,
    Secure
};

The SecureType really had nothing to do with IPv4 (or IP at all), so I changed the names to a more descriptive form.

OCSerialization::ConvertOCAddrToString()

This turns the callback IP and port into a URI. It calls the C SDK helper file OCDevAddrToString() to handle the OCDevAddr struct that I described above. It now handles IPv6 addressing.

InProcClientWrapper::ListenForResource

The only real change here is to call OCDoResource with OC_REST_GETALL if no serviceUrl is given.

InProcClientWrapper::ListenForDevice

The only real change here is to call OCDoResource with OC_REST_GETALL if no serviceUrl is given.

What works at this point

The changes so far have been little tested. I used the following simple configuration to test: Client:

out/linux/x86_64/release/examples/OICMiddle/OICMiddle -client -console
show
get 0

Server:

out/linux/x86_64/debug/resource/examples/simpleserver

This uses the Ethernet adapter, but currently it uses all interfaces (INADDR_ANY), so it should work on WiFi as well.

It doesn’t fully support multiple interfaces in IPv6 (%eth0) at this time.

ipv6_changes_to_iotvity.txt · Last modified: 2015/03/20 19:52 by Ossama Othman