Overview

Introduction

OSER is an easy to use, flexible object oriented serializer and deserializer that can be used for translation between binary data formats and its internal structure represented by python classes.

OSER is like a toolbox. It offers many building blocks to build your data format quickly and easily.

Unlike other serializers OSER makes it possible to inspect your data instances in an hierarchical way with or without the binary data aligned to the members. In addition OSER ist capable of building conditional serializers and deserializers using oser.IfElse, oser.Switch, oser.Array and oser.String by accessing the context.

For example OSER can be used to build and parse network protocols, image data and various binary files or you can create the content for an EEPROM for an embedded system, etc..

General Concepts

An OSER instance consists of a tree of other OSER instances that implement oser.OserNode. Every OSER instance sublasses the oser.OserNode. The composite pattern is used here.

Every member that does not start with _ is included in serialization and deserialization.

>>> from oser import ByteStruct, UBInt8, UBInt16, to_hex

>>> class SubData(ByteStruct):
...     def __init__(self):
...         super(SubData, self).__init__()
...         self.sub1 = UBInt8(1)
...         self.sub2 = UBInt16(1000)
...

>>> class Data(ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.a = UBInt8(1)
...         self.b = UBInt16(1000)
...         self.sub = SubData()
...
>>> instance = Data()

Every OSER instance can be viewed when it is converted into a string, for example:

>>> print(instance)
Data():
    a: 1 (UBInt8)
    b: 1000 (UBInt16)
    sub: SubData():
        sub1: 1 (UBInt8)
        sub2: 1000 (UBInt16)

This shows the internal structure only. To view the internal structure and the binary structure, you can use oser.OserNode.introspect():

>>> print(instance.introspect())
   -    -  Data():
   0 \x01      a: 1 (UBInt8)
   1 \x03      b: 1000 (UBInt16)
   2 \xe8
   -    -      sub: SubData():
   3 \x01          sub1: 1 (UBInt8)
   4 \x03          sub2: 1000 (UBInt16)
   5 \xe8

To build binary data from an OSER instance, simply use oser.OserNode.encode():

>>> binary = instance.encode()
>>> print(to_hex(binary))
   0|  1|  2|  3|  4|  5
\x01\x03\xE8\x01\x03\xE8

To decode binary data into an OSER instance, simply use oser.OserNode.decode():

>>> data = b"\x01\x02\x03\x04\x05\x06"
>>> bytes_decoded = instance.decode(data)
>>> print(bytes_decoded)
6
>>> print(instance.introspect())
   -    -  Data():
   0 \x01      a: 1 (UBInt8)
   1 \x02      b: 515 (UBInt16)
   2 \x03
   -    -      sub: SubData():
   3 \x04          sub1: 4 (UBInt8)
   4 \x05          sub2: 1286 (UBInt16)
   5 \x06

To access the root element call oser.OserNode.root():

>>> root = instance.sub.sub2.root()
>>> print(root)
Data():
    a: 1 (UBInt8)
    b: 515 (UBInt16)
    sub: SubData():
        sub1: 4 (UBInt8)
        sub2: 1286 (UBInt16)

To access the upper element (parent element) call oser.OserNode.up():

>>> up = instance.sub.sub2.up()
>>> print(up)
SubData():
    sub1: 4 (UBInt8)
    sub2: 1286 (UBInt16)


>>> upup = up.up()
>>> print(upup)
Data():
    a: 1 (UBInt8)
    b: 515 (UBInt16)
    sub: SubData():
        sub1: 4 (UBInt8)
        sub2: 1286 (UBInt16)

Navigation in the element tree is useful to build conditional blocks.

OserNode

class oser.OserNode

__str__(indent: int = 0, name: str | None = None, stop_at: ByteStruct | BitStruct | ByteType | BitType | None = None) → str: Return the representation of the object as a string.

decode(data: bytes, full_data: bytes = b'', context_data: bytes = b'') → int

Decode a binary string into a byte type and return the number of bytes that were decoded.

Parameters:

data – the data buffer that is decoded.
full_data – the binary data string until the part to be decoded. The user normally does not need to supply this.
context_data – the binary data of the current context. The user normally does not need to supply this.

Returns:

the number of bytes that were decoded.

Return type:

int

encode(full_data: bytes = b'', context_data: bytes = b'') → bytes

Return the encoded binary string.

Parameters:

full_data – the binary data string until the part to be encoded. The user normally does not need to supply this.
context_data – the binary data of the current context. The user normally does not need to supply this.

Returns:

the encoded binary string.

Return type:

bytes

introspect(stop_at: ByteStruct | BitStruct | ByteType | BitType | None = None) → str

Return the introspection representation of the object as a string.

Parameters:: stop_at=None – stop introspection at stop_at.

root() → ByteStruct | BitStruct: Return the root element.

up() → ByteStruct | BitStruct: Return the parent element.

Accessing Data

Accessing Members

To access members in OSER instances simply use the dot. Your IDE is able to help you expanding the valid values. In eclipse you can press CTRL+SPACE when the curser is right behind the instance variable to get a list of valid members of an instance.

Example:

>>> import oser

>>> class SubData(oser.ByteStruct):
...     def __init__(self, *args, **kwargs):
...         super(SubData, self).__init__(*args, **kwargs)
...
...         self.subdata = oser.UBInt16(23)
...
>>> class Data(oser.ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.enum = oser.Enum(prototype=oser.UBInt16,
...                          values={
...                              "A": 1,
...                              "B": 2,
...                              "C": 3,
...                              "D": 4,
...                          }, value="D")
...
...         self.s = SubData()
...
>>> data = Data()

>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3
>>> print(data.enum.get())
D

Reading Values

Every oser.ByteType is an instance of a class. Simply use oser.ByteType.get() to get the current value.

Since arithmetic emulation is implemented for all primitive types, there is no need to call oser.ByteType.get() if you want to compare two values, etc..

Example:

>>> from oser import ByteStruct, Enum, UBInt16

>>> class SubData(ByteStruct):
...     def __init__(self, *args, **kwargs):
...         ByteStruct.__init__(self, *args, **kwargs)
...
...         self.subdata = UBInt16(23)
...
>>> class Data(ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.s = SubData()
...
>>> data = Data()
>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3

Setting Values

Every oser.ByteType is an instance of a class. Simply use oser.ByteType.set() to set the current value.

Example:

>>> from oser import ByteStruct, Enum, UBInt16, to_hex

>>> class Data(ByteStruct):
...         def __init__(self):
...             super(Data, self).__init__()
...             self.enum = Enum(prototype=UBInt16,
...                              values={
...                                  "A": 1,
...                                  "B": 2,
...                                  "C": 3,
...                                  "D": 4,
...                              }, value="C")
...
>>> instance = Data()
>>> print(instance)
Data():
    enum: 'C' (UBInt16)

>>> print(instance.introspect())
   -    -  Data():
   0 \x00      enum: 3 (UBInt16)
   1 \x03

>>> binary = instance.encode()
>>> print(to_hex(binary))
   0|  1
\x00\x03
>>> instance.enum.set("B")
>>> print(instance)
Data():
    enum: 'B' (UBInt16)

>>> print(instance.introspect())
   -    -  Data():
   0 \x00      enum: 2 (UBInt16)
   1 \x02

>>> binary = instance.encode()
>>> print(to_hex(binary))
   0|  1
\x00\x02
>>> bytes_decoded = instance.decode(binary)
>>> print(bytes_decoded)
2
>>> print(instance)
Data():
    enum: 'B' (UBInt16)

The Context

Every building block is aware of its context. Conditional building blocks use the context to decide how to proceed.

Variable Length String

The simplest example is a oser.String with a variable length.

>>> from oser import ByteStruct, String, Switch, Null, IfElse, \
    UBInt8, UBInt16, UBInt32, Array

>>> class VariableLengthString(ByteStruct):
...     def __init__(self):
...         super(VariableLengthString, self).__init__()
...         self.length = UBInt16(1)
...         self.data = String(
...             length=lambda self: self.length.get(),
...             value=b"abcdefghijklmnopqrstuvwxyz")
...
>>> instance = VariableLengthString()
>>> print(instance.introspect())
   -    -  Data():
   0 \x00      length: 1 (UBInt16)
   1 \x01
   -    -      data: String():
   2 \x61          'a'

>>> instance.length.set(16)
>>> print(instance.introspect())
   -    -  Data():
   0 \x00      length: 16 (UBInt16)
   1 \x10
   -    -      data: String():
   2 \x61          'a'
   3 \x62          'b'
   4 \x63          'c'
   5 \x64          'd'
   6 \x65          'e'
   7 \x66          'f'
   8 \x67          'g'
   9 \x68          'h'
  10 \x69          'i'
  11 \x6a          'j'
  12 \x6b          'k'
  13 \x6c          'l'
  14 \x6d          'm'
  15 \x6e          'n'
  16 \x6f          'o'
  17 \x70          'p'

The length of the oser.String is a callable here that returns the length’s value.

For simple true-false-decisions oser.IfElse can be used.

>>> class IfElseData(ByteStruct):
...     def __init__(self):
...         super(IfElseData, self).__init__()
...         self.true_false = UBInt8(1)
...         self.data = IfElse(condition=lambda self: bool(self.true_false.get()),
...                            if_true=UBInt8(1),
...                            if_false=UBInt32(0xffffffff)
...                            )
...
>>> instance = IfElseData()
>>> print(instance.introspect())
   -    -  IfElseData():
   0 \x01      true_false: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.true_false.set(0)
>>> print(instance.introspect())
   -    -  IfElseData():
   0 \x00      true_false: 0 (UBInt8)
   1 \xff      data: 4294967295 (UBInt32)
   2 \xff
   3 \xff
   4 \xff

The condition of oser.IfElse is a callable here that returns the true_false’s value as bool.

A more complex example is a oser.Switch that decides the type of the payload.

>>> class SwitchData(ByteStruct):
...     def __init__(self):
...         super(SwitchData, self).__init__()
...         self.type = UBInt8(1)
...         self.data = Switch(condition=lambda self: self.type.get(),
...                            values={
...                                1: UBInt8(1),
...                                2: UBInt16(2),
...                                4: UBInt32(3)
...                            },
...                            default=Null())
...
>>> instance = SwitchData()
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x01      type: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.type.set(2)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x02      type: 2 (UBInt8)
   1 \x00      data: 2 (UBInt16)
   2 \x02

>>> instance.type.set(4)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x04      type: 4 (UBInt8)
   1 \x00      data: 3 (UBInt32)
   2 \x00
   3 \x00
   4 \x03

>>> instance.type.set(3)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x03      type: 3 (UBInt8)
   -    -      data: Null

The condition of oser.Switch is a callable here that returns the type’s value. If type is not in [1,2,4] oser.Null is used.

To access parent elements use oser.OserNode.up().

>>> class SubSwitch(ByteStruct):
...     def __init__(self):
...         super(SubSwitch, self).__init__()
...         self.data = Switch(condition=lambda self: self.up().type.get(),
...                            values={
...                                1: UBInt8(1),
...                                2: UBInt16(2),
...                                4: UBInt32(3)
...                            },
...                            default=Null())
...

>>> class SwitchData2(ByteStruct):
...     def __init__(self):
...         super(SwitchData2, self).__init__()
...         self.type = UBInt8(1)
...         self.data = SubSwitch()
...
>>> instance = SwitchData()
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x01      type: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.type.set(2)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x02      type: 2 (UBInt8)
   1 \x00      data: 2 (UBInt16)
   2 \x02

>>> instance.type.set(4)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x04      type: 4 (UBInt8)
   1 \x00      data: 3 (UBInt32)
   2 \x00
   3 \x00
   4 \x03

>>> instance.type.set(3)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x03      type: 3 (UBInt8)
   -    -      data: Null

In this example the type is accessed by self.up().type.get().

It is also possible to create variable length repeated fields using oser.Array.

>>> class VariableLengthArray(ByteStruct):
...     def __init__(self):
...         super(VariableLengthArray, self).__init__()
...         self.length = UBInt16(1)
...         self.data = Array(
...             length=lambda self: self.length.get(),
...             prototype=UBInt8,
...             values=[UBInt8(ii) for ii in range(256)])
...
>>> instance = VariableLengthArray()
>>> print(instance.introspect())
   -    -  VariableLengthArray():
   0 \x00      length: 1 (UBInt16)
   1 \x01
   -    -      data: Array():
   -    -      [
   2 \x00          @0: 0 (UBInt8)
   -    -      ]

>>> instance.length.set(5)
>>> print(instance.introspect())
   -    -  VariableLengthArray():
   0 \x00      length: 5 (UBInt16)
   1 \x05
   -    -      data: Array():
   -    -      [
   2 \x00          @0: 0 (UBInt8)
   3 \x01          @1: 1 (UBInt8)
   4 \x02          @2: 2 (UBInt8)
   5 \x03          @3: 3 (UBInt8)
   6 \x04          @4: 4 (UBInt8)
   -    -      ]

In this example length is a callable like in the variable length string example.

Copying

OSER instances can be deeply copied using copy.deepcopy(). Shallow copies are not allowed since the context of each element must be distinct. If copy.copy() is applied on an OSER instance and exception is raised. While being copied encode() is called with all its side effects.

>>> import oser
>>> import copy


>>> class Struct(oser.ByteStruct):
...     def __init__(self):
...         super(Struct, self).__init__()
...         self.a = oser.ULInt8(1)
...
>>> s = Struct()
>>> id(s)
6596088
>>> print(s)
Struct():
    a: 1 (ULInt8)


>>> deep_copy = copy.deepcopy(s)
>>> id(deep_copy)
38738520

>>> s.a.set(100)  # this does not influence the copy
>>> print(deep_copy)
Struct():
    a: 1 (ULInt8)