Type validator

• Type validator is a Python module for validating Python objects from untrusted source
• It was developed for a P2P application that sends and receives bencoded messages from network
• It is recommended that type validator is used with type validator
• The reference implementation is in public domain (you may do whatever you want with the code)

Motivational background

• XML is a widely used serialization format but it is somewhat bloated for network protocols
• Want a Pythonish approach for network serialization that is safe and supports binary data (Python 2.x strings are binary safe)
• Minimize risks and code for validating network messages

How it works

• Read data from network, unserialize data to Python objects with bencode, and finally, validate the objects with type validator. Data comes as recursive structures of Python primitives consisting from integers, strings, booleans, dictionaries and lists.
• Use Python's iterators and type system for a compact implementation (the code is just 80 lines with comments)
• Message format is specified using Python syntax to make type validator intuitive to use
• It is possible to validate values by specifying a function callback to test a value
• Bencode was extended with bool type for Python type completeness

Download type validator

View type validator source code

Examples

# type validator documentation as a commented source code :-)

from bencode.py import fmt_bdecode
from typevalidator import validate, INT, STRING, ONE_OR_MORE, OPTIONAL_KEY, 

# Syntax of validate: validate(specification, object)
# Syntax of fmt_bdecode: fmt_bdecode(specification, blob)
#
# Blob is first transformed to an object with bdecode, then validated with
# type validator. This can be done with one call with fmt_bdecode()

# Asserted examples with type validator:

# Demand a dictionary whose keys and values are strings:
assert validate({STRING_KEY: STRING}, {'name': 'Cheradenine'})
assert validate({STRING_KEY: STRING}, {1: 'Cheradenine'}) == False
assert validate({STRING_KEY: STRING}, {'name': 1}) == False

# Demand a dictionary whose keys are integers but values may have any
# (supported) type. Furthermore, key with value 123 must exist.
assert validate({INT_KEY: ANY, 123: ANY}, {123: 'x'})
assert validate({INT_KEY: ANY, 123: ANY}, {123: 456})
assert validate({INT_KEY: ANY, 123: ANY}, {4: 'x'}) == False # 123 does not exist

# List may begin with ZERO_OR_MORE or ONE_OR_MORE to specify that minimum
# length of the list is either zero or one, respectively. If either is
# used, then also a type must be specified after this.
assert validate([ZERO_OR_MORE, ANY], [])
assert validate([ONE_OR_MORE, ANY], ['x'])
assert validate([ONE_OR_MORE, ANY], []) == False
assert validate([ONE_OR_MORE, STRING], ['x', 1]) == False

# Recursive data structures are easy to specify! Define a list of
# dictionaries.
assert validate([ZERO_OR_MORE, {'name': STRING}], [{'name': 'User1'}, {'name': 'User2'}])
assert validate([ZERO_OR_MORE, {'name': STRING}], [1, {'name': 'User1'}]) == False

# Define a list that contains one string and one dictionary:
assert validate([STRING, {}], ['foo', {}])
assert validate([STRING, {}], [1, {}]) == False
assert validate([STRING, {}], ['foo', 1]) == False
assert validate([STRING, {}], ['foo', {}, 3]) == False # Too long a list

# Extra keys are allowed in the dictionary. Even if only 'x' key is specified,
# other keys are allowed. This was a choice to make message protocols
# extensible.
assert validate({'age': INT}, {'age': 1, 'other': 'stuff'})

# Require positive integers by using lambdas
assert validate({'x': lambda x: type(x) == int and x > 0}, {'x': -1}) == False

# bencode example

blob = read_from_socket()
if blob == None:
    return
specification = {'name': STRING,
                 OPTIONAL_KEY('email'): STRING,
                 OPTIONAL_KEY('age'): INT,
                 'a-list': [],
                 'non-empty-string-list': [ONE_OR_MORE, STRING],
                }
msg = fmt_bdecode(specification, blob)
if msg == None:
    # Invalid object
    return

# Now it is guaranteed that msg is a dictionary with previous specification.
# msg['name'] exists, msg['email'] and msg['age'] may exist.
# If msg['email'] exists, it is a string. If msg['age'] exists, it is an
# integer. 'a-list' exists and it is
# a list, but nothing is specified about type inside the list.
# msg['non-empty-string-list'] exists, and it is a non-empty list that
# contains strings.

Author

Heikki Orsila

Acknowledgements

Thanks to Janne Kulmala for the idea of using Python's iterators and type system in syntax.