Advanced Usage

In this part, Some of the advanced usages of TreeValue will be introduced one by one with sample code and graph to explain them.

For further information or usage of TreeValue and FastTreeValue, you may take a look at the following pages:

Function Modes

In the basic usage description in Make function tree supported, we can see that a common function which only support the calculation of the common values can be wrapped to support the tree-based calculation painlessly.

Here is the documentation of function treevalue.tree.func.func_treelize.

treevalue.tree.func.func_treelize(mode: str = 'strict', return_type: Optional[Type[TreeClassType_]] = <class 'treevalue.tree.tree.tree.TreeValue'>, inherit: bool = True, missing: Union[Any, Callable] = <SingletonMark 'missing_not_allow'>, delayed: bool = False, subside: Optional[Union[Mapping, bool]] = None, rise: Optional[Union[Mapping, bool]] = None)[source]
Overview:

Wrap a common function to tree-supported function.

Arguments:

  • mode (str): Mode of the wrapping, default is strict.

  • return_type (Optional[Type[TreeClassType_]]): Return type of the wrapped function, default is TreeValue.

  • inherit (bool): Allow inheriting in wrapped function, default is True.

  • missing (Union[Any, Callable]): Missing value or lambda generator of when missing, default is MISSING_NOT_ALLOW, which means raise KeyError when missing detected.

  • delayed (bool): Enable delayed mode or not, the calculation will be delayed when enabled, default is False, which means to all the calculation at once.

  • subside (Union[Mapping, bool, None]): Subside enabled to function’s arguments or not, and subside configuration, default is None which means do not use subside. When subside is True, it will use all the default arguments in subside function.

  • rise (Union[Mapping, bool, None]): Rise enabled to function’s return value or not, and rise configuration, default is None which means do not use rise. When rise is True, it will use all the default arguments in rise function. (Not recommend to use auto mode when your return structure is not so strict.)

Returns:

  • decorator (Callable): Wrapper for tree-supported function.

Example:
>>> @func_treelize()
>>> def ssum(a, b):
>>>     return a + b  # the a and b will be integers, not TreeValue
>>>
>>> t1 = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
>>> t2 = TreeValue({'a': 11, 'b': 22, 'x': {'c': 33, 'd': 5}})
>>> ssum(1, 2)    # 3
>>> ssum(t1, t2)  # TreeValue({'a': 12, 'b': 24, 'x': {'c': 36, 'd': 9}})

In the arguments listed above, 3 of them are key arguments:

  • mode, Tree process mode of the calculation. Will be introduced in this section.

  • inherit, Inheriting mode of the calculation on the tree. Will be introduced in Inheriting on Trees.

  • missing, Missing processor of the calculation on the tree. Will be introduced in Process Missing Values.

The mode argument is the most important argument in function func_treelize. For it depends the basic logic of the graph calculation.

There are 4 kinds of the modes, listed below:

  • strict mode, which is most usually used, means all the values on the tree should be mapped one by one.

  • inner mode, means only the keys which is commonly owned by all the trees will be processed.

  • outer mode, means all the keys involved in any of the trees will be processed.

  • left mode, means only the keys involved in the leftest tree will be processed.

In this part, all of the 4 modes will be introduced with details and samples one by one.

Strict Mode

Strict mode is the most frequently-used mode in real cases. It is also the default value of the mode argument in the treelize functions.

To be simple, strict mode need the keys of a node in the tree be strictly mapped one by one, missing or overage of the keys are both forbidden in strict mode, which means KeyError will be raised.

For example, in the trees in strict_demo_1, if we need to added t1 and t2 together, t3 will be the result. Because all the keys (a, b, x of the root nodes, and c, d, e of the x node) can be mapped one by one.

../../_images/strict_demo_1.gv.svg

In another situation, when the keys are not so regular like that in strict_demo_1, like the trees in strict_demo_2. If we try to add t1 and t2 in strict mode, KeyError will be raised due to the missing of key a in t1 and key f in t2.x.

../../_images/strict_demo_2.gv.svg

Here is a real code example of strict mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from treevalue import FastTreeValue, func_treelize


@func_treelize(mode='strict')
def plus(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5}})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})
    t4 = FastTreeValue({'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5, 'f': 6}})

    print('plus(t1, t2):', plus(t1, t2))
    print('plus(t4, t2):', plus(t4, t2))

The stdout and stderr should like below, a KeyError is raised at the second print statement.

1
2
3
4
5
6
7
plus(t1, t2): <TreeValue 0x7ff4e41f7b20>
├── 'a' --> 12
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7ff4e4595ee0>
    ├── 'c' --> 33
    ├── 'd' --> 52
    └── 'e' --> 59

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:23: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(TreeValue)
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:24: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(FastTreeValue)
Traceback (most recent call last):
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/docs/source/tutorials/advanced_usage/strict_demo.demox.py", line 15, in <module>
    print('plus(t4, t2):', plus(t4, t2))
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/func/func.py", line 56, in _new_func
    return _treelized(*args, **kwargs)
  File "treevalue/tree/func/cfunc.pyx", line 181, in treevalue.tree.func.cfunc._w_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 89, in treevalue.tree.func.cfunc._c_func_treelize_run
  File "treevalue/tree/func/modes.pyx", line 132, in treevalue.tree.func.modes._c_keyset
  File "treevalue/tree/func/modes.pyx", line 54, in treevalue.tree.func.modes._c_strict_keyset
KeyError: "Argument keys not match in strict mode, key set of argument 0 is {'x', 'b'} but 1 in {'x', 'b', 'a'}."

Note

How does the treelized function work?

Here is another example which show the actual calculation process of the wrapped function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import numpy as np

from treevalue import FastTreeValue, func_treelize


@func_treelize(mode='strict')
def plus(a, b):
    print("Current a and b:", type(a), a, type(b), b)
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})
    t2 = FastTreeValue({
        'a': np.array([1, 3, 5, 7]),
        'b': np.array([2, 5, 8, 11]),
        'x': {
            'c': 3,
            'd': 4.8,
            'e': np.array([[2, 3], [5, 6]]),
        }
    })
    t3 = FastTreeValue({
        'a': np.array([1, 3, 5, 7]),
        'b': [2, 5, 8, 11],
        'x': {
            'c': 3,
            'd': 4.8,
            'e': np.array([[2, 3], [5, 6]]),
        }
    })

    print('plus(t1, t2):', plus(t1, t2))
    print()
    print('plus(t1, t3):', plus(t1, t3))
    print()

In this code, once the original function plus is called, the actual value of argument a and b will be printed to stdout.

In the plus(t1, t2), all the calculation can be carried on because add operator between primitive int and np.ndarray is supported, so no error occurs and the result of + between int and np.ndarray will be one of the values of the result tree.

But in plus(t1, t3), when get the result of plus(t1, t3).b, it will try to add t1.b and t2.b together, but there is no implement of operator + between primitive int and list objects, so TypeError will be raised when do this calculation.

The complete stdout and stderr should be like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
Current a and b: <class 'int'> 11 <class 'numpy.ndarray'> [1 3 5 7]
Current a and b: <class 'int'> 54 <class 'numpy.ndarray'> [[2 3]
 [5 6]]
Current a and b: <class 'int'> 30 <class 'int'> 3
Current a and b: <class 'int'> 48 <class 'float'> 4.8
Current a and b: <class 'int'> 22 <class 'numpy.ndarray'> [ 2  5  8 11]
plus(t1, t2): <TreeValue 0x7f68bf115ee0>
├── 'a' --> array([12, 14, 16, 18])
├── 'b' --> array([24, 27, 30, 33])
└── 'x' --> <TreeValue 0x7f68b1f20cd0>
    ├── 'c' --> 33
    ├── 'd' --> 52.8
    └── 'e' --> array([[56, 57],
                       [59, 60]])


Current a and b: <class 'int'> 11 <class 'numpy.ndarray'> [1 3 5 7]
Current a and b: <class 'int'> 54 <class 'numpy.ndarray'> [[2 3]
 [5 6]]
Current a and b: <class 'int'> 30 <class 'int'> 3
Current a and b: <class 'int'> 48 <class 'float'> 4.8
Current a and b: <class 'int'> 22 <class 'list'> [2, 5, 8, 11]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:23: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(TreeValue)
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:24: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(FastTreeValue)
Traceback (most recent call last):
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/docs/source/tutorials/advanced_usage/strict_demo_show.demox.py", line 35, in <module>
    print('plus(t1, t3):', plus(t1, t3))
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/func/func.py", line 56, in _new_func
    return _treelized(*args, **kwargs)
  File "treevalue/tree/func/cfunc.pyx", line 181, in treevalue.tree.func.cfunc._w_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 156, in treevalue.tree.func.cfunc._c_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 76, in treevalue.tree.func.cfunc._c_func_treelize_run
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/docs/source/tutorials/advanced_usage/strict_demo_show.demox.py", line 9, in plus
    return a + b
TypeError: unsupported operand type(s) for +: 'int' and 'list'

In strict mode, missing or overage of keys are not tolerated at all. So in some of the cases, especially when you can not suppose that all the trees involved in calculation has exactly the same structure. Based on this demand, left mode, inner mode and outer mode is designed, and they will be introduced with details and examples in the next 3 sections.

Left Mode

In left mode, the result tree’s key set will be aligned to the first tree from the left.

In the trees in left_demo_1, t3 is the plus result of t1 and t2. We can see that t2.a is ignored because of t1.a ‘s non-existence. Finally no error will be raised, the calculation will be processed properly with the tree structure of t1.

../../_images/left_demo_1.gv.svg

But in the left_demo_2 (actually it is exactly the same as strict_demo_2), KeyError will be raised because t2.x.f` not found in tree t2`, so the result \ can not be aligned to tree ``t1, calculation failed.

../../_images/left_demo_2.gv.svg

When you decorate a function as left mode, the decorated function will try to find the first tree from left. If there is no less than 1 positional arguments detected, the first positional argument will be used as the left tree. Otherwise, the tree with the smallest lexicographic key will be assigned as the left tree. Obviously, As least 1 positional or key-word-based argument should be in the arguments.

Here is a real code example of left mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from treevalue import FastTreeValue, func_treelize


@func_treelize(mode='left')
def plus(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5}})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})
    t4 = FastTreeValue({'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5, 'f': 6}})

    print('plus(t1, t2):', plus(t1, t2))
    print('plus(t4, t2):', plus(t4, t2))

The stdout and stderr should like below, a KeyError is raised at the second print statement.

1
2
3
4
5
6
plus(t1, t2): <TreeValue 0x7fa288b52b20>
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7fa288bd5ee0>
    ├── 'c' --> 33
    ├── 'd' --> 52
    └── 'e' --> 59

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:23: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(TreeValue)
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:24: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(FastTreeValue)
Traceback (most recent call last):
  File "treevalue/tree/func/cfunc.pyx", line 94, in treevalue.tree.func.cfunc._c_func_treelize_run
KeyError: 'f'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/docs/source/tutorials/advanced_usage/left_demo.demox.py", line 15, in <module>
    print('plus(t4, t2):', plus(t4, t2))
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/func/func.py", line 56, in _new_func
    return _treelized(*args, **kwargs)
  File "treevalue/tree/func/cfunc.pyx", line 181, in treevalue.tree.func.cfunc._w_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 156, in treevalue.tree.func.cfunc._c_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 107, in treevalue.tree.func.cfunc._c_func_treelize_run
KeyError: "Missing is off, key 'f' not found in {'c': 30, 'd': 48, 'e': 54}."

In the code example above, the result is aligned to tree a in wrapped function plus. So the first plus calculation will get the proper result, but the second one will fail.

Inner Mode

In inner mode, the result tree’s key set will be aligned to the intersection set of all the tree’s key set in arguments.

In the trees in inner_demo_1, t3 is the plus result of t1 and t2. We can see that t2.a and t1.x.f is ignored because of t1.a and t2.x.f’s non-existence. Finally no error will be raised, the calculation will be processed properly with the tree structure of the intersection between tree t1 and t2.

../../_images/inner_demo_1.gv.svg

Note

In most cases of inner mode, the calculation will be processed properly because of the intersection operation on the key set and key missing or overage are both avoided completely.

But the shortage of inner mode is the loss of some information, for keys and its value or subtrees will be directly ignored without any warning. So before using inner mode, please confirm that you know its running logic and the missing of some nodes.

Here is a real code example of inner mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from treevalue import FastTreeValue, func_treelize


@func_treelize(mode='inner')
def plus(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5}})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})
    t4 = FastTreeValue({'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5, 'f': 6}})

    print('plus(t1, t2):', plus(t1, t2))
    print('plus(t4, t2):', plus(t4, t2))

The stdout (no stderr content in this case) should like below, no errors occurred with exit code 0.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
plus(t1, t2): <TreeValue 0x7f4eed537b20>
├── 'a' --> 12
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7f4eed594ee0>
    ├── 'c' --> 33
    ├── 'd' --> 52
    └── 'e' --> 59

plus(t4, t2): <TreeValue 0x7f4eed594ee0>
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7f4eed537b20>
    ├── 'c' --> 33
    ├── 'd' --> 52
    └── 'e' --> 59

Outer Mode

In outer mode, the result tree’s key set will be aligned to the union set of the tree’s key set in arguments.

In the trees in outer_demo_1, t3 is the plus result of t1 and t2, with the missing value of 0. We can see that all the values in tree t1 and t2 are involved in this calculation, and because of the missing of t1.a and t2.x.f they will be actually treated as 0 in calculation process. Finally no error will be raised, and sum of the tree nodes’ value will form the result tree t3.

../../_images/outer_demo_1.gv.svg

Note

In outer mode, it is strongly recommended to set the value of argument missing. When missing’s value is not set and there are key missing in some of the argument trees, KeyError will be raised due to this missing and value of missing.

For further information and examples of missing argument, take a look at Process Missing Values.

Here is a real code example of inner mode.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from treevalue import FastTreeValue, func_treelize


# missing value is very important when use outer mode
@func_treelize(mode='outer', missing=0)
def plus(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5, 'f': 6}})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})

    print('plus(t1, t2):', plus(t1, t2))

The stdout (no stderr content in this case) should like below, no errors occurred with exit code 0.

1
2
3
4
5
6
7
8
plus(t1, t2): <TreeValue 0x7f94d0b6eeb0>
├── 'a' --> 11
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7f94d0fd5ee0>
    ├── 'c' --> 33
    ├── 'd' --> 52
    ├── 'e' --> 59
    └── 'f' --> 6

Inheriting on Trees

Inheriting is another important feature of treelize functions. It will be introduced in this section with necessary examples.

In some cases, we need to calculation between values with different dimensions, such as in numpy, you can add primitive integer with a ndarray object, like this

1
2
3
4
5
6
7
import os

import numpy as np

if __name__ == '__main__':
    ar1 = np.array([[1, 2], [3, 4]])
    print('ar1 + 9:', ar1 + 9, sep=os.linesep)

The result will be like below, result of ar1 + 9 is actually the same as the result of expression ar1 + np.array([[9, 9], [9, 9]]).

1
2
3
ar1 + 9:
[[10 11]
 [12 13]]

In treelize functions, you can achieve this processing way by use the inherit argument. Considering this situation can be find almost anywhere, so the default value of inherit argument is True, which means the inheriting option is on by default.

We can see the example in inherit_demo_1, the tree t1 and tree t2 have different structures, and cannot be processed by any modes in Function Modes, because t1.x is a primitive integer value but t2.x is a subtree node, their types are structurally different.

../../_images/inherit_demo_1.gv.svg

But when inheriting is enabled, this adding operation can still be carried on, for the value of t1.x (is 9 in inherit_demo_1) will be applied to the whole subtree t2.x, like the result tree t3 shows. The result of tree t3.x can be considered as the sum of primitive integer 9 and subtree t2.x.

Based on the structural properties above, in the example of inherit_demo_2, a primitive value can be directly added with a complete tree, like the result tree t2 = t1 + 5 shows.

../../_images/inherit_demo_2.gv.svg

When inheriting is disabled, all the cases with primitive value and tree node at the same path position will cause TypeError. For example, when inheriting is disabled, inherit_demo_1 and inherit_demo_2 will failed, but strict_demo_1 will still success because no primitive value appears at the same path position of tree node.

../../_images/strict_demo_1.gv.svg

Here is a real code example of inheriting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from treevalue import FastTreeValue, func_treelize


@func_treelize(mode='strict')
def plus(a, b):
    return a + b


@func_treelize(mode='strict', inherit=False)
def plusx(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': 9})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})

    print('plus(t1, t2):', plus(t1, t2))
    print('plus(t2, 5):', plus(t2, 5))
    print()

    print('plusx(t1, t2):', plusx(t1, t2))
    print()

The stdout and stderr should be like below, the the calculation of plus function and the first calculation of plusx function will be processed properly, but the last one will failed due to the disablement of inheriting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
plus(t1, t2): <TreeValue 0x7f0871595ee0>
├── 'a' --> 12
├── 'b' --> 24
└── 'x' --> <TreeValue 0x7f08711c17c0>
    ├── 'c' --> 39
    ├── 'd' --> 57
    └── 'e' --> 63

plus(t2, 5): <TreeValue 0x7f087112ff70>
├── 'a' --> 16
├── 'b' --> 27
└── 'x' --> <TreeValue 0x7f08711c17c0>
    ├── 'c' --> 35
    ├── 'd' --> 53
    └── 'e' --> 59

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:23: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(TreeValue)
/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/integration/torch.py:24: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  register_for_torch(FastTreeValue)
Traceback (most recent call last):
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/docs/source/tutorials/advanced_usage/inherit_demo.demox.py", line 22, in <module>
    print('plusx(t1, t2):', plusx(t1, t2))
  File "/tmp/tmp4q2ykbbk/66de8e06e92e92c7452902eb8564aeac8e1de431/treevalue/tree/func/func.py", line 56, in _new_func
    return _treelized(*args, **kwargs)
  File "treevalue/tree/func/cfunc.pyx", line 181, in treevalue.tree.func.cfunc._w_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 156, in treevalue.tree.func.cfunc._c_func_treelize_run
  File "treevalue/tree/func/cfunc.pyx", line 117, in treevalue.tree.func.cfunc._c_func_treelize_run
TypeError: Inherit is off, tree value expected but 'int' found in args 0.

Note

In most cases, disablement of inheriting is not recommended because it will cause some errors. Disable inheriting can increase the structural strictness of calculation, but all the cross-dimensional tree operation will be forbidden as well.

So before disabling inheriting, make sure you know well about what you are going to do.

Process Missing Values

In some cases of left mode and outer mode, some of keys in result tree can not find the node nor value at the corresponding path position in some of the argument trees. In order the make these cases be processable, missing value can be used by setting value or lambda expression in missing argument.

Just like the example outer_demo_1 mentioned forwards, when t1.a and t2.x.f not found in trees, the given missing value 0 will be used instead.

../../_images/outer_demo_1.gv.svg

Another case is the missing_demo_1 which is based on left mode. In this case, t2.a is ignore due to the property of left mode, t2.x.f is missing but t1.x.f can be found, so t2.x.f will use the missing value 0. Finally, t3 will be the result of left addition of t1 and t2, with missing value of 0.

../../_images/missing_demo_1.gv.svg

Considering another complex case, when the values of tree are primitive lists, like the missing_demo_2 which is based on outer mode. At this time, if we need to do addition with missing value supported, we can set the missing value to an empty list. The result is like below.

../../_images/missing_demo_2.gv.svg

Note

In missing_demo_2, lambda :[] will be the value of missing argument in actual code.

In missing argument, its value will be automatically executed when it is callable, and its execution’s return value will be the actual missing value.

By passing a lambda expression in, you can construct a new object everytime missing value is required, and the duplication of the missing value’s object will be avoided. So it is not recommended to use missing argument like [], {} or myobject() in python code, but lambda :[], lambda :{} and lambda :myobject() are better practices.

Here is a real code example of missing_value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import os

from treevalue import func_treelize, FastTreeValue


@func_treelize(mode='outer', missing=lambda: [])
def plus(a, b):
    return a + b


if __name__ == '__main__':
    t1 = FastTreeValue({
        'b': [2, 3],
        'x': {
            'c': [5],
            'd': [7, 11, 13],
            'e': [17, 19],
        }
    })
    t2 = FastTreeValue({
        'a': [23],
        'b': [29, 31],
        'x': {
            'c': [37],
            'd': [41, 43],
        }
    })

    print('plus(t1, t2):', plus(t1, t2), sep=os.linesep)

The stdout (no error occurred) should be like below, the calculation of plus function will be processed properly, like the result in missing_demo_2.

1
2
3
4
5
6
7
8
plus(t1, t2):
<TreeValue 0x7fbb3ff6ef40>
├── 'a' --> [23]
├── 'b' --> [2, 3, 29, 31]
└── 'x' --> <TreeValue 0x7fbb3ff6eeb0>
    ├── 'c' --> [5, 37]
    ├── 'd' --> [7, 11, 13, 41, 43]
    └── 'e' --> [17, 19]

Note

Missing value will be only applied in left mode and outer mode.

In strict mode, missing or overage of keys are absolutely not tolerated, missing value will make no sense and a RuntimeWarning will be logged.

In inner mode, missing value will never be actually in use because all the key sets of final result are intersection set of argument trees. There will never be any key missing cases at all when inner mode is used, a RuntimeWarning will be logged with the usage of missing value in inner mode.

Functional Utilities

In the primitive python, some convenient functional operations are supported, such as

1
2
3
4
5
6
if __name__ == '__main__':
    # map function
    print("Result of map:", list(map(lambda x: x + 1, [2, 3, 5, 7])))

    # filter function
    print("Result of filter:", list(filter(lambda x: x % 4 == 3, [2, 3, 5, 7])))

The result should be

1
2
Result of map: [3, 4, 6, 8]
Result of filter: [3, 7]

Actually, functional operations are supported in tree values as well, and they will be introduced in this section.

Mapping

Mapping function is similar to the primitive map function. The relation between TreeValue and mapping function is like that between list and map function.

Here is a simple real code example

1
2
3
4
5
6
7
from treevalue import mapping, FastTreeValue

if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})

    print('mapping(t, lambda x: x ** x + 2):')
    print(mapping(t, lambda x: x ** x + 2))

The mapped tree result should be like below.

1
2
3
4
5
6
7
mapping(t, lambda x: x ** x + 2):
<FastTreeValue 0x7f82549b7b20>
├── 'a' --> 3
├── 'b' --> 6
└── 'x' --> <FastTreeValue 0x7f8254c6dee0>
    ├── 'c' --> 29
    └── 'd' --> 258

For further definition or source code implement of function mapping, take a look at mapping.

Filter

Filter function is similar to the primitive filter function. The relation between TreeValue and filter_ function is like that between list and filter function .

Here is a simple real code example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from treevalue import FastTreeValue, filter_

if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}, 'y': {'e': 6, 'f': 8}})

    print('filter_(t, lambda x: x % 2 == 1):')
    print(filter_(t, lambda x: x % 2 == 1))

    print('filter_(t, lambda x: x % 2 == 1, remove_empty=False):')
    print(filter_(t, lambda x: x % 2 == 1, remove_empty=False))

The filtered tree result should be like below. When remove_empty is disabled, the empty tree node will be kept to make sure the structure of the result tree is similar to the unfiltered tree.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
filter_(t, lambda x: x % 2 == 1):
<FastTreeValue 0x7f12897eef70>
├── 'a' --> 1
└── 'x' --> <FastTreeValue 0x7f1289ebeee0>
    └── 'c' --> 3

filter_(t, lambda x: x % 2 == 1, remove_empty=False):
<FastTreeValue 0x7f12897eef40>
├── 'a' --> 1
├── 'x' --> <FastTreeValue 0x7f1289ebeee0>
│   └── 'c' --> 3
└── 'y' --> <FastTreeValue 0x7f12897eef70>

Note

Actually, the code above is equal to the code below. The filter_ function can be seen as a combination of mask function and mapping function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from treevalue import FastTreeValue, mapping, mask

if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}, 'y': {'e': 6, 'f': 8}})

    print('mask(t, mapping(t, lambda x: x % 2 == 1)):')
    print(mask(t, mapping(t, lambda x: x % 2 == 1)))

    print('mask(t, mapping(t, lambda x: x % 2 == 1), remove_empty=False):')
    print(mask(t, mapping(t, lambda x: x % 2 == 1), remove_empty=False))

The result should be like below, exactly the same as the code above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
mask(t, mapping(t, lambda x: x % 2 == 1)):
<FastTreeValue 0x7fd7a85aeee0>
├── 'a' --> 1
└── 'x' --> <FastTreeValue 0x7fd7a85aef40>
    └── 'c' --> 3

mask(t, mapping(t, lambda x: x % 2 == 1), remove_empty=False):
<FastTreeValue 0x7fd7a7a5adf0>
├── 'a' --> 1
├── 'x' --> <FastTreeValue 0x7fd7a85aef70>
│   └── 'c' --> 3
└── 'y' --> <FastTreeValue 0x7fd7a85aef10>

For further information about mask function, take a look at Mask.

For further definition or source code implement of function filter_, take a look at filter_.

Mask

Mask function allow your choice in one tree, by another tree of true-or-false flags. A simple code example is

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from treevalue import FastTreeValue, mask

if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}, 'y': {'e': 6, 'f': 8}})
    m = FastTreeValue({'a': True, 'b': False, 'x': {'c': True, 'd': True}, 'y': {'e': False, 'f': False}})

    print('mask(t, m):')
    print(mask(t, m))

    print('mask(t, m, remove_empty=False):')
    print(mask(t, m, remove_empty=False))

The result will be like below, values mapped with the tree m which mask flag is False are all deleted. Also, remove_empty argument is supported.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
mask(t, m):
<FastTreeValue 0x7fb92efff250>
├── 'a' --> 1
└── 'x' --> <FastTreeValue 0x7fb92efd5ee0>
    ├── 'c' --> 3
    └── 'd' --> 4

mask(t, m, remove_empty=False):
<FastTreeValue 0x7fb92e31adc0>
├── 'a' --> 1
├── 'x' --> <FastTreeValue 0x7fb92efff250>
│   ├── 'c' --> 3
│   └── 'd' --> 4
└── 'y' --> <FastTreeValue 0x7fb92efd5ee0>

For further definition or source code implement of function mask, take a look at mask.

Reduce

By using reduce_ function, you can get some calculation result based on the tree structure. Its meaning is similar to primitive reduce_ function, but its base structure is TreeValue instead of sequence or iterator. For example, we can get the sum and multiply accumulation of the values in the tree.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from functools import reduce
from operator import __mul__

from treevalue import reduce_, FastTreeValue


def multi(items):
    return reduce(__mul__, items, 1)


if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}, 'y': {'e': 6, 'f': 8}})

    print("Sum of t:", reduce_(t, lambda **kwargs: sum(kwargs.values())))
    print("Multiply of t:", reduce_(t, lambda **kwargs: multi(kwargs.values())))

The result should be like below.

1
2
Sum of t: 24
Multiply of t: 1152

If we use reduce_ function with mapping function, huffman weight sum can also be easily calculated with the code below.

1
2
3
4
5
6
7
8
9
from treevalue import reduce_, FastTreeValue, mapping

if __name__ == '__main__':
    t = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}, 'y': {'e': 6, 'f': 8}})

    weights = mapping(t, lambda v, p: v * len(p))
    print("Weight tree:", weights)
    print("Huffman weight sum of t:",
          reduce_(weights, lambda **kwargs: sum(kwargs.values())))

The result should be like below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
Weight tree: <FastTreeValue 0x7f50473eff70>
├── 'a' --> 1
├── 'b' --> 2
├── 'x' --> <FastTreeValue 0x7f5047abeee0>
│   ├── 'c' --> 6
│   └── 'd' --> 8
└── 'y' --> <FastTreeValue 0x7f50477817c0>
    ├── 'e' --> 12
    └── 'f' --> 16

Huffman weight sum of t: 45

Besides, we can easily calculate sum of the np.ndarray objects’ bytes size, like the following code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import numpy as np

from treevalue import reduce_, FastTreeValue

if __name__ == '__main__':
    t = FastTreeValue({
        'a': np.identity(3),
        'b': np.array([[1, 2], [3, 4]]),
        'x': {
            'c': np.zeros(4),
            'd': np.array([[5, 6, 7], [8, 9, 10]])
        },
    })

    print("Size tree:", t.nbytes)
    print("Total bytes of arrays in t:",
          reduce_(t.nbytes, lambda **kwargs: sum(kwargs.values())))

The result should be like below.

1
2
3
4
5
6
7
8
Size tree: <FastTreeValue 0x7ff7812beee0>
├── 'a' --> 72
├── 'b' --> 32
└── 'x' --> <FastTreeValue 0x7ff780ff7b20>
    ├── 'c' --> 32
    └── 'd' --> 48

Total bytes of arrays in t: 184

For further definition or source code implement of function reduce_, take a look at reduce_.

Structural Utilities

In order to process some structured data, especially when you need to process a sort of TreeValue objects which is in the primitive collections or mappings, the structural utilities are designed, and they will be introduced in this section with examples.

Union

The union function is similar to the primitive zip function some ways, it can combine a list of TreeValue objects as one TreeValue object which leaf values are tuples, like the simple example below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from treevalue import FastTreeValue, union

if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4, 'e': 5}})
    t2 = FastTreeValue({'a': 11, 'b': 22, 'x': {'c': 30, 'd': 48, 'e': 54}})
    t3 = FastTreeValue({'a': -13, 'b': -7, 'x': {'c': -5, 'd': -3, 'e': -2}})
    t4 = FastTreeValue({'a': -13, 'b': -7, 'x': 8})

    print("union(t1, t2):")
    print(union(t1, t2))

    print("union(t1, t2, t3):")
    print(union(t1, t2, t3))

    print("union(t1, t2, t3, t4):")
    print(union(t1, t2, t3, t4))

The result should be like below, the leaf values are unionised as tuples.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
union(t1, t2):
<FastTreeValue 0x7f2d0f32d1c0>
├── 'a' --> (1, 11)
├── 'b' --> (2, 22)
└── 'x' --> <FastTreeValue 0x7f2d0f314ee0>
    ├── 'c' --> (3, 30)
    ├── 'd' --> (4, 48)
    └── 'e' --> (5, 54)

union(t1, t2, t3):
<FastTreeValue 0x7f2d0f314ee0>
├── 'a' --> (1, 11, -13)
├── 'b' --> (2, 22, -7)
└── 'x' --> <FastTreeValue 0x7f2d0f32d1c0>
    ├── 'c' --> (3, 30, -5)
    ├── 'd' --> (4, 48, -3)
    └── 'e' --> (5, 54, -2)

union(t1, t2, t3, t4):
<FastTreeValue 0x7f2d0f32d1c0>
├── 'a' --> (1, 11, -13, -13)
├── 'b' --> (2, 22, -7, -7)
└── 'x' --> <FastTreeValue 0x7f2d0f314ee0>
    ├── 'c' --> (3, 30, -5, 8)
    ├── 'd' --> (4, 48, -3, 8)
    └── 'e' --> (5, 54, -2, 8)

Note

In union function (actually subside function has the same property), all the trees to be unionised need to have the same structure. You can just consider it as strict mode with inheriting is enabled.

For further information and arguments of function union, take a look at union.

Subside

The subside function can transform the collections or mapping nested TreeValue objects into one TreeValue object which leaf values has the dispatch’s structure. The union function mentioned above is also based on the subside function. Function subside will greatly simplify the code when the structured data need to be calculated. Just like this following example code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from treevalue import TreeValue, subside

if __name__ == '__main__':
    t1 = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = TreeValue({'a': -14, 'b': 9, 'x': {'c': 3, 'd': 8}})
    t3 = TreeValue({'a': 6, 'b': 0, 'x': {'c': -5, 'd': 17}})
    t4 = TreeValue({'a': 0, 'b': -17, 'x': {'c': -8, 'd': 15}})
    t5 = TreeValue({'a': 3, 'b': 9, 'x': {'c': 11, 'd': -17}})

    st = {'first': (t1, t2), 'second': [t3, {'x': t4, 'y': t5}]}
    print("subside(st):")
    print(subside(st))

The result should be like below, the leaf values are combined as the dispatch structure of st.

1
2
3
4
5
6
7
subside(st):
<TreeValue 0x7fcd186b2220>
├── 'a' --> {'first': (1, -14), 'second': [6, {'x': 0, 'y': 3}]}
├── 'b' --> {'first': (2, 9), 'second': [0, {'x': -17, 'y': 9}]}
└── 'x' --> <TreeValue 0x7fcd17e29fa0>
    ├── 'c' --> {'first': (3, 3), 'second': [-5, {'x': -8, 'y': 11}]}
    └── 'd' --> {'first': (4, 8), 'second': [17, {'x': 15, 'y': -17}]}

Note

In function subside, only primitive list, tuple and dict (or subclasses) objects will be subsided to leaf values.

For further information and arguments of function subside, take a look at subside.

Rise

Function rise can be seen as the inverse operation of function subside, it will try to extract the greatest common structure of the leaf values, and rise them up to the dispatch above the TreeValue objects. Like the following example code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from treevalue import TreeValue, subside, rise

if __name__ == '__main__':
    # The same demo as the subside docs
    t1 = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = TreeValue({'a': -14, 'b': 9, 'x': {'c': 3, 'd': 8}})
    t3 = TreeValue({'a': 6, 'b': 0, 'x': {'c': -5, 'd': 17}})
    t4 = TreeValue({'a': 0, 'b': -17, 'x': {'c': -8, 'd': 15}})
    t5 = TreeValue({'a': 3, 'b': 9, 'x': {'c': 11, 'd': -17}})
    st = {'first': (t1, t2), 'second': (t3, {'x': t4, 'y': t5})}
    tx = subside(st)

    # Rising process
    st2 = rise(tx)
    assert st2 == st
    print('st2:', st2)

    print("st2['first'][0]:")
    print(st2['first'][0])

    print("st2['first'][1]:")
    print(st2['first'][1])

    print("st2['second'][0]:")
    print(st2['second'][0])

    print("st2['second'][1]['x']:")
    print(st2['second'][1]['x'])

    print("st2['second'][1]['y']:")
    print(st2['second'][1]['y'])

The result should be like below, the subsided tree st can be extract back to the original structure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
st2: {'first': (<TreeValue 0x7fbd465711f0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7fbd46571190>
    ├── 'c' --> 3
    └── 'd' --> 4
, <TreeValue 0x7fbd4746dee0>
├── 'a' --> -14
├── 'b' --> 9
└── 'x' --> <TreeValue 0x7fbd2ee8ddc0>
    ├── 'c' --> 3
    └── 'd' --> 8
), 'second': (<TreeValue 0x7fbd4655ae50>
├── 'a' --> 6
├── 'b' --> 0
└── 'x' --> <TreeValue 0x7fbd4655ad90>
    ├── 'c' --> -5
    └── 'd' --> 17
, {'y': <TreeValue 0x7fbd4655af10>
├── 'a' --> 3
├── 'b' --> 9
└── 'x' --> <TreeValue 0x7fbd4655ae80>
    ├── 'c' --> 11
    └── 'd' --> -17
, 'x': <TreeValue 0x7fbd2eaf6e20>
├── 'a' --> 0
├── 'b' --> -17
└── 'x' --> <TreeValue 0x7fbd45cd1c10>
    ├── 'c' --> -8
    └── 'd' --> 15
})}
st2['first'][0]:
<TreeValue 0x7fbd465711f0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7fbd46571190>
    ├── 'c' --> 3
    └── 'd' --> 4

st2['first'][1]:
<TreeValue 0x7fbd4746dee0>
├── 'a' --> -14
├── 'b' --> 9
└── 'x' --> <TreeValue 0x7fbd2ee8ddc0>
    ├── 'c' --> 3
    └── 'd' --> 8

st2['second'][0]:
<TreeValue 0x7fbd4655ae50>
├── 'a' --> 6
├── 'b' --> 0
└── 'x' --> <TreeValue 0x7fbd4655ad90>
    ├── 'c' --> -5
    └── 'd' --> 17

st2['second'][1]['x']:
<TreeValue 0x7fbd2eaf6e20>
├── 'a' --> 0
├── 'b' --> -17
└── 'x' --> <TreeValue 0x7fbd45cd1c10>
    ├── 'c' --> -8
    └── 'd' --> 15

st2['second'][1]['y']:
<TreeValue 0x7fbd4655af10>
├── 'a' --> 3
├── 'b' --> 9
└── 'x' --> <TreeValue 0x7fbd4655ae80>
    ├── 'c' --> 11
    └── 'd' --> -17

Note

In function rise, only primitive list, tuple and dict (or subclasses) objects will join the rising operation.

Actually, when template argument is not assigned, the rise function will try to find the greatest common structure and rise them to the dispatch. The following rules will be strictly followed when doing this:

  • If the values in current level have the same type ( must all be list, all be tuple or all be dict), the rising function will carry on the find sub structure, otherwise values in current level will be treated as atomic values.

  • If all of them are dicts, and they have exactly the same key sets with each other, the finding of sub structures will be carried on, otherwise these dicts will be treated as atomic values.

  • If all of them are lists, and they have the same length with each other, the finding of sub structures will be carried on, otherwise these dicts will be treated as atomic values.

  • If all of them are tuples, and they have the same length with each other, the finding of sub structures will be carried on, otherwise these dicts will be treated as atomic values. (Actually, this rule is the same as the last one which is about lists.)

Considering this automatic structure finding process, if you only want to extract some of the structure (make sure the extraction will not be too deep or too shallow, and make sure the result will have the same structure as your expectation), you can assign value in template arguments. Like the example code below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from treevalue import TreeValue, subside, rise

if __name__ == '__main__':
    # The same demo as the subside docs
    t11 = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t12 = TreeValue({'a': 3, 'b': 9, 'x': {'c': 15, 'd': 41}})
    t21 = TreeValue({'a': -14, 'b': 9, 'x': {'c': 3, 'd': 8}})
    t22 = TreeValue({'a': -31, 'b': 82, 'x': {'c': 47, 'd': 32}})
    t3 = TreeValue({'a': 6, 'b': 0, 'x': {'c': -5, 'd': 17}})
    t4 = TreeValue({'a': 0, 'b': -17, 'x': {'c': -8, 'd': 15}})
    t5 = TreeValue({'a': 3, 'b': 9, 'x': {'c': 11, 'd': -17}})
    st = {'first': [(t11, t21), (t12, t22)], 'second': (t3, {'x': t4, 'y': t5})}
    tx = subside(st)

    # Rising process, with template
    # only the top-leveled dict will be extracted,
    # neither tuple, list nor low-leveled dict will not be extracted
    # because they are not defined in `template` argument
    st2 = rise(tx, template={'first': [object, ...], 'second': (object, ...)})
    print('st2:', st2)

    print("st2['first'][0]:")
    print(st2['first'][0])

    print("st2['first'][1]:")
    print(st2['first'][1])

    print("st2['second'][0]:")
    print(st2['second'][0])

    print("st2['second'][1]:")
    print(st2['second'][1])

The result should be like below, the subsided tree st can be extract back to the structure of dict with only first and second keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
st2: {'first': [<TreeValue 0x7f3fd8471220>
├── 'a' --> (1, -14)
├── 'b' --> (2, 9)
└── 'x' --> <TreeValue 0x7f3fd7be2fd0>
    ├── 'c' --> (3, 3)
    └── 'd' --> (4, 8)
, <TreeValue 0x7f3fc0d70ca0>
├── 'a' --> (3, -31)
├── 'b' --> (9, 82)
└── 'x' --> <TreeValue 0x7f3fd8471190>
    ├── 'c' --> (15, 47)
    └── 'd' --> (41, 32)
], 'second': (<TreeValue 0x7f3fd7bc4c40>
├── 'a' --> 6
├── 'b' --> 0
└── 'x' --> <TreeValue 0x7f3fd7bc4a60>
    ├── 'c' --> -5
    └── 'd' --> 17
, <TreeValue 0x7f3fc09decd0>
├── 'a' --> {'x': 0, 'y': 3}
├── 'b' --> {'x': -17, 'y': 9}
└── 'x' --> <TreeValue 0x7f3fc09ded00>
    ├── 'c' --> {'x': -8, 'y': 11}
    └── 'd' --> {'x': 15, 'y': -17}
)}
st2['first'][0]:
<TreeValue 0x7f3fd8471220>
├── 'a' --> (1, -14)
├── 'b' --> (2, 9)
└── 'x' --> <TreeValue 0x7f3fd7be2fd0>
    ├── 'c' --> (3, 3)
    └── 'd' --> (4, 8)

st2['first'][1]:
<TreeValue 0x7f3fc0d70ca0>
├── 'a' --> (3, -31)
├── 'b' --> (9, 82)
└── 'x' --> <TreeValue 0x7f3fd8471190>
    ├── 'c' --> (15, 47)
    └── 'd' --> (41, 32)

st2['second'][0]:
<TreeValue 0x7f3fd7bc4c40>
├── 'a' --> 6
├── 'b' --> 0
└── 'x' --> <TreeValue 0x7f3fd7bc4a60>
    ├── 'c' --> -5
    └── 'd' --> 17

st2['second'][1]:
<TreeValue 0x7f3fc09decd0>
├── 'a' --> {'x': 0, 'y': 3}
├── 'b' --> {'x': -17, 'y': 9}
└── 'x' --> <TreeValue 0x7f3fc09ded00>
    ├── 'c' --> {'x': -8, 'y': 11}
    └── 'd' --> {'x': 15, 'y': -17}

For further information and arguments of function rise, take a look at rise.

Tree Utilities

In this section, utilities with TreeValue class and its objects themselves will be introduced with examples.

Jsonify

With the usage of jsonify function, you can transform a TreeValue object to json-formatted data.

For example, the following real code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import json

from treevalue import TreeValue, raw, jsonify

if __name__ == '__main__':
    t = TreeValue({'a': 1, 'b': [2, 3], 'x': {'c': raw({'x': 1, 'y': 2}), 'd': "this is a string"}})

    print("Tree t:")
    print(t)

    print("Json data of t:")
    print(json.dumps(jsonify(t), indent=4, sort_keys=True))

The result should be

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Tree t:
<TreeValue 0x7f4045ded1c0>
├── 'a' --> 1
├── 'b' --> [2, 3]
└── 'x' --> <TreeValue 0x7f4045ded460>
    ├── 'c' --> {'x': 1, 'y': 2}
    └── 'd' --> 'this is a string'

Json data of t:
{
    "a": 1,
    "b": [
        2,
        3
    ],
    "x": {
        "c": {
            "x": 1,
            "y": 2
        },
        "d": "this is a string"
    }
}

Note

The function raw in the example code above is a wrapper for dictionary object. It can be used to pass dict object as simple value in TreeValue instead of being treated as sub tree.

For further information of function raw, take a look at raw.

For further informaon of function jsonify, take a look at jsonify.

Clone

The TreeValue objects can be cloned deeply by clone function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
from treevalue import TreeValue, clone, raw

if __name__ == '__main__':
    t = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4, 'y': raw({'e': 5, 'f': 6})}})

    print("Tree t:")
    print(t)
    print("Id of t.x.y: %x" % id(t.x.y))
    print()
    print()

    print("clone(t):")
    print(clone(t))
    print("Id of clone(t).x.y: %x" % id(clone(t).x.y))
    print()
    print()

    print('clone(t, copy_value=True):')
    print(clone(t, copy_value=True))
    print("Id of clone(t, copy_value=True).x.y: %x" % id(clone(t, copy_value=True)))
    print()
    print()

The result will be like below, all the memory address of the tree nodes in cloned tree are different from those in the original tree t when copy_value argument is not set. But when copy_value is assigned as True, the address of values will be changed because of the deep copy of value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Tree t:
<TreeValue 0x7f12c2b41a30>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7f12c27aef70>
    ├── 'c' --> 3
    ├── 'd' --> 4
    └── 'y' --> {'e': 5, 'f': 6}

Id of t.x.y: 7f12c2e91280


clone(t):
<TreeValue 0x7f12c27aeeb0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7f12c2e6dee0>
    ├── 'c' --> 3
    ├── 'd' --> 4
    └── 'y' --> {'e': 5, 'f': 6}

Id of clone(t).x.y: 7f12c2e91280


clone(t, copy_value=True):
<TreeValue 0x7f12c2b417c0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7f12c27aeeb0>
    ├── 'c' --> 3
    ├── 'd' --> 4
    └── 'y' --> {'e': 5, 'f': 6}

Id of clone(t, copy_value=True).x.y: 7f11facc86d0

Note

Attention that in function clone, the values will not be deeply copied together with the tree nodes by default. The newly cloned tree’s values have the same memory address with those in original tree.

If deep copy of values is required when using clone, copy_value argument need to be assigned as True. And then a copy.deepcopy will be performed in clone function in order to do the deep copy. Also, you can define your own copy function in copy_value argument by just assign it as a lambda expression like lambda x: pickle.loads(pickle.dumps(x)).

For further informaon of function clone, take a look at clone.

Typetrans

You can use function typetrans to change the type of tree value object, just like the example below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
from treevalue import TreeValue, method_treelize, FastTreeValue, typetrans


class MyTreeValue(TreeValue):
    @method_treelize()
    def pw(self):
        return (self + 1) * (self + 2)


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    print('t1:')
    print(t1)
    print('t1 ** 2:')
    print(t1 ** 2)
    print()

    # Transform t1 to MyTreeValue,
    # __pow__ operator will be disabled and pw method will be enabled.
    t2 = typetrans(t1, MyTreeValue)
    print('t2:')
    print(t2)
    print('t2.pw():')
    print(t2.pw())
    print()

The result will be like below. After the type transformation, ** operator can not be used in t2 but method pw which is implemented in MyTreeValue will be enabled.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
t1:
<FastTreeValue 0x7f0b303017c0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <FastTreeValue 0x7f0b303ff250>
    ├── 'c' --> 3
    └── 'd' --> 4

t1 ** 2:
<FastTreeValue 0x7f0b303d4ee0>
├── 'a' --> 1
├── 'b' --> 4
└── 'x' --> <FastTreeValue 0x7f0b30377b20>
    ├── 'c' --> 9
    └── 'd' --> 16


t2:
<MyTreeValue 0x7f0b303017c0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <MyTreeValue 0x7f0b303ff250>
    ├── 'c' --> 3
    └── 'd' --> 4

t2.pw():
<MyTreeValue 0x7f0b303d4ee0>
├── 'a' --> 6
├── 'b' --> 12
└── 'x' --> <MyTreeValue 0x7f0b30377b20>
    ├── 'c' --> 20
    └── 'd' --> 30

Note

In treevalue library, there can be more classes based on TreeValue class by your definition.

Different tree value class will have the different operators, methods and class methods for service, and the __eq__ operator’s result between different type of tree values will always be False because of the difference of their types.

For further information of TreeValue class and user’s definition practice, just take a look at:

For further informaon of function typetrans, take a look at typetrans.

Walk

You can use function treevalue.tree.tree.walk() to iterate all the nodes in the tree, like the example below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from treevalue import TreeValue, raw, walk

if __name__ == '__main__':
    t = TreeValue({
        'a': 1,
        'b': 2,
        'c': raw({'x': 3, 'y': 4}),
        'd': {
            'x': 3,
            'y': 4
        },
    })

    for path, node in walk(t):
        print(path, '-->', node)

The output should be like below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
() --> <TreeValue 0x7f60eca9ca30>
├── 'a' --> 1
├── 'b' --> 2
├── 'c' --> {'x': 3, 'y': 4}
└── 'd' --> <TreeValue 0x7f60ec7aef70>
    ├── 'x' --> 3
    └── 'y' --> 4

('a',) --> 1
('b',) --> 2
('c',) --> {'x': 3, 'y': 4}
('d',) --> <TreeValue 0x7f60ec7aef70>
├── 'x' --> 3
└── 'y' --> 4

('d', 'x') --> 3
('d', 'y') --> 4

For further informaon of function walk, take a look at walk.

Flatten Utilities

In order to support the parallel calculation in values of TreeValue object, treevalue.tree.tree.flatten() and treevalue.tree.tree.unflatten() is provided to dump a sequence of nodes’ information and value, and recover it to original tree structure when calculation is completed.

Flatten

The values and node structures can be dumped by function treevalue.tree.tree.flatten() as a list, like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from treevalue import TreeValue, raw, flatten

if __name__ == '__main__':
    t = TreeValue({
        'a': 1,
        'b': 2,
        'c': raw({'x': 3, 'y': 4}),
        'd': {
            'x': 3,
            'y': 4
        },
    })

    print('flatten(t):')
    print(flatten(t))

The result should be like below:

1
2
flatten(t):
[(('a',), 1), (('b',), 2), (('c',), {'x': 3, 'y': 4}), (('d', 'x'), 3), (('d', 'y'), 4)]

Note

Function treevalue.tree.tree.flatten() is different from treevalue.tree.tree.walk() because flatten has fewer features because it is designed entirely for the dumping process before parallel calculation.

So when you need to do parallel calculation, please use treevalue.tree.tree.flatten() to dump the values to ensure the speed performance.

For further informaon of function flatten, take a look at flatten.

Unflatten

The dumped nodes’ information and value can be recovered to a TreeValue object by function treevalue.tree.tree.unflatten(), like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from treevalue import unflatten

if __name__ == '__main__':
    flatted = [
        (('a',), 1),
        (('b',), 2),
        (('c',), {'x': 3, 'y': 4}),
        (('d', 'x'), 3),
        (('d', 'y'), 4)
    ]

    print('unflatten(flatted):')
    print(unflatten(flatted))

The result should be like below:

1
2
3
4
5
6
7
8
unflatten(flatted):
<TreeValue 0x7ff3083f7b20>
├── 'a' --> 1
├── 'b' --> 2
├── 'c' --> {'x': 3, 'y': 4}
└── 'd' --> <TreeValue 0x7ff3083807c0>
    ├── 'x' --> 3
    └── 'y' --> 4

Note

It is recommended to pass an ordered iterable object in pairs, this will improve the speed performance of function treevalue.tree.tree.unflatten().

Because of this, it is a good idea to keep the treevalue.tree.tree.flatten()’s result’s order when executing your own parallel processing logic.

For further informaon of function unflatten, take a look at unflatten.

IO Utilities

Simple Serialize and Deserialize

In this version, pickle.dumps and pickle.loads operations are supported, just like the code below.

import os
import pickle

from treevalue import TreeValue

if __name__ == '__main__':
    t = TreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': [3, 4]}})
    binary = pickle.dumps(t)
    print('t:', t, sep=os.linesep)

    tx = pickle.loads(binary)
    assert tx == t
    print('tx:', tx, sep=os.linesep)

The output is

t:
<TreeValue 0x7f9753dc07c0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7f9753dc0a30>
    ├── 'c' --> 3
    └── 'd' --> [3, 4]

tx:
<TreeValue 0x7f9754060b20>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <TreeValue 0x7f97540bdee0>
    ├── 'c' --> 3
    └── 'd' --> [3, 4]

Advanced Dump and Load

In this project, we implemented our load and dump function which can be used like this

import os
import tempfile

from treevalue import TreeValue, dump, load

if __name__ == '__main__':
    t = TreeValue({'a': 'ab', 'b': 'Cd', 'x': {'c': 'eF', 'd': 'GH'}})
    print('t:', t, sep=os.linesep)

    with tempfile.NamedTemporaryFile() as tf:
        with open(tf.name, 'wb') as wf:  # dump t to file
            dump(t, file=wf)

        with open(tf.name, 'rb') as rf:  # load dt from file
            dt = load(file=rf)

    assert dt == t
    print('dt:', dt, sep=os.linesep)

The output should be

t:
<TreeValue 0x7fefe6b817c0>
├── 'a' --> 'ab'
├── 'b' --> 'Cd'
└── 'x' --> <TreeValue 0x7fefe6b81a30>
    ├── 'c' --> 'eF'
    └── 'd' --> 'GH'

dt:
<TreeValue 0x7fefe56d7c70>
├── 'a' --> 'ab'
├── 'b' --> 'Cd'
└── 'x' --> <TreeValue 0x7fefe56d7a90>
    ├── 'c' --> 'eF'
    └── 'd' --> 'GH'

Note

In this load (loads function is the same), you must assign a TreeValue class by the type_ argument because the dumped data will not save the type of trees. When the type is not assigned, the default class will be simply TreeValue.

Also, dumps function and loads function are provided to deal with binary data.

import os

from treevalue import dumps, FastTreeValue, loads

if __name__ == '__main__':
    t = FastTreeValue({'a': 'ab', 'b': 'Cd', 'x': {'c': 'eF', 'd': 'GH'}})
    print('t:', t, sep=os.linesep)

    binary = dumps(t)
    dt = loads(binary, type_=FastTreeValue)

    assert dt == t
    print('dt:', dt, sep=os.linesep)

The output should be

t:
<FastTreeValue 0x7f0d0e1c17c0>
├── 'a' --> 'ab'
├── 'b' --> 'Cd'
└── 'x' --> <FastTreeValue 0x7f0d0e1c1a30>
    ├── 'c' --> 'eF'
    └── 'd' --> 'GH'

dt:
<FastTreeValue 0x7f0d0d8dae20>
├── 'a' --> 'ab'
├── 'b' --> 'Cd'
└── 'x' --> <FastTreeValue 0x7f0d0d8f1190>
    ├── 'c' --> 'eF'
    └── 'd' --> 'GH'

Note

The main differences between our dump / load and pickle ‘s dump / load are:

  • Ours are based on dill project, it can serialize and deserialize more types than native pickle, such as functions and classes.

  • Compression and decompression can be defined when dumping, and when the compressed binary data is going to be loaded, decompression function is not necessary, the decompression can be carried on with the function defined in dumping process.

Here is an example:

import gzip
import os

from treevalue import FastTreeValue, dumps, loads

if __name__ == '__main__':
    t = FastTreeValue({'a': 'ab', 'b': 'Cd', 'x': {'c': 'eF', 'd': 'GH'}})
    st = t.upper
    print('st:', st, sep=os.linesep)  # st is a function tree
    print('st():', st(), sep=os.linesep)  # st() if an upper-case-string tree
    print()

    binary = dumps(st, compress=(gzip.compress, gzip.decompress))
    print('Length of binary:', len(binary))

    # compression function is not needed here
    dst = loads(binary, type_=FastTreeValue)
    print('dst:', dst, sep=os.linesep)
    assert st() == dst()  # st() should be equal to dst()
    print('dst():', dst(), sep=os.linesep)

The output should be

st:
<FastTreeValue 0x7f3692f807c0>
├── 'a' --> <built-in method upper of str object at 0x7f3692b2d0b0>
├── 'b' --> <built-in method upper of str object at 0x7f369326c5f0>
└── 'x' --> <FastTreeValue 0x7f3692ff7b20>
    ├── 'c' --> <built-in method upper of str object at 0x7f3692fe26f0>
    └── 'd' --> <built-in method upper of str object at 0x7f3692fe2830>

st():
<FastTreeValue 0x7f36923bcac0>
├── 'a' --> 'AB'
├── 'b' --> 'CD'
└── 'x' --> <FastTreeValue 0x7f36932bdee0>
    ├── 'c' --> 'EF'
    └── 'd' --> 'GH'


Length of binary: 213
dst:
<FastTreeValue 0x7f36923b4a90>
├── 'a' --> <built-in method upper of str object at 0x7f367a92aa70>
├── 'b' --> <built-in method upper of str object at 0x7f35cb108db0>
└── 'x' --> <FastTreeValue 0x7f36923b4730>
    ├── 'c' --> <built-in method upper of str object at 0x7f35cb108d30>
    └── 'd' --> <built-in method upper of str object at 0x7f35cb1087f0>

dst():
<FastTreeValue 0x7f3691b13a60>
├── 'a' --> 'AB'
├── 'b' --> 'CD'
└── 'x' --> <FastTreeValue 0x7f367acb4d00>
    ├── 'c' --> 'EF'
    └── 'd' --> 'GH'

Object Oriented Usage

In FastTreeValue class, plenty of object-oriented operators, methods and classmethods are supported in order to simplify the actual usage. Here is a simple code example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
from treevalue import FastTreeValue

if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = FastTreeValue({'a': 5, 'b': 6, 'x': {'c': 7, 'd': 8}})

    # operator support
    print('t1 + t2:')
    print(t1 + t2)
    print('t2 ** t1:')
    print(t2 ** t1)
    print()

    # utilities support
    print('t1.map(lambda x: (x + 1) * (x + 2)):')
    print(t1.map(lambda x: (x + 1) * (x + 2)))
    print('t1.reduce(lambda **kwargs: sum(kwargs.values())):',
          t1.reduce(lambda **kwargs: sum(kwargs.values())))
    print()
    print()

    # linking usage
    print('t1.map(lambda x: (x + 1) * (x + 2)).filter(lambda x: x % 4 == 0):')
    print(t1.map(lambda x: (x + 1) * (x + 2)).filter(lambda x: x % 4 == 0))
    print()

    # structural support
    print("Union result:")
    print(FastTreeValue.union(
        t1.map(lambda x: (x + 1) * (x + 2)),
        t2.map(lambda x: (x - 2) ** (x - 1)),
    ).map(lambda x: 'first: %d, second: %d, which sum is %d' % (x[0], x[1], sum(x))))
    print()

The result should be like below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
t1 + t2:
<FastTreeValue 0x7fd43bcd2b20>
├── 'a' --> 6
├── 'b' --> 8
└── 'x' --> <FastTreeValue 0x7fd43bd54ee0>
    ├── 'c' --> 10
    └── 'd' --> 12

t2 ** t1:
<FastTreeValue 0x7fd43bcd2b20>
├── 'a' --> 5
├── 'b' --> 36
└── 'x' --> <FastTreeValue 0x7fd43bd54ee0>
    ├── 'c' --> 343
    └── 'd' --> 4096


t1.map(lambda x: (x + 1) * (x + 2)):
<FastTreeValue 0x7fd43bd54ee0>
├── 'a' --> 6
├── 'b' --> 12
└── 'x' --> <FastTreeValue 0x7fd43bcd2b20>
    ├── 'c' --> 20
    └── 'd' --> 30

t1.reduce(lambda **kwargs: sum(kwargs.values())): 10


t1.map(lambda x: (x + 1) * (x + 2)).filter(lambda x: x % 4 == 0):
<FastTreeValue 0x7fd43b96ef10>
├── 'b' --> 12
└── 'x' --> <FastTreeValue 0x7fd43b96eee0>
    └── 'c' --> 20


Union result:
<FastTreeValue 0x7fd43bcd2b20>
├── 'a' --> 'first: 6, second: 81, which sum is 87'
├── 'b' --> 'first: 12, second: 1024, which sum is 1036'
└── 'x' --> <FastTreeValue 0x7fd43bd54ee0>
    ├── 'c' --> 'first: 20, second: 15625, which sum is 15645'
    └── 'd' --> 'first: 30, second: 279936, which sum is 279966'

For further information of FastTreeValue class, take a look at FastTreeValue, all the supported methods and operators are listed here.

DIY TreeValue Class

You can define your own TreeValue class with your own method and class methods. Like the example code below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import os

from treevalue import classmethod_treelize, TreeValue, method_treelize


class MyTreeValue(TreeValue):
    # return type will be automatically detected as `MyTreeValue`
    @method_treelize()
    def append(self, b):
        print("Append arguments:", self, b)
        return self + b

    # return type will be automatically detected as `MyTreeValue`
    @classmethod
    @classmethod_treelize()
    def sum(cls, *args):
        print("Sum arguments:", cls, *args)
        return sum(args)


if __name__ == '__main__':
    t1 = MyTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = TreeValue({'a': -14, 'b': 9, 'x': {'c': 3, 'd': 8}})
    t3 = TreeValue({'a': 6, 'b': 0, 'x': {'c': -5, 'd': 17}})

    print('t1.append(t2).append(t3):',
          t1.append(t2).append(t3), sep=os.linesep)

    print('MyTreeValue.sum(t1, t2, t3):',
          MyTreeValue.sum(t1, t2, t3), sep=os.linesep)

The result should be like below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Append arguments: 1 -14
Append arguments: 2 9
Append arguments: 3 3
Append arguments: 4 8
Append arguments: -13 6
Append arguments: 11 0
Append arguments: 6 -5
Append arguments: 12 17
t1.append(t2).append(t3):
<MyTreeValue 0x7f2df115aeb0>
├── 'a' --> -7
├── 'b' --> 11
└── 'x' --> <MyTreeValue 0x7f2df115ae20>
    ├── 'c' --> 1
    └── 'd' --> 29

Sum arguments: <class '__main__.MyTreeValue'> 1 -14 6
Sum arguments: <class '__main__.MyTreeValue'> 2 9 0
Sum arguments: <class '__main__.MyTreeValue'> 3 3 -5
Sum arguments: <class '__main__.MyTreeValue'> 4 8 17
MyTreeValue.sum(t1, t2, t3):
<MyTreeValue 0x7f2df115aeb0>
├── 'a' --> -7
├── 'b' --> 11
└── 'x' --> <MyTreeValue 0x7f2df115ae20>
    ├── 'c' --> 1
    └── 'd' --> 29

Note

Actually, self-calculation method can be defined like the code below

import os

from treevalue import TreeValue, method_treelize


class MyTreeValue(TreeValue):
    # return type will be automatically detected as `MyTreeValue`
    @method_treelize(self_copy=True)
    def append(self, b):
        return self + b


if __name__ == '__main__':
    t1 = MyTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = TreeValue({'a': -14, 'b': 9, 'x': {'c': 3, 'd': 8}})
    t3 = TreeValue({'a': 6, 'b': 0, 'x': {'c': -5, 'd': 17}})

    print('t1:', t1, sep=os.linesep)
    _t1_id = id(t1)

    print('t1.append(t2).append(t3):',
          t1.append(t2).append(t3), sep=os.linesep)
    assert id(t1) == _t1_id

The result should be

t1:
<MyTreeValue 0x7f5829f417c0>
├── 'a' --> 1
├── 'b' --> 2
└── 'x' --> <MyTreeValue 0x7f582a298250>
    ├── 'c' --> 3
    └── 'd' --> 4

t1.append(t2).append(t3):
<MyTreeValue 0x7f5829f417c0>
├── 'a' --> -7
├── 'b' --> 11
└── 'x' --> <MyTreeValue 0x7f582a298250>
    ├── 'c' --> 1
    └── 'd' --> 29

We can see that the calculation result will replace the self object, without changes of the address. Just enable the self_copy option is okay.

For further information of function method_treelize and classmethod_treelize, just take a look at:

More Flexible Ways to DIY Class

When you implement TreeValue class like the section above (take a look at here DIY TreeValue Class), you have to define your own methods and operators one by one, because in class TreeValue, only the minimum necessary methods and operators are provided. But when you use FastTreeValue class, operators like + and methods like map can be used directly. Let’s see how FastTreeValue is defined in source code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from .general import general_tree_value


class FastTreeValue(general_tree_value()):
    """
    Overview:
        Fast tree value, can do almost anything with this.
    """
    pass

The definition is quite simple, just inherit from the return value of function general_tree_value, and most of the operators and methods can be used.

Like the FastTreeValue class, we can define our TreeValue class like the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import os

from treevalue import general_tree_value


class MyTreeValue(general_tree_value()):
    pass


if __name__ == '__main__':
    t1 = MyTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = MyTreeValue({'a': 11, 'b': 24, 'x': {'c': 30, 'd': 47}})

    # __add__ operator can be directly used
    print('t1 + t2:', t1 + t2, sep=os.linesep)

The output should be

1
2
3
4
5
6
7
t1 + t2:
<MyTreeValue 0x7fc5cde1af10>
├── 'a' --> 12
├── 'b' --> 26
└── 'x' --> <MyTreeValue 0x7fc5cde1ae50>
    ├── 'c' --> 33
    └── 'd' --> 51

In some cases, you can update the behaviours of some operator or methods, like the code below

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import os

from treevalue import general_tree_value, FastTreeValue


class AddZeroTreeValue(general_tree_value(methods=dict(
    __add__=dict(missing=0, mode='outer'),
    __radd__=dict(missing=0, mode='outer'),
    __iadd__=dict(missing=0, mode='outer'),
))):
    pass


class MulOneTreeValue(general_tree_value(methods=dict(
    __mul__=dict(missing=1, mode='outer'),
    __rmul__=dict(missing=1, mode='outer'),
    __imul__=dict(missing=1, mode='outer'),
))):
    pass


if __name__ == '__main__':
    t1 = FastTreeValue({'a': 1, 'b': 2, 'x': {'c': 3}})
    t2 = FastTreeValue({'a': 11, 'x': {'c': 30, 'd': 47}})

    # __add__ with default value 0
    at1 = t1.type(AddZeroTreeValue)
    at2 = t2.type(AddZeroTreeValue)
    print('at1 + at2:', at1 + at2, sep=os.linesep)

    # __mul__ with default value 1
    mt1 = t1.type(MulOneTreeValue)
    mt2 = t2.type(MulOneTreeValue)
    print('mt1 * mt2:', mt1 * mt2, sep=os.linesep)

The output should be

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
at1 + at2:
<AddZeroTreeValue 0x7f57ef6194f0>
├── 'a' --> 12
├── 'b' --> 2
└── 'x' --> <AddZeroTreeValue 0x7f57ef619fa0>
    ├── 'c' --> 33
    └── 'd' --> 47

mt1 * mt2:
<MulOneTreeValue 0x7f57ef6194f0>
├── 'a' --> 11
├── 'b' --> 2
└── 'x' --> <MulOneTreeValue 0x7f57ef619fa0>
    ├── 'c' --> 90
    └── 'd' --> 47

More than this, you can do the following things in your DIY TreeValue class based on function general_tree_value:

  • change treelize arguments (like the example code above)

  • disable some operators and methods

  • change behaviour of some operators and methods

Here is another complex demo code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import os

from treevalue import general_tree_value


class DemoTreeValue(general_tree_value(methods=dict(
    # - operator will be disabled
    __sub__=NotImplemented,
    __rsub__=NotImplemented,
    __isub__=NotImplemented,

    # / operator will raise ArithmeticError
    __truediv__=ArithmeticError('True div is not supported'),
    __rtruediv__=ArithmeticError('True div is not supported'),
    __itruediv__=ArithmeticError('True div is not supported'),

    # +t will be changed to t * 2
    __pos__=lambda x: x * 2,
))):
    pass


if __name__ == '__main__':
    t1 = DemoTreeValue({'a': 1, 'b': 2, 'x': {'c': 3, 'd': 4}})
    t2 = DemoTreeValue({'a': 11, 'b': 24, 'x': {'c': 30, 'd': 47}})

    # __add__ can be used normally
    print('t1 + t2:', t1 + t2, sep=os.linesep)

    # __sub__ will cause TypeError due to the NotImplemented
    try:
        _ = t1 - t2
    except TypeError as err:
        print('t1 - t2:', err, sep=os.linesep)
    else:
        assert False, 'Should not reach here!'

    # __truediv__ will cause ArithmeticError
    try:
        _ = t1 / t2
    except ArithmeticError as err:
        print('t1 / t2:', err, sep=os.linesep)
    else:
        assert False, 'Should not reach here!'

    # __pos__ will be like t1 * 2
    print('+t1:', +t1, sep=os.linesep)

The output should be

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
t1 + t2:
<DemoTreeValue 0x7f339fb95f40>
├── 'a' --> 12
├── 'b' --> 26
└── 'x' --> <DemoTreeValue 0x7f3466e1af10>
    ├── 'c' --> 33
    └── 'd' --> 51

t1 - t2:
unsupported operand type(s) for -: 'DemoTreeValue' and 'DemoTreeValue'
t1 / t2:
True div is not supported
+t1:
<DemoTreeValue 0x7f339fb95d00>
├── 'a' --> 2
├── 'b' --> 4
└── 'x' --> <DemoTreeValue 0x7f3466e1af10>
    ├── 'c' --> 6
    └── 'd' --> 8

For more details about general_tree_value, take a look at general_tree_value and its source code to find out all the implemented operators and methods.

Draw Graph For TreeValue

You can easily draw a graph of the TreeValue objects with function graphics. Like the following example code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import numpy as np

from treevalue import FastTreeValue, graphics, TreeValue


class MyFastTreeValue(FastTreeValue):
    pass


if __name__ == '__main__':
    t = MyFastTreeValue({
        'a': 1,
        'b': np.array([[5, 6], [7, 8]]),
        'x': {
            'c': 3,
            'd': 4,
            'e': np.array([[1, 2], [3, 4]])
        },
    })
    t2 = TreeValue({'ppp': t.x, 'x': {'t': t, 'y': t.x}})

    g = graphics(
        (t, 't'), (t2, 't2'),
        title="This is a demo of 2 trees.",
        cfg={'bgcolor': '#ffffff00'},
    )
    g.render('graphics.dat.gv', format='svg')

The generated code and graph should be like below ( name of the image file should be graphics.dat.gv.svg, name of the graphviz source code file should be graphics.dat.gv ).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// This is a demo of 2 trees.
digraph this_is_a_demo_of_2_trees {
	graph [bgcolor="#ffffff00" label="This is a demo of 2 trees."]
	node_7fdba41f5b20 [label=t color="#3eb24899" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname="monospace bold" penwidth=3 shape=diamond style=filled]
	node_7fdb977e6070 [label=t2 color="#5b3eb299" fillcolor="#9673ff99" fontcolor="#0d0033ff" fontname="monospace bold" penwidth=3 shape=diamond style=filled]
	value__node_7fdba41f5b20__a [label=1 color="#3eb24800" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname=monospace penwidth=1.5 shape=box style=filled]
	node_7fdba41f5b20 -> value__node_7fdba41f5b20__a [label=a arrowhead=dot arrowsize=0.5 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	value__node_7fdba41f5b20__b [label="[[5&ensp;6]\l&ensp;[7&ensp;8]]\l" color="#3eb24800" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname=monospace penwidth=1.5 shape=box style=filled]
	node_7fdba41f5b20 -> value__node_7fdba41f5b20__b [label=b arrowhead=dot arrowsize=0.5 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	node_7fdb97596e80 [label="t.x" color="#3eb24899" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname="monospace bold" penwidth=1.5 shape=ellipse style=filled]
	node_7fdba41f5b20 -> node_7fdb97596e80 [label=x arrowhead=vee arrowsize=1.0 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	node_7fdb977e6070 -> node_7fdb97596e80 [label=ppp arrowhead=vee arrowsize=1.0 color="#3d1f99cc" fontcolor="#1d0073ff" fontname="Times-Roman bold"]
	node_7fdba38d9d90 [label="t2.x" color="#5b3eb299" fillcolor="#9673ff99" fontcolor="#0d0033ff" fontname="monospace bold" penwidth=1.5 shape=ellipse style=filled]
	node_7fdb977e6070 -> node_7fdba38d9d90 [label=x arrowhead=vee arrowsize=1.0 color="#3d1f99cc" fontcolor="#1d0073ff" fontname="Times-Roman bold"]
	value__node_7fdb97596e80__c [label=3 color="#3eb24800" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname=monospace penwidth=1.5 shape=box style=filled]
	node_7fdb97596e80 -> value__node_7fdb97596e80__c [label=c arrowhead=dot arrowsize=0.5 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	value__node_7fdb97596e80__d [label=4 color="#3eb24800" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname=monospace penwidth=1.5 shape=box style=filled]
	node_7fdb97596e80 -> value__node_7fdb97596e80__d [label=d arrowhead=dot arrowsize=0.5 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	value__node_7fdb97596e80__e [label="[[1&ensp;2]\l&ensp;[3&ensp;4]]\l" color="#3eb24800" fillcolor="#73ff7e99" fontcolor="#003304ff" fontname=monospace penwidth=1.5 shape=box style=filled]
	node_7fdb97596e80 -> value__node_7fdb97596e80__e [label=e arrowhead=dot arrowsize=0.5 color="#1f9929cc" fontcolor="#00730aff" fontname="Times-Roman bold"]
	node_7fdba38d9d90 -> node_7fdba41f5b20 [label=t arrowhead=vee arrowsize=1.0 color="#3d1f99cc" fontcolor="#1d0073ff" fontname="Times-Roman bold"]
	node_7fdba38d9d90 -> node_7fdb97596e80 [label=y arrowhead=vee arrowsize=1.0 color="#3d1f99cc" fontcolor="#1d0073ff" fontname="Times-Roman bold"]
}
../../_images/graphics.dat.gv1.svg

Note

The return value’s type of function graphics is class graphviz.dot.Digraph, from the opensource library graphviz, for further information of this project and graphviz.dot.Digraph’s usage, take a look at:

For further information of function graphics, just take a look at