I have started to run the Python tests on the flatbuffers
package in Fedora Linux, and I am seeing some failures on the big-endian s390x platform. I’m currently working with version 22.12.06, but I don’t believe anything relevant to this issue has changed since then.
======================================================================
FAIL: test_consistenty_with_repeated_pack_and_unpack (__main__.TestObjectBasedAPI.test_consistenty_with_repeated_pack_and_unpack)
Checks the serialization and deserialization between a buffer and
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 153, in test_consistenty_with_repeated_pack_and_unpack
CheckReadBuffer(b1.Bytes, b1.Head(), sizePrefix)
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 705, in CheckReadBuffer
asserter(VectorOfLongs.dtype == np.dtype('int64'))
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 614, in asserter
raise AssertionError('CheckReadBuffer case failed')
AssertionError: CheckReadBuffer case failed
======================================================================
FAIL: test_wire_format (__main__.TestWireFormat.test_wire_format)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 110, in test_wire_format
CheckReadBuffer(
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 705, in CheckReadBuffer
asserter(VectorOfLongs.dtype == np.dtype('int64'))
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_test.py", line 614, in asserter
raise AssertionError('CheckReadBuffer case failed')
AssertionError: CheckReadBuffer case failed
----------------------------------------------------------------------
Previous issues related to big-endian support include https://github.com/google/flatbuffers/issues/7366, https://github.com/google/flatbuffers/issues/7671, and https://github.com/google/flatbuffers/issues/7685.
It seems like the solution here is to compare against explicitly little-endian dtypes. For example, in test_consistenty_with_repeated_pack_and_unpack
(which has a typo, consistenty
/consistency
in its name), the value of VectorOfLongs.dtype
is np.dtype('<i8')
on big-endian platforms. For this test, it should be appropriate to just change the expected dtype to np.dtype('<i8')
, since np.dtype('int64') == np.dtype('<i8')
on little-endian platforms.
I will follow up with a PR to do that. Once all of the “reference” dtypes in py_test.py
are converted to explicitly little-endian ones, I’m left with the following error.
----------------------------------------------------------------------
Ran 121 tests in 0.032s
OK
Testing with multi-file generated code
numpy available
vtable deduplication rate (n=1, vtables=1): 216615.37 sec
vtable deduplication rate (n=10, vtables=16): 87679.23 sec
vtable deduplication rate (n=100, vtables=130): 45469.22 sec
vtable deduplication rate (n=1000, vtables=1039): 30760.56 sec
traversed 100 512-byte flatbuffers in 0.02sec: 6313.29/sec, 3.08MB/sec
built 100 512-byte flatbuffers in 0.02sec: 5822.80/sec, 2.84MB/sec
........................................E...................
======================================================================
ERROR: test_gold_from_file (__main__.EncoderTest.test_gold_from_file)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/flatbuffers-22.12.06/tests/py_flexbuffers_test.py", line 1423, in test_gold_from_file
self.assertEqual(flexbuffers.Loads(data), GOLD_FLEXBUFFER_OBJ)
^^^^^^^^^^^^^^^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 1527, in Loads
return GetRoot(buf).Value
^^^^^^^^^^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 828, in Value
return self.AsMap.Value
^^^^^^^^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 537, in Value
return {k.Value: v.Value for k, v in zip(self.Keys, self.Values)}
^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 529, in Keys
return TypedVector(buf, byte_width, Type.KEY)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 449, in __init__
super().__init__(buf, byte_width, size)
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 340, in __init__
self._size = _Unpack(U, self.SizeBytes)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/builddir/build/BUILDROOT/flatbuffers-22.12.06-6.fc38.s390x/usr/lib/python3.11/site-packages/flatbuffers/flexbuffers.py", line 98, in _Unpack
return struct.unpack(fmt[len(buf)], buf)[0]
~~~^^^^^^^^^^
KeyError: 0
----------------------------------------------------------------------
This appears to be because the format strings for the struct
module in flatbuffers/flexbuffers.py
and its tests (tests/py_flexbuffers_test.py
) are all implicitly host-endian, even though as I understand it the flatbuffers wire format is always little-endian, and even though the other modules in flexbuffers
appear to already use explicitly little-endian format strings. I will fix that in my PR, too.