OverflowError: cannot serialize a bytes object larger than x GiB is an error which occurs when you use _pickle.dump in your code.
In the blog post I attempt to explain why this error takes place and how you can solve it, I will also add other solutions that could solve the error if possible.
Exploring the Error Python Pickle OverflowError: cannot serialize a bytes object larger than x GiB
This is an error which occurs when you use _pickle.dump in your code.
The error you should have should be similar to the error bellow. Please make sure you do not confuse between errors.
#
OverflowError: cannot serialize a bytes object larger than x GiB
#
Bellow I make my best attempt at solving the error and present the best possible solution.
Solution 1 : break your data before using _pickle.dump
The error happens when you use _pickle.dump in your script, since pickle has a hard limit of 4GB.
The first solution is simple and straightforward, break up the object into multiple objects, each object has a size bellow 4GB and you can save each object to a pickle.
This way you can use your data, while respecting the pickle size limitation, but if this did not work for you or if you do not wish to split your data, check out the solution bellow.
Solution 2 : specify the latest version of pickle when using pickle.dump()
Another solution, which is way easier is to specify the version of pickle when using pickle.dump().
As of now, the latest version of pickle is pickle 5.
#
pickle.dump(d, open("file", 'w'), protocol=5)
#
Also make sure you have one of the latest python versions, combined with pickle 5, the error will never occur again
The solutions above should be enough to solve the problem, if you like our effort make sure to consider donating to our Kofi account, there is a red button that you can use if you are feeling generous.
Summing-up :
This is the end of our article, the error : overflowError cannot serialize a bytes object larger than x GiB could be confusing but with a little bit of investigating the error could be solved easily, thank you for reading.
If you want to learn more about Python, please check out the Python Documentation : https://docs.python.org/3/