Unable to allocate array with shape and data type in Python is an error that occurs on Ubuntu and even on windows computers due to the system’s overcommit handling mode.
In this article I’m going to do my best in order to describe how I encountered te error and solved it on my own, I’m also going to describe other solutions that worked for other developers.
Explaining The Error : Unable to allocate array with shape and data type in Python
I use the code bellow to allocate memory for a numpy array with shape …
The result is an error on Ubunto but not on MacOS
After a lot of research I solved the problem and encountered other possible solutions that worked for other people, bellow are all the possible solutions in detail.
Solution 1 : Correctly use np.zeros
The cause of the error is my system’s overcommit handling mode.
In order to check my current overcommit mode I ran the code bellow :
#
# run this code to change the overcommit mode
cat /proc/sys/vm/overcommit_memory
#
After I used cd to root in the command line then I ran the code bellow :
#
# run this code after cd to root in the command line
echo 1 > /proc/sys/vm/overcommit_memory
#
The system allowd me to make the allocation, it does not matter how large it is, it should be within the limits of my system of course.
With overcommit mode 1, I managed to make it work
Now, I can write to any location within the array
If you find our blog useful please consider helping us by donating to our Kofi account above, you do not have to do this, our blog is free.
Solution 2 : increase the pagefile size in windows
I had this same problem on Window’s and came across this solution. So if someone comes across this problem in Windows the solution for me was to increase the pagefile size, as it was a Memory over commitment problem for me too.
For windows developers the problem is an over commitment problem and the solution is to increase the pagefile size, in order to that in Windows 10 follow the steps bellow :
- Start by Pressing the Windows key on your desktop.
- In the search bar Type SystemPropertiesAdvanced.
- Click Run as administrator.
- Click Settings under Performance.
- Then make sure to select the advanced tab.
- Then select Change.
- Please uncheck Automatically managing paging file size for all drives.
- Select Custom size and fill in the appropriate size.
- Click Set then Click OK and then exit from all the Dialogs.
- Now Reboot your system and after that, everything should work as intended.
Solution 3 : restart your system
According to other developers on Linux, restarting the system was enough, since for them the error occurs when the kernel has reached its limit. I can’t vouch for this solution since I did not try it, but you can try it and if it does work for you then congrats.
Solution 4 : change the data type to numpy.uint8
Another trick is to use a data type that uses less memory, so you can change the data type to numpy.uint8
#
# use this code to change the data type to numpy.uint8
data['label'] = data['label'].astype(np.uint8)
#
This is a nice solution if you are ready to change your data type
Solution 5 : change the type float64 to uint8
Similar to the solution above but a little bit different, some users have added a dtype attribute in order to change the dtype of the array to a smaller type.
In the example bellow we changed the type float64 to uint8 and it worked for a lot of users.
#
# use this code to change the type float64 to uint8
mask = np.zeros(edges.shape)
#
#
# use this code to change the type float64 to uint8
mask = np.zeros(edges.shape,dtype='uint8')
#
Summing-up
This is the end of our guide, I hope you found at least one solution useful for the error Unable to allocate array with shape and data type , we try to provide solutions for both Linux and windows users, If you did not solve the issue, I hope you find it online, do not give up, errors are normal in Python, Cheers.
If you want to learn more about Python, please check out the Python Documentation : https://docs.python.org/3/