Thank you for advises!
I've tried it with bigger memory computer that has 512 GB.
Then the error message has been changed to
Error : package or namespace load failure for 'lme4'
.
.
.
**Error:
Quitting due to the model test failure...
however when I test the same code with small number of subjects (n=10) it worked fine without any errors.
I've also tried to split my data set into 5 proportions,my question is in that case should I average all 5 results to get the whole data set's results?