Dtyjlui

Let us assume that I need to storage in a file, or files, data with the following form

alist = 

  Table[Table[RandomReal[{0, 1}, 2], {i, 1, 200}], {j, 1, 10000}];

blist = 

  Table[Table[{RandomReal[{0, 1}, 6], RandomReal, 

     RandomReal[{0, 1}, 6], RandomReal[{0, 1}, 6]}, {i, 1, 200}], {j, 

    1, 10000}];

clist = RandomReal[{0, 1}, 10000];

In a few seconds, Mathematica is able to create these lists.

However, when I try to export this data to a .txt file, using

Export["C:\pathname\abclists.txt", {alist, blist, clist}];

My computer starts to freeze, and I have to abort the evalutation. When I check the result, Mathematica has created a 600MB .txt file.

Is there a more efficient way to storage data in a file or files? I have no preference for the file type, as long as I can easily rebuild the original data, preserving the list levels.

Using ByteCount will tell you to expect more than 1 GB:

ByteCount @ { alist, blist, clist }



(* 1 136 880 448 *)

Compress

I would thus try Compress:

cexpr = Compress @ { alist, blist, clist };

(* Export["pathname\data.m", cexpr ]; *)

ByteCount @ cexpr



(* 437 359 280 *)

I am getting a 433 MB file (which roughly matches ByteCount). You can Uncompress the expression after loading.

BinarySerialize

Another possibility as of Version 11.1 or later is BinarySerialize:

bexpr = BinarySerialize @ { alist, blist, clist };

ByteCount @ bexpr



(* 378 170 128 *)

So we are down to about 378 MB (the file is 312 MB on my computer). You can use BinaryDeserialize to get the original expression again (see below for explicit instructions for writing/reading binary data).

If we give the option PerformanceGoal -> "Size"

bexpr = BinarySerialize[ {alist,blist,clist}, PerformanceGoal -> "Size" ];

ByteCount @ bexpr



(* 327 494 245 *)

we are down to about 327 MB.

Writing and Reading Binary Data

The documentation tells you how to write/read binary data:

stream = OpenWrite[ "pathname\data.mx", BinaryFormat -> True ];

BinaryWrite[ stream, bexpr ];

Close @ stream;

Reading the data:

data = BinaryDeserialize @ ByteArray @ BinaryReadList[ "pathname\data.mx", "Byte" ]; 

Old (OS and Mma version dependent!) "MX" format is also pretty fast and good for storing.

abcList = {alist, blist, clist};

ByteCount[abcList]

Out[2]= 1136880448

Export[

 FileNameJoin[{NotebookDirectory[EvaluationNotebook], 

   "test.mx"}], abcList, "MX"];



FileByteCount[

 FileNameJoin[{NotebookDirectory[EvaluationNotebook], "test.mx"}]]

Out[4]= 434116760

abcListRead = 

  Import[FileNameJoin[{NotebookDirectory[EvaluationNotebook], 

     "test.mx"}]];



ByteCount[abcListRead]

Out[6]= 1136880448