Dtyjlui

I have a pretty good understanding of LaTex, but I've spent a while now digging up information on catcodes, but because they're lesser known, I haven't seen any applicable examples that will help with my problem.

For those of you familiar with any experimental work with data tables, you'll probably be aware that the standard way to store data as numbers in is a table, either tab or space delimited, with enters for a new line. This is how many data processing programs read and write data.

As a result, if I want to enter data into LaTex, I have to reformat my data table to include ampersands between columns and a double-backslash to end lines. Is there a straight forward way to create a new environment, which redefines spaces and tabs and double-backslashes, inserts a tabular environment, and then undoes the redefinitions so that they do not have any effect on the rest of the code, something like

newenvironment{datatable}[2]  

catcode{10}=catcode{4} 

% catcode 10 is spaces and tabs, catcode 4 is an ampersand 



catcode{5}=\ % catcode 5 is the return character

% I haven't a clue how to define it as a double-backslash   



begin{tabular}{#1}  

   #2  

end{tabular}  

% I have no idea how I would undo the definitions from above, because if I said   

catcode{10}=catcode{10} 

% I'm not sure it would make any sense

% catcode 10 already points to catcode 4

% and I haven't seen anyone use an undo command for catcodes yet 



catcode{10}=catcode{4} 

% catcode 10 is spaces and tabs, catcode 4 is an ampersand 



catcode{5}=\ 

% catcode 5 is the return character, I haven't a clue how to define it

% as a double-backslash   



begin{tabular}{#1}  

   #2  

end{tabular}  

% I have no idea how I would undo the definitions from above

% because if I said   

catcode{10}=catcode{10} 

% I'm not sure it would make any sense

% catcode 10 already points to catcode 4

% and I haven't seen anyone use an undo command for catcodes yet 

Hopefully it's clear what I want to do, but effectively I just want to get a better understanding of redefining catcodes, and then undoing redefinitions. Hopefully this question also helps some other Tex users as well. Thanks for your help!

Edit:

The solution does not require the use of catcodes. In my case, I require the ability to implement the solution on overleaf.com (which is just an online latex editor and compiler), but other than that, I have no restrictions.

TeX maintains a catcode array; this array has length 256 (indexed from 0) in 8-bit engines and 0x110000 in Unicode engines (XeTeX and LuaTeX). Each entry in the array should be a 4-bit number.

You assign an entry by a declaration of the form

catcode<number> = <4-bit number>

(the = and the spaces around it are optional); the first <number> must be in the allowed range, that is, 0–255 for 8-bit engines (Knuth TeX, pdftex) or 0–1114111 for Unicode engines.

If used in a non assignment context, catcode<number> returns the corresponding entry in the array. For instance, thecatcode`a returns 11 (under standard setup).

Numbers can be input in one of the standard TeX ways: an integer in its decimal representation, an octal number, a hexadecimal number, or by character code:

catcode 97 = 11

catcode '141 = 11

catcode "61 = 11

catcode `a = 11

catcode `a = 11

are all equivalent and assign character a the catcode 11; more technically, they assign the number 11 to the entry in the catcode array indexed by 97. The <number> can also be anything that returns a number in the context (a counter's value, a chardef token, a mathchardef token, numexpr, an internal array entry). Also the right-hand side in the assignment can be expressed in different formats. For instance

catcode`a=catcode`b

will assign a the same category code as b (whatever is the current category code of b. A common idiom is catcode<number>=active, where active is a chardef token whose value is 13.

Thus your catcode{10}=catcode{4} is syntactically wrong. Note that there's no way of telling “change all characters of category code 10 into category code 4” without looping through the array; in an 8-bit engine

count255=0

loopifnumcount255<256

  ifnumcatcodecount255=10 catcodecount255=4 fi

  advancecount255 by 1

repeat

would change the characters of category code 10 into characters of category code 4.

When TeX is in the tokenization phase, interpreting input and forming tokens from it, it attaches to character tokens the corresponding category code.

Let's attack your problem. You want to input a file, but assigning different category codes to some characters, namely tab and spaces, but also interpreting the end-of-line as \. Let's use a syntax such as

 maketabularfromfile{<table specs>}{<filename>}

Here's a possible code:

begin{filecontents*}{jobname.dat}

a   b c

1   2 3

4   5 6

end{filecontents*}



documentclass{article}



makeatletter

newcommand{maketabularfromfile}[2]{%

  % confine the changes

  begingroup

  catcode` =4   % space is column separator

  catcode`^^I=4 % tab is column separator

  catcode`^^M=active % end-of-line is active

  begingrouplccode`~=`^^M lowercase{endgroupdef~}{\}%

  begin{tabular}{#1}

  @@input #2 % use the primitive input

  end{tabular}

  endgroup

}

makeatother



begin{document}



maketabularfromfile{ccc}{jobname.dat}



end{document}

I used <TAB>s as separators for the second column, just by way of example (the site will most likely convert them to spaces).

This is very crude, of course. But since there are packages such as datatool or cvssimple that do a much better job, I don't think it's worthwhile to reinvent the wheel.

I think you're complicating your life, what you actually need is "read a text file and produce a table".

There are many packages for doing that.
In the following MWE, you can find some examples with csvsimple and pgfplotstable, but there are others: datatool is more powerful, even if perhaps not for beginners. I recommend to you to read their documentation and leave catcodes to egreg & Co.

I've made my example on Overleaf, with the usual pdfLaTeX compiler.

documentclass{article}



usepackage{booktabs}

usepackage{siunitx}

usepackage{csvsimple}

usepackage{pgfplotstable}

pgfplotsset{compat=1.14}

% you can set some option for all your table if you write this 

pgfplotstableset{string type, 

    every head row/.style={before row=toprule,after row=midrule},

    every last row/.style={after row=bottomrule}}



usepackage{filecontents}% only needed to create the csv in this example

% the following filecontents* environments are needed only to create the text files, you don't need the if you already have yourfile....dat

begin{filecontents*}{yourfilecommasep.dat}

Name,Surname,Length,Gender

Paulinho,van Duck,.56,M

Paulette,de la Quack,.52,F

end{filecontents*}

begin{filecontents*}{yourfilespacesep.dat}

Name Surname Length Gender

Paulinho {van Duck} .56 M

Paulette {de la Quack} .52 F

end{filecontents*}



begin{document}

verb|pgfplotstable| accepts space, tab, comma, semicolon, colon, braces, and ampersand as 

separators.



begin{center}

pgfplotstabletypeset[

    col sep=space, 

    ]

    {yourfilespacesep.dat}

end{center}



With verb|pgfplotstabletypeset|, you don't even need a file, you can type your data directly.



begin{center}

pgfplotstabletypeset[

    col sep=space, 

    ]

    {

    Name Surname Length Gender

    Paulinho {van Duck} .56 M

    Paulette {de la Quack} .52 F

    }

end{center}



verb|csvsimple| accepts comma (the default), semicolon, pipe, and tab as separators.



Here I also merged the two colums "Name" and "Surname", and inverted the two colums "Gender" and "Length":

begin{center}

begin{tabular}{

    lcS[table-format=1.2,round-mode=places]}

    toprule

    Ducks and drakes & Gender & {Length} \

    & & {(si{metre})} \

    midrule

    csvreader[head to column names,

        late after line=\]{yourfilecommasep.dat}{}%

        {Name Surname & Gender & Length}

    bottomrule

end{tabular}

end{center}



end{document}

Changing catcodes is always a bit tricky especially with space, and especially if here input may contained braced items with spaces too.

So I am changing only catcode of end-of-line.

I am illustrating here inline input, if data is in external file, small variant is needed. But anyhow, there are packages for this, of course method here is only a few lines of macros.

documentclass{article}



usepackage{array}

usepackage{booktabs}



makeatletter

defmytabular@end{end{mytabular}}

defmytabular@parse#1 %

{%

  ifrelaxdetokenize{#1}relax

    g@addto@macromytabular@line{\}%

    xdefmytabular@tabular{%

       unexpandedexpandafter{mytabular@tabular}%

       unexpandedexpandafterexpandafterexpandafter{mytabular@line}%

       }%

    expandaftermytabular@again

  else

    g@addto@macromytabular@line{&#1}%

    expandaftermytabular@parse

  fi

}%

% some complications to insert toprule but then midrule only once

% after header line

defmytabular@newline{%

    letmytabular@newlinemytabular@@newline

    gdefmytabular@line{mytabular@atstarttoprule}%

}%

defmytabular@@newline{%

    letmytabular@newlinemytabular@@@newline

    gdefmytabular@line{mytabular@atstartmidrule}%

}%

defmytabular@@@newline{%

    gdefmytabular@line{mytabular@atstart{}}%

}%

defmytabular@atstart#1&{#1}%

defmytabular@finish{%

    g@addto@macromytabular@tabular{bottomruleend{tabular}}%

    end{mytabular}mytabular@tabular

}%

begingroupcatcode`^^Mactive

  gdefmytabular@again{expandafter^^M}%

endgroup

defzzzzzzzzzzzzzzzzzzzzz#1{%

newenvironment{mytabular}[1]%

  {%

   gdefmytabular@tabular{begin{tabular}{##1}}%

   obeylines

   begingrouplccode`~`^^M 

   lowercase{endgroupdef~####1~}{%

     defmytabular@tmp{####1}%

     ifxmytabular@tmpmytabular@end

       expandaftermytabular@finish

     else

       mytabular@newline%

       mytabular@parse####1 #1%

     fi}%

  }%

  {}%

}zzzzzzzzzzzzzzzzzzzzz{ }%

makeatother



begin{document}



begin{mytabular}{cccc}

    Name Surname Length Gender

    Paulinho {van Duck} .56 M

    Paulette {de la Quack} .52 F

end{mytabular}



end{document}