ps is the 'process status' command. It returns a list of running processes with information about them, for purposes here only the name is relevant.
Sidenote: the busybox version of ps (which is what runs within TWRP) is more full featured than the toybox version. One example - toybox version won't accept the '-A' parameter. I think the only thing it does accept is a pid.
For more information on ps check Google. Try 'ps command Linux' or 'ps man page'. For now just know it returns a list of processes, by default ones which belong to the user running the command.
So the output from ps is passed to grep which checks the output for 'zygote' (which is the "master" process when you boot), this output is then passed to fgrep again to remove any lines with 'grep' in them.
Next the return code from the grep is checked. If return code is 0 (&& test) set BOOTMODE to true. If not 0 (|| test) set to false.
The second line starts by testing the value of BOOTMODE. If it's false it runs the 'ps -A' command, tests further for zygote, sets to true if found.
As noted above this second line won't really do anything because 'ps -A' will throw an error in a booted system since toybox will reject it. But what I described is the intent.
Hope this helps. If you have a specific question about part of this then ask.