SMPOV-Files / RenderAgent - Intern * How things work internally

 
Whats this page about / Inhalt

SMPOV and the RenderAgents communicate with an simple cleartext-protocoll. If you know how it works, you can make your own RenderAgents.

When developing the first version of SMPOV, I thought of it like a bakery. SMPOV is the boss and the RenderAgents are the employees. In our case the boss (SMPOV) gives "RenderJobs" to the employees. This is done by writting out the "JOB-Files" and additional informations in the COM-Folder.
Currently SMPOV does not tell a RenderAgent which Job to take first or what to do. There are all the Jobs, each RenderAgent can take any of these. SMPOV just waits till the Job is done. Internally SMPOV waits till it finds a ".rdy"-File for each RenderJob. Assume you have 625 tiles, then SMPOV waits for these 625 ".rdy"-files before starting the RenderAgent.

On this page I'll desribe these things more in detail.

Everything that is been exchanged between any part of SMPOV is inside the so called "COM" (like Communication) folder.

Actually inside the COM-Folder there are three subfolders:

  • BMP (for all resulting BMP's
  • JOB (contains the JOB-Files)
  • INI (contains everything needed by POV-Ray for rendering)

Additionally we can find a Folder for each RenderAgent that has ever been accessing this COM-Folder. These Folders are for private use of the renderAgents and currently not used by SMPOV-Main.

 

Step 1 / Schritt 1

When we drag one or more ".pov"-file(s) into SMPOV, it is added to the SMPOV-Render-Que. SMPOV splits the File in one or more "RenderJobs".
Inside this process the file gets a longer filename, to prevent any confusion with files of the same name. This longer filename contains the actual date as well as some random components.

  • Example: The file: "benchmark.pov" becomes internally "benchmark_01082004_104626_87176...".
  • All files from one ".pov"-Source-file have the same starting-filename and diffrent endings.
    We will refer to this internal name further as FN.
 

Step 2 / Schritt 2

This is the JOB-Folder before any RenderAgent is been activated. We see the following files:

  • FN.info (not needed by RenderAgent, contains the SMPOV-Settings for this file)
  • FN.pnd.txt (not needed by RenderAgent, this is for the PicPender.exe)
  • FN_(tile-number)T.bat (this is the later "Job-File", RenderAgent is looking for!) there is one of these files for each tile.
  • FN_PP.bat (this file is not needed by RenderAgent, it can be used to manually start the PicPender.exe).

Only one of these files, is used for RenderAgent. We will take a closer look inside this file later. On the left side, you can see that we have two JOB-Files for this picture. Therefore we can see that is has to be rendered in two tiles.
RenderAgent itself will reserve the file by renaming it to a unique filename and as well as changing the file-ending to ".job". Internally there are few more steps to enshure that never two RenderAgents can take the same RenderJob. The key to this is that the file has been renamed to a unique Filename and got another ending.

 
Step 3 / Schritt 3

At the same time, all Information that are necessary for POV-Ray have been copied into the "INI-Folder". These are:

  • FN.pov as the POV-Ray source file to render by all instances
  • FN_(tile-number).ini as the parameter-file to tell POV-Ray which part of the picture to render and with which parameters. This file is later been changed by RenderAgent, so we take a look at this later.

Thats all we need to start a distributed Renderjob.

 
Step 4 / Schritt 4

Before we start a RenderAgent.exe and watch what will happen to these files. There are some important aspects to think about:

  • POV-Ray is installed possibly on another location on the client where RenderAgent is running
  • the COM folder has maybe another path on the client then it has on the same machine

Therefore RenderAgent needs to do 2 steps in the following:

  1. It needs to reserve a Job-File for its own exclusive use (that is one of the FN_(tile-number)T.bat files).
  2. RenderAgent will need to change all pathes in the JOB and in the INI-File to the pathes which are valid for this local client.
 
The Job File from SMPOV before it has been changed by the RenderAgent
cd "C:\Programme\POV-Ray for Windows v3.5\"
C:
REM %POVEXE /EXIT /RENDER "%MEPbenchmark_01082004_104626_87176_1_0x0-80x120a.ini"
REM REN "%MEPbenchmark_01082004_104626_87176_1_0x0-80x120a.ini" *.rdy
REM Exit
bin\pvengine.exe /RENDER "f:\00_rpgm\01_bs\01_projekte\pb_smpov 4\com\ini\benchmark_01082004_104626_87176_1_0x0-80x120a.ini" /EXIT
REN "f:\00_rpgm\01_bs\01_projekte\pb_smpov 4\com\ini\benchmark_01082004_104626_87176_1_0x0-80x120a.ini" *.rdy
Exit

The JOB-File is a batch file. On the same machine it could have been started manually to start the RenderJob (which was the case in first versions). On another machine this would fail, cause the pathes would not be valid. Therefore it contains some meta-information which are prefixed by "REM" so they won't be used for SMP-Rendering. For distributed rendering, these meta-informations will later be completed by the RenderAgent with the actual pathes of the client. These meta-informations are:

  • %POVEXE. - which is the path and filename of POV-Ray/Mega-POV on the client.
  • %MEP which is the path to the COM\INI-Folder
 
The INI-File before it has been touched by RenderAgent
; Diese Datei wurde automatisch erzeugt von SMPOV. 01-08-2004 / 10:46:28
; Diese INI-Datei ist Nr. 1 von insgesamt: 2
; Mode: POV-Ray 3.5 Mode
;
; Höhe des gesamten Bildes:
Height=120
;
; Breite des gesamten Bildes:
Width=160
;
; Start-Spalte des Bild-Ausschnittes:
Start_Column=0
;
; End-Spalte des Bild-Ausschnittes:
End_Column=80
;
; Input-Filename
; Path%F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\INI\

Input_File_Name=F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\INI\benchmark_01082004_104626_87176.POV
;
; Output-Filename
Output_File_Type=S
Output_File_Name=F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\BMP\benchmark_01082004_104626_87176_1_0x0-80x120A.BMP
;
; Dieser Text wird aus der Common.ini übernommen
; und ist für alle Teilbilder gültig.

-D

Statistic_Console=off

In the INI-File there are in the same way pathes for Input and Output-Filenname as well as a necessary meta-information. The final INI will be produced temporarily by RenderAgent out of these informations.

  • Path% - this meta information shows which part of the Input/Output-Filename has to be replaced by the actual, valid path for the client.

Please note that this INI-File contains all the settings given in the SMPOV-GUI and (appended) the common.ini which is an optional feature in the SMPOV-GUI..

 
Step 4/ Schritt 4
Now we start a RenderAgent.exe and watch what will happen to these files.
 
What happened to the JOB-File?
C:
start /wait bin\pvengine.exe /EXIT /RENDER "J:\INI\benchmark_01082004_104626_87176_1_0x0-80x120a.ini"

REN "J:\INI\benchmark_01082004_104626_87176_1_0x0-80x120a.ini" *.rdy
DEL "J:\JOB\benchmark_01082004_104626_87176_1T.job"
EXIT

Taking a look into the RenderAgents private Folder, we find a file called "_1_private.bat" (Name is unimportant) which looks like we have expected. All pathes are changed so the rendering will work on that particular client.
 
Step 5/ Schritt 5
Taking a peek into the INI-Folder, we notice that the INI-File for this RenderJob has been renamed and therefore marked as "in use".
 
Step 6/ Schritt 6

Thats all the RenderAgents have to do. Let me put all the steps together, one after the other:

  1. Initialise:
  2. find the COM-Folder make a private Folder inside
  3. find POV-Ray and Mega-POV, remember their path

Then after the RenderAgents are running what they do is this:

  1. Look for a JOB-File in COM\JOB.
  2. reserve it by moving it to the private Folder.
  3. change the Job-File ...
  4. as well as the INI-File that belongs to that JOB-File so the pathes fit to the client
  5. Look if the FN.pov is a MEGA-POV-File or POV 3.5 (search for "#version ...")
  6. Run POV or MEGA-POV with the corrected INI-File as paramter
  7. Wait till POV-Ray ends.
  8. Rename the "INI-File"-Ending from ".ini" to ".rdy". This is the Signal for SMPOV that the JOB is ready.
  9. RenderAgent will now open the CPU Slot and continue with Step 1.
 
Additional comments

The current Windows-Implementation of RenderAgent (as well as SMPOV) is multithreaded, which makes the source-code a biit more complex then necessary.
The above described operations overlap each other in real time. Another implementation could be single-threaded without big disadvantages.

If someone has 4 CPU's he could start 4 of these single-threaded RenderAgents, while this is not necessary for the current windows implementation.

 
Additional Informations about the SMPOV-Rendering Pipeline
The following Informatios are not needed to make a custom RenderAgent. They are provided for completeness.
 
The benchmark_01082004_104626_87176.pnd.txt
SF:F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\BMP\benchmark_01082004_104626_87176_1_0x0-80x120A.BMP
SF:F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\BMP\benchmark_01082004_104626_87176_2_80x0-80x120A.BMP

DF:F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\BMP\benchmark_01082004_104626_87176_2_80x0-80x120A.BMP
MV:F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\BMP\benchmark_01082004_104626_87176_2_80x0-80x120A.BMP>F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\Ani-Test\benchmark.bmp
ER:F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\JOB\benchmark_01082004_104626_87176.pnd.txt

The FN.pnd.txt contains the instructions for the PicPender.exe how to combine the tiles to one file. Even if you do not use tiling (so you have one single picture result) the PicPender.exe is called to copy the resulting file to its final destination. The instructions work as follows:

  • SF: - Sourcefile - each tile has one Line SF: which marks the tile as a sourcefile. Please take a look at the BMP-Filename which therefore contains additional informations.
  • DF: - Destinationfile - this can only be given once and is typically a copy of the last Sourcefile.
  • MV: - FileMove - this command makes PicPender.exe to move a file from A -> B
  • ER: - Erase File - given this command, PicPender.exe will delete the the specidfied File, no matter if its write-protected or not.
 
The benchmark_01082004_104626_87176_PP.bat
"F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\PicPend-1.exe" "F:\00_RPGM\01_BS\01_Projekte\PB_SMPOV 4\COM\JOB\benchmark_01082004_104626_87176.pnd.txt"

The file FN_PP.bat is the batchfile that will be called from SMPOV after all Renderjobs have been completed. It could be called manually as well, in case SMPOV would have been ended before time. Currently it contains just one line, that is the call to PicPender.exe with the FN.pnd.txt as parameter.
 
The current "times.log"
irid.pov rendered in 3200x2400 in 16 Tiles (16,1)-(H) from 01-06-2004 / 23:51:44 to 01-06-2004/23:54:00. Tiling, copying ( 255 Sek.) ended: 01-06-2004 / 23:58:15 (393 Sek.).
irid.pov rendered in 3200x2400 in 25 Tiles (1,25)-(V) from 01-06-2004 / 23:58:52 to 01-07-2004/00:00:57. Tiling, copying ( 3 Sek.) ended: 01-07-2004 / 00:01:00 (131 Sek.).
irid.pov rendered in 9600x9600 in 10 Tiles (1,10)-(V) from 01-07-2004 / 00:05:37 to 01-07-2004/00:12:03. Tiling, copying ( 68 Sek.) ended: 01-07-2004 / 00:13:11 (456 Sek.).
irid.pov rendered in 12288x12288 in 4 Tiles (1,4)-(V) from 01-07-2004 / 00:23:19 to 01-07-2004/00:28:15. Tiling, copying ( 130 Sek.) ended: 01-07-2004 / 00:30:25 (427 Sek.).
irid.pov rendered in 3200x2400 in 2 Tiles (1,2)-(V) from 01-07-2004 / 00:41:57 to 01-07-2004/00:42:21. Tiling, copying ( 3 Sek.) ended: 01-07-2004 / 00:42:23 (26 Sek.).
irid.pov rendered in 3200x2400 in 2 Tiles (2,1)-(H) from 01-07-2004 / 00:42:34 to 01-07-2004/00:43:01. Tiling, copying ( 19 Sek.) ended: 01-07-2004 / 00:43:20 (46 Sek.).
benchmark1.pov rendered in 3200x2400 in 4 Tiles (1,4)-(V) from 01-07-2004 / 01:07:00 to 01-07-2004/09:35:08. Tiling, copying ( 30 Sek.) ended: 01-07-2004 / 09:35:38 (30520 Sek.).
benchmark2.pov rendered in 3200x2400 in 4 Tiles (1,4)-(V) from 01-07-2004 / 01:07:00 to 01-07-2004/09:35:43. Tiling, copying ( 17 Sek.) ended: 01-07-2004 / 09:35:59 (30540 Sek.).
Ani wood1.pov rendered in 160x120 in 1 Tiles (1,1)-(-) from 01-08-2004 / 09:47:01 to 01-08-2004/09:58:57. Tiling, copying ( 1 Sek.) ended: 01-08-2004 / 09:58:57 (716 Sek.).
benchmark.pov rendered in 160x120 in 2 Tiles (2,1)-(H) from 01-08-2004 / 10:46:28 to 01-08-2004/10:57:30. Tiling, copying ( 0 Sek.) ended: 01-08-2004 / 10:57:30 (662 Sek.).

The "times.log"contains informations how long each file has been in the SMPOV-Rendering-Pipeline, as well as information how many seconds have been used for appending the tiles and copying the final picture. There have been changes about time-measurement before Distri 15, which make these results more reliable, even if the rendering goes over several days or weeks.
 
 
Hits | Theo Gottwald * Wolfartsweierer Str.1 * 76131 Karlsruhe | Telefon (07 21) 9 66 33-00 | Fax (07 21) 9 66 33-99 |Info@it-berater.org | Stand: 01/13/2004