lib/gis: Add a helper function to determine the number of threads for OpenMP

[cyliang368]

Based on the discussion in #3917, this PR adds a function to make the parser determine the number of threads, mainly for the standard option G_OPT_M_NPROCS.

• nprocs > 0 for min(nprocs, max_threads_on_device)
• nprocs <= 0 for max(max_threads_on_device + nprocs, 1) (e.g., nprocs=0 for max_threads_on_device, nprocs=-1 for max_threads_on_device - 1, and nprocs=-10000 with 24 threads for 1, etc.)

[cyliang368]

This PR failed on four tests that ran t.rast.what in parallel. Here is a part of its detail:

======================================================================
FAIL: test_row_stdout_where_parallel_cat (__main__.TestRasterWhat)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "temporal/t.rast.what/testsuite/test_what.py", line 273, in test_row_stdout_where_parallel_cat
    self.assertLooksLike(text, str(t_rast_what.outputs.stdout))
  File "etc/python/grass/gunittest/case.py", line 202, in assertLooksLike
    self.fail(self._formatMessage(msg, standardMsg))
AssertionError: "cat|x|y|start|end|value
1|[115](https://github.com/OSGeo/grass/actions/runs/9669690114/job/26676756189?pr=3929#step:14:116).0043586274|36.3593955783|2001-04-01 00:00:00|2001-07-01 00:00:00|200
2|79.6816763826|45.2391522853|2001-04-01 00:00:00|2001-07-01 00:00:00|200
3|97.4892579600|79.2347263950|2001-04-01 00:00:00|2001-07-01 00:00:00|200
1|115.0043586274|36.3593955783|2001-07-01 00:00:00|2001-10-01 00:00:00|300
2|79.6816763826|45.2391522853|2001-07-01 00:00:00|2001-10-01 00:00:00|300
3|97.4892579600|79.2347263950|2001-07-01 00:00:00|2001-10-01 00:00:00|300
1|115.0043586274|36.3593955783|2001-10-01 00:00:00|2002-01-01 00:00:00|400
2|79.6816763826|45.2391522853|2001-10-01 00:00:00|2002-01-01 00:00:00|400
3|97.4892579600|79.2347263950|2001-10-01 00:00:00|2002-01-01 00:00:00|400
" does not correspond with "cat|x|y|start|end|value
1|115.0043586274|36.3593955783|2001-04-01 00:00:00|2001-07-01 00:00:00|200
1|115.0043586274|36.3593955783|2001-07-01 00:00:00|2001-10-01 00:00:00|300
1|115.0043586274|36.3593955783|2001-10-01 00:00:00|2002-01-01 00:00:00|400
2|79.6816763826|45.2391522853|2001-04-01 00:00:00|2001-07-01 00:00:00|200
2|79.6816763826|45.2391522853|2001-07-01 00:00:00|2001-10-01 00:00:00|300
2|79.6816763826|45.2391522853|2001-10-01 00:00:00|2002-01-01 00:00:00|400
3|97.4892579600|79.2347263950|2001-04-01 00:00:00|2001-07-01 00:00:00|200
3|97.4892579600|79.2347263950|2001-07-01 00:00:00|2001-10-01 00:00:00|300
3|97.4892579600|79.2347263950|2001-10-01 00:00:00|2002-01-01 00:00:00|400
"

----------------------------------------------------------------------
Ran 17 tests in 38.755s
FAILED (failures=4)
========================================================================
FAILED ./temporal/t.rast.what/testsuite/test_what.py (4 tests failed)

In the test, the args in SimpleModule still go through the parser I modified in this PR. Although OpenMP is not supported, this Python module (t.rast.what) can still be parallelized by subprocess in Python. Without OpenMP, nprocs=4 is passed to this Python module before this PR, but nprocs is changed to 1 in this PR before it is passed to this Python module. That's why the tests failed.

Original: t.rast.what nprocs=4 -> parser (nothing happens) -> nprocs=4 main function in Python
This PR: t.rast.what nprocs=4 -> parser (nprocs is set to 1) -> nprocs=1 main function in Python

grass/temporal/t.rast.what/testsuite/test_what.py

Lines 275 to 301 in 0938951

	def test_row_stdout_where_parallel2(self):
	"""Here without output definition, the default is used then"""
	t_rast_what = SimpleModule(
	"t.rast.what",
	strds="A",
	points="points",
	flags="n",
	where="start_time > '2001-03-01'",
	nprocs=4,
	overwrite=True,
	verbose=True,
	)
	self.assertModule(t_rast_what)
	text = """x|y|start|end|value
	115.0043586274|36.3593955783|2001-04-01 00:00:00|2001-07-01 00:00:00|200
	79.6816763826|45.2391522853|2001-04-01 00:00:00|2001-07-01 00:00:00|200
	97.4892579600|79.2347263950|2001-04-01 00:00:00|2001-07-01 00:00:00|200
	115.0043586274|36.3593955783|2001-07-01 00:00:00|2001-10-01 00:00:00|300
	79.6816763826|45.2391522853|2001-07-01 00:00:00|2001-10-01 00:00:00|300
	97.4892579600|79.2347263950|2001-07-01 00:00:00|2001-10-01 00:00:00|300
	115.0043586274|36.3593955783|2001-10-01 00:00:00|2002-01-01 00:00:00|400
	79.6816763826|45.2391522853|2001-10-01 00:00:00|2002-01-01 00:00:00|400
	97.4892579600|79.2347263950|2001-10-01 00:00:00|2002-01-01 00:00:00|400
	"""
	self.assertLooksLike(text, str(t_rast_what.outputs.stdout))

My questions are:

1. Should we let Python modules run in parallel by subprocess when OpenMP is not supported?
2. If we want to do so, how can I avoid the situation I mentioned above?

@HuidaeCho @wenzeslaus @marisn Do you have any suggestions?

[petrasovaa]

Perhaps the function should be called in the particular C tools, that would avoid the problem with Python tools.

Also, there is the NPROCS variable, the function should take into account:
https://github.com/OSGeo/grass/blob/main/lib/gis/parser_standard_options.c#L765

[cyliang368]

Perhaps the function should be called in the particular C tools, that would avoid the problem with Python tools.

I checked and found that every C module parallelized by OpenMP has the keyword "parallel", which I think it should. Thus, I use this keyword to distinguish whether a multithread/multiprocess is spawned by C or Python.

Also, there is the NPROCS variable, the function should take into account: https://github.com/OSGeo/grass/blob/main/lib/gis/parser_standard_options.c#L765

The parser runs after options are defined, so the function can also handle an environment variable.

[petrasovaa]

Perhaps the function should be called in the particular C tools, that would avoid the problem with Python tools.

I checked and found that every C module parallelized by OpenMP has the keyword "parallel", which I think it should. Thus, I use this keyword to distinguish whether a multithread/multiprocess is spawned by C or Python.

That's probably not a good strategy, the parallel keyword was meant for any parallelization, not just openmp, we would have to have a new keyword, but I am not sure using a keyword is good idea anyway.

Also, there is the NPROCS variable, the function should take into account: https://github.com/OSGeo/grass/blob/main/lib/gis/parser_standard_options.c#L765

The parser runs after options are defined, so the function can also handle an environment variable.

[wenzeslaus]

Perhaps the function should be called in the particular C tools...

That I think is a good approach. Other values require this approach too. RGB colors are one example, but even the current string provided to nprocs needs to be converted to integer.

[cyliang368]

I left the parser unchanged and just added a helper function instead. A C module can call this function if it is needed.

That's probably not a good strategy, the parallel keyword was meant for any parallelization, not just openmp, we would have to have a new keyword, but I am not sure using a keyword is good idea anyway.

Though this PR does not use the keyword now, I think new keywords should be considered. As you said, parallelizations could be done by different libraries. New keywords to indicate which libraries/methods are used can help with documentation and maintenance.

[wenzeslaus]

New keywords to indicate which libraries/methods are used can help with documentation and maintenance.

Agreed. I think this will work well when we add description for each keyword, an extended version of what we have for topics (topic keywords).

[wenzeslaus]

Should the function just return number of threads as an integer?

Can you please provide a breakdown how this will behave with the default value for nprocs and the possibility to change that using g.gisenv or settings in GUI? (Previously mentioned by @petrasovaa)

[cyliang368]

Should the function just return number of threads as an integer?

You are right. I make it return the value now, and it won't change the answer of the nprocs option.

Can you please provide a breakdown how this will behave with the default value for nprocs and the possibility to change that using g.gisenv or settings in GUI? (Previously mentioned by @petrasovaa)

Let's take r.texture as an example.

1. The option objects are defined here in main.c under r.texture.
   

   grass/raster/r.texture/main.c

   Lines 107 to 113 in b5bfcd5

   	parm.input = G_define_standard_option(G_OPT_R_INPUT);
   	parm.output = G_define_standard_option(G_OPT_R_BASENAME_OUTPUT);
   	parm.nproc = G_define_standard_option(G_OPT_M_NPROCS);
   	parm.size = G_define_option();

2. The G_define_standard_option function is called.
   

   grass/lib/gis/parser_standard_options.c

   Line 139 in b5bfcd5

   struct Option *G_define_standard_option(int opt)

3. The environment variable is fetched here by G_getenv_nofatal.
   

   grass/lib/gis/parser_standard_options.c

   Lines 757 to 770 in b5bfcd5

   	case G_OPT_M_NPROCS:
   	Opt->key = "nprocs";
   	Opt->type = TYPE_INTEGER;
   	Opt->required = NO;
   	Opt->multiple = NO;
   	Opt->answer = "1";
   	/* start dynamic answer */
   	/* check NPROCS in GISRC, set with g.gisenv */
   	memstr = G_store(G_getenv_nofatal("NPROCS"));
   	if (memstr && *memstr)
   	Opt->answer = memstr;
   	/* end dynamic answer */
   	Opt->description = _("Number of threads for parallel computing");
   	break;

4. Then, the G_parser is called after all option and flag are defined.
   

   grass/raster/r.texture/main.c

   Lines 166 to 167 in b5bfcd5

   	if (G_parser(argc, argv))
   	exit(EXIT_FAILURE);

I put the helper function in G_parser in the previous commits. Even if the environment variable changes, the parser will get it and handle it. However, we want particular C modules instead of the parser to call it, so it does not matter now. That's my understanding. I hope this is what you need.