1

I have multiple Python processes, running on a server (RPI5) by cron, which read data from web API's, and write it then into a common InfluxDB database - on the same bucket.

However, some of the data is lost. The code to write into Influx is:

influxdb_client = InfluxDBClient(url=url, token=token, org=org) 
...

def f(df): 
  write_api = influxdb_client.write_api()
  ...
  record = [] 
  for i in range(df.shape[0]):
    point = Point(measurement).tag("location", ...).time(...)  
    for col in list(df.columns): 
      value = df.loc[i, col]  
      point = point.field(col, value) 
      record += [point]   
  write_api.write(bucket=bucket, org=org, record=record) 

...
## Let df be a data.frame with 20-500 rows, and 10-20 columns.   
f(df) 

What could be the reason of this issue? Some problem with asynchronous/synchronous?

Thx

4
  • Have a number of issues with this, have you checked the logs for errors, how do you actually know data is missing ? . defining a function f() has taking no arguments then calling it with an argument should throw a TypeError ... , is this actual code ...
    – ticktalk
    Commented Jul 5 at 6:25
  • Ok, I missed the "df" argument in f(df), when simplifying the code for the example. The data loss is shown in Influx Data Explorer, but also read clients. I struggle between SYCHRONOUS api and ASYNC. Both are not really working as expected for this parallel use case. Commented Jul 5 at 10:16
  • maybe InfluxDB is not "thread safe". maybe when you use it with many processes then it can't control it and finally some processes delete values from other processes. Maybe you should have only one process which access InfluxDB and other should use queue to send data to this process.
    – furas
    Commented Jul 5 at 12:39
  • Given I used first MongoDB as a database, I am sort of disappomited about InfluxDB. Mongo had no struggles at all with parallel processes writing and reading. Sigh. Adding now async or queue stuff makes the code more complex, which I wanted to avoid. Commented Jul 8 at 7:14

0