Questions tagged [dataframe]
A data frame is a 2D tabular data structure. Usually, it contains data where rows are observations and columns are variables and are allowed to be of different types (as distinct from an array or matrix). While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.
dataframe
146,882
questions
0
votes
0
answers
17
views
spark sql query returns column has 0 length but non null
I have a spark dataframe for a parquet file. The column is string type.
spark.sql("select col_a, length(col_a) from df where col_a is not null")
+-------------------+------------------------...
- 159
1
vote
2
answers
33
views
Converting JSON list with multiple nested dictionaries to csv or excel
I have a JSON that I download from a website that has multiple nested dictionaries inside the main list. This is a very simplified version of it.
[
{
"id": 1,
"...
- 11
3
votes
2
answers
56
views
How do I get variable length slices of values using Pandas?
I have data that includes a full name and first name, and I need to make a new column with the last name. I can assume full - first = last.
I've been trying to use slice with an index the length of ...
- 65
2
votes
0
answers
52
views
How to compare rows within the same csv file faster
I have a csv file containing 720,000 rows with and 10 columns, the columns that are relevant to the problem are ['timestamp_utc', 'looted_by__name', 'item_id', 'quantity']
This File is logs of items ...
- 31
0
votes
0
answers
20
views
Dataframe replace columns and save to new df
I loaded in a .dat file to a pandas dataframe. Two of the columns are mean and error. I used the values in these two columns to create a randomized value for mean. I want to replace the mean column in ...
0
votes
0
answers
13
views
how to apply an expression from a column to another column in pyspark dataframe
I would like to know if this possible to apply.
for example i have this table:
new_feed_dt regex_to_apply expr_to_apply
053021 | _(\d+) | date_format(to_date(new_feed_dt, '...
0
votes
0
answers
24
views
How to calculate the Relative Strength Index (RSI) through record iterations in pandas dataframe
I have created a pandas dataframe as follows:
import pandas as pd
import numpy as np
ds = { 'trend' : [1,1,1,1,2,2,3,3,3,3,3,3,4,4,4,4,4], 'price' : [23,43,56,21,43,55,54,32,9,12,11,12,23,3,2,1,1]...
- 1,392
0
votes
0
answers
37
views
Using Polars expressions to apply `eval()` to a column
I would like to achieve the following via Polars expressions, as opposed to mapping the elements row-by-row, but I have not been able to figure out a way.
import polars
def foo():
return 1 + 1
...
- 63
1
vote
1
answer
39
views
How to group rows based on column ID in a pandas dataframe?
I have below the dataframe below df1:
ID Label Value
id_1 A
id_1 B
id_1 C
id_1 D
id_1 E
id_1 10
id_1 20
id_1 30
id_2 F
id_2 G
...
- 25
0
votes
1
answer
56
views
split pandas datafram based on given row string
I have a text file with a data set of the form
Line 1
Line 2
!
1.01499999 0.504999995 6.19969398E-7 5.38933136E-7 1.35450875E-6
1.74000001 0.220000029 7.92876381E-6 4.1831604E-6 6.61433387E-6
2....
- 2,019
1
vote
5
answers
77
views
String Manipulation based on Char Length in a dataframe
I wanted to do some string manipulation based on Char length condition.
I have this table, let's called it sample table.
RiskCode
A01
A02.999
I want to transform the RiskCode column in sample ...
3
votes
1
answer
34
views
Resample ohlc pandas
I have a problem with resample in pandas.
There is a dataframe with the data specified below.
Datetime as an index
I need to generate a resample of a 3-hour interval, even if there is only 2 hours of ...
- 31
1
vote
4
answers
63
views
Shift part of row in dataframe to new row
I have a dataframe (pandas) that I want to transform for displaying purposes.
Therefore I want to shift some parts of the dataframe to new rows like below :
col1 col2 col_to_shift col_not_to_shift1 ...
- 623
1
vote
2
answers
59
views
Interpolate dataframe in R subject to totals
I have the following dataset which has the number of A, B and C both in 2000 and 2005. I now need to inpolate the dataframe subject to the constraint that the sum of A, B and C must be consistent with ...
- 25
-2
votes
0
answers
15
views
i am in the process of using pandasai, but keep getting this error from pandas.compat import is_numpy_dev as _is_numpy_dev
from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
i am running numpy version 2
python version 3.11
pandas 1....
- 71