Questions tagged [pipeline]
A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.
pipeline
5,676
questions
0
votes
0
answers
46
views
Exit handler never called! This is an error with npm itself
I am trying to run tests in the pipeline for a react application. Script looks as below:
Earlier the first three lines weren't there. I have tried running the command npm install with --force/--legacy-...
- 266
0
votes
0
answers
6
views
Passing table names as parameters from dataflow to notebook in pipelines
I want to create a pipeline with two elements. One is a dataflow. This dataflow creates 5 tables in my lakehouse. The second element of the pipeline is a notebook. I want the notebook to load each of ...
- 1
0
votes
0
answers
31
views
Make the pipeline fail if a specific sentence appears in the log – Find a substring, but ignore a specific substring
In my first question, the goal was to intentionally make the CI/CD pipeline fail if a transformation is not possible in the Web.Live.Config. I had to save the build in a variable. If a transformation ...
- 497
0
votes
0
answers
14
views
Not able to create job in dataflow for streaming data
I am executing my Apache-beam code in google cloud shell, I am able to execute code without errors, but jobs not creating in data flow.
**below roles I assigned to service account
**
Dataflow Worker, ...
- 13
-1
votes
0
answers
22
views
How do I pass a Java Maven project's version value from the pom.xml to a Bitbucket pipelines YAML file?
I have a Spring Boot Maven Java project. My POM file has a version number property in it of 1.2.3, as seen below:
<project ...>
<modelVersion>#.#.#</modelVersion>
<groupId>...
- 2,474
1
vote
1
answer
51
views
Make the pipeline fail if a specific sentence appears in the log
The goal of the task is to verify a build process in a GitLab CI/CD pipeline using PowerShell without saving any files on the server. Specifically, the task involves checking for configuration ...
- 497
1
vote
1
answer
21
views
Running a gitlab-ci job when a file exists
I'm trying to configure gitlab-ci to run two jobs depending on the presence of a Dockerfile using the exists directive.
I want job01 to be executed by tag, but only if there is no Dockerfile, and ...
- 87
0
votes
0
answers
11
views
Predicting new data with a model - preprocessing
I have studied a dataset and built a model. During the exploration I made several transformations and before training the model I applied a pandas pipe to my train dataset. This pipe includes ...
- 175
0
votes
1
answer
43
views
Iterate through an API until no records found through ADF
I have an ADF pipeline where I have an API call where it returns data in chunks (by 1000 max). This pipeline works if you only need 1000 users. I have been trying to figure a way to call the API again ...
- 35
0
votes
0
answers
29
views
GStreamer VC++ for converting RTP (Payload Type 96 (PT-96) ) to wav audio file
My application requirement is that I need to sniff network packets from the network card and filter RTP packet of payload type 96 (PT-96) and convert and save them in a wav audio file.
I have filtered ...
0
votes
0
answers
21
views
GitLab yaml config contains unknown keys: secrets, when using trigger
I have a GitLab Job, which needs secrets from our vault and triggers a downstream pipeline:
myJob:
secrets:
API_KEY:
vault: myVault/ApiKey@secret
file: false
USERNAME:
...
- 63
0
votes
1
answer
22
views
Gitlab Pipeline - Use WiX Toolset v3 in Gitlab Pipeline
I need to run WiX Toolset V3.14.1 light command during Gitlab Pipeline, but it's never available to run.
Job configuration
tags:
- saas-windows-medium-amd64 # to use Windows runner
Job execution
...
-3
votes
1
answer
38
views
Efficient scaling scikit-learn pipelines with dask for large datasets [closed]
I am working on this ML project using scikit-learn and facing a issue where a large dataset doesn't fit into the memory. I have heard that Dask can be used to scale teh pipelines for large datasets. I ...
3
votes
1
answer
68
views
+50
Conditionally display the drop-down list of values of a variable based the branch name GitLab pipeline
The globalvariables: section of my .gitlab-ci.yml file has been constructed like below:
variables:
HOST_NAME:
description: Select a host name to deploy
value: web-server-dev-01
options:
...
- 1,514
0
votes
0
answers
35
views
Llama-3-70B with pipeline cannot generate new tokens (texts)
I have sucessfully downloaded Llama-3-70B, and when I want to test its "text-generation" ability, it always outputs my prompt and no other more texts.
Here is my demo code (copied from ...
- 11