Join the Databricks Learning Festival (Virtual)! Mark your calendars from 10 July to 24 July 2024! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career with D...
- 45718 Views
- 116 replies
- 22 kudos
06-03-2024
We are enhancing the Community login process to provide a more secure and streamlined experience for all users. To avoid losing access to your Community account, please update your primary and secondary email addresses now (refer to “Action Required”...
- 2850 Views
- 0 replies
- 4 kudos
3 weeks ago
Your feedback is crucial to us and directly influences how we innovate and improve our customer experience. To share your success and feedback with Databricks, please visit this link. The first 20 people to submit a completed survey response will ...
- 741 Views
- 3 replies
- 5 kudos
a week ago
Learn generative AI basics in 120 minutes Start building your generative AI expertise with this self-paced course — and then earn an industry-recognized Databricks completion certificate. These five short tutorial videos will help you learn generativ...
- 347 Views
- 0 replies
- 1 kudos
Monday
Community Activity
- 18 Views
- 0 replies
- 0 kudos
Convert Date String to Date type Returning null
Hi,I am using Databricks SQL and I am converting a integer field which is of format ('20240719' or 'yyyyMMdd'),now I am able to convert it to date type using to_date(forecastedHorizonPeriodDate,'yyyyMMdd')I then tried to change the format of this dat...
- 18 Views
- 0 replies
- 0 kudos
- 61 Views
- 2 replies
- 0 kudos
DevOps CICD for Databricks Schema
Does anyone know how to do DevOps CICD pipeline to deploy changes to a Databricks Catalog? I've added a couple tables in DEV and need to deploy in Prod. Could'nt find any info on this anywhere.
- 61 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Thanks @Kaniz_Fatma , the links refer mostly deployments of workspace objects rather than changes in database objects, eg. a new schema, table, or new column added to a table etc. I'll keep looking...
- 0 kudos
- 1288 Views
- 2 replies
- 0 kudos
How to Add value to Comment Column in Databricks Catalog View for Foreign table (Azure SQL)
Hi.Struggling to add description by script to the comment column within the catalog view in Databricks, particularly for foreign/external tables sourced from Azure SQL.I have no issue doing that for delta tables. Also for information schema columns f...
- 1288 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@Kaniz_Fatma Sorry, I did read this way back but forgot to reply.Thanks for information.And all those steps I found as well and for "3. Programmatic Approach (For Azure SQL)" if you are adding to sql any description that way I don't think it will app...
- 0 kudos
- 87 Views
- 3 replies
- 0 kudos
Resolved! DLT Online Table with VNnet Enable on Blob Storage Get 403 Issue
I am trying to create an online table in a Unity catalog. However, I get a GET, 403 error. DataPlaneException: Failed to start the DLT service on cluster . Please check the stack trace below or driver logs for more details. com.databricks.pipelines....
- 87 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
I figured it out. It was because of the Network Connectivity Configurations. I did not have one setup with a private endpoint connection to the ADLS Gen2. I followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/security/...
- 0 kudos
- 653 Views
- 2 replies
- 1 kudos
What are best practices for the Datatabricks workflow jobs?
Recommendations on how many tables per workflow?inter dependency between the workflows?Custom schedule?Monitoring?Reports?
- 653 Views
- 2 replies
- 1 kudos
- 49 Views
- 2 replies
- 0 kudos
Cannot manage permissions for Databricks workspace disconnected from metastore
So in the Databricks Account console, under "Workspaces", I am trying to configure permissions for a specific workspace that was formerly attached to a metastore, but the metastore is now deleted (on purpose).No errors show up in the Account console ...
- 49 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @m997al, Ensure that the workspace is set to operate in legacy mode. This might involve checking the workspace settings to confirm that it is not expecting a Unity Catalog metastore.
- 0 kudos
- 888 Views
- 4 replies
- 0 kudos
Is there any better way to manage workflow dependencies using databricks workflows
Looking for solution on how to manage job dependencies using databricks workflows.
- 888 Views
- 4 replies
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
Can't load model from UC due to DBFS issue
I want to load a model I have registered in Unity Catalog using a Shared cluster, but it seems to be trying to use dbfs under the hood and it gives me an error.I am using DBR 13.3 LTS and mlflow-skinny[databricks]==2.14.3My code import mlflow mlflow...
- 20 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Mount point in unity catalog
Hi Everyone,In my existing notebooks we have used mount points url as /mnt/ and we have more than 200 notebooks where we have used the above url to fetch the data/file from the container. Now as we are upgrading to unity catalog these url will no lon...
- 32 Views
- 0 replies
- 0 kudos
- 6394 Views
- 11 replies
- 5 kudos
What are the different ways to pull the log data from Splunk to Databricks?
Hi,I have recently started Splunk Integration with Databricks. Basically I am trying to ingest the data from Splunk to Databricks. I have gone through the documentation regarding Splunk Integration. There are some basic information about the integrat...
- 6394 Views
- 11 replies
- 5 kudos
Latest Reply
- 5 kudos
Hi @Arch_dbxlearner Did you done integration with splunk if yes can you please help
- 5 kudos
- 69 Views
- 2 replies
- 0 kudos
Data factory logs into databricks delta table
Hi Databricks Community,I am looking for a solution to efficiently integrate Azure Data Factory pipeline logs with Databricks at minimal cost. Currently, I have a dashboard that consumes data from a Delta table, and I would like to augment this table...
- 69 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @8b1tz, configure ADF to send pipeline logs to an Azure Storage Account, Azure Log Analytics, or Event Hubs. This ensures that logs are persisted and can be accessed by Databricks.If you need more detailed guidance or run into specific issues, fee...
- 0 kudos
- 798 Views
- 4 replies
- 3 kudos
Resolved! Databricks Pyspark Dataframe error while displaying data read from mongodb
Hi ,We are trying to read data from mongodb using databricks notebook with pyspark connectivity.When we try to display data frame data using show or display method , it gives error "org.bson.BsonInvalidOperationException:Document does not contain key...
- 798 Views
- 4 replies
- 3 kudos
Latest Reply
- 3 kudos
UPDATE:Installing mongo-spark-connector_2.12-10.3.0-all.jar from Maven does NOT require the JAR files below to be installed on the cluster to display the dataframebsonmongodb-driver-coremongodb-driver-syncAlso, I noticed that both DBR 13.3 LTS and 14...
- 3 kudos
- 418 Views
- 4 replies
- 2 kudos
Accessing shallow cloned data through an External location fails
I have two external locations. On both of these locations I have `ALL PRIVILEGES` access.I am creating a table on the first external location using the following command:%sqlcreate or replace table delta.`s3://avinashkhamanekar/tmp/test_table_origina...
- 418 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
Hello , This is an underlying exception that should occur with any SQL statement that require access to this file: part-00000-36ee2e95-cfb1-449b-a986-21657cc01b22-c000.snappy.parquet It looks like the Delta log is referencing a file that doesn't exi...
- 2 kudos
- 35 Views
- 0 replies
- 0 kudos
Databricks job stalls without error, unable to pin-point error, all compute metrics seem ok
I have a job that gets stuck on "Determining DBIO File fragment" and I have not been able to figure out why this job keeps getting stuck. I monitor the job cluster metrics through out the job and it doesnt seem like its hitting any bottlenecks with m...
- 35 Views
- 0 replies
- 0 kudos
- 741 Views
- 3 replies
- 7 kudos
Submit your feedback and win a $25 gift card!
Your feedback is crucial to us and directly influences how we innovate and improve our customer experience. To share your success and feedback with Databricks, please visit this link. The first 20 people to submit a completed survey response will ...
- 741 Views
- 3 replies
- 7 kudos
Latest from our Blog
Foundation Models API Prompting Guide 1: Lifecycle of a Prompt
This is the first part of a guide on writing prompts for models accessible via the Databricks Foundation Model API, such as DBRX and Llama 3. Prompting, whether in the context of interacting with a ch...
Why DBSQL is Best for BI Workloads - Part 5: Query Optimization with Primary Key Constraints
Authors: Andrey Mirskiy (@AndreyMirskiy) and Marco Scagliola (@MarcoScagliola) IntroductionQuery optimization using primary key constraints - OverviewDemo ScenarioPreparationExecuting Sample Query wit...
MLOps Gym - Beginners Guide to Monitoring
Eventually, all software systems will encounter failures, and AI systems are not exempt from this reality. Failures in traditional software systems can stem from various sources, including infrastruct...
Enhancing the new PySpark Custom Data Sources Streaming API: Adding Progress Tracking Capability
A new PySpark Custom Data Sources API was introduced at DAIS 2024 which simplifies the integration with custom data sources in Apache Spark. Imagine seamlessly streaming incremental data from any API ...
