How does the script gem work with Spark code if it’s running in a dbt / SQL project?
Hi @Andrew_G! When you use a script gem in a SQL project, the input is saved as a temporary table. Then, the script runs in Spark serverless. Essentially, the script gem provides a SparkSession and allows you to run custom PySpark code (see the doc here, which has a helpful table with execution environments based on your SQL warehouse provider).
The output is also stored as a temporary table, so that the following SQL transformations can run after it.