Unlocking Databricks: Advanced Techniques for Optimizing Data Warehouse Performance & UDF-driven Processing