https://degangwang.com/tags/pyspark/Recent content in PySpark on Degang WangHugoen-usSun, 10 May 2026 00:00:00 -0500https://degangwang.com/2026/05/10/databricks-rwd-workflow/Sun, 10 May 2026 00:00:00 -0500https://degangwang.com/2026/05/10/databricks-rwd-workflow/<h2 id="why-not-just-load-everything-locally">Why Not Just Load Everything Locally?</h2> <p>Real-world data (RWD) — claims, EHR, registries — is massive. A single claims database can have billions of rows across diagnosis, procedure, and pharmacy tables. You cannot <code>pd.read_csv()</code> your way through it.</p> <p>The workflow that works:</p> <ol> <li><strong>Manipulate data in Databricks</strong> — filter, join, aggregate using SQL on distributed compute</li> <li><strong>Create an analytic-ready table</strong> — a focused cohort with only the columns and rows you need</li> <li><strong>Bring the result to your analytics tool</strong> — Python, R, or SAS for statistical analysis and visualization</li> </ol> <p>This post walks through that workflow using a diabetes cohort as an example. While we use Databricks here, the same pattern applies to any big data computing engine — Snowflake, Impala, AWS Athena, or Google BigQuery. The principle is the same: reduce data remotely, then bring the result to your analytics tool.</p>https://degangwang.com/2026/05/08/databricks-connect/Fri, 08 May 2026 00:00:00 -0500https://degangwang.com/2026/05/08/databricks-connect/<h2 id="why-databricks-connect">Why Databricks Connect?</h2> <p>The Databricks web notebook is great for interactive exploration, but sometimes you want to:</p> <ul> <li>Use your preferred local IDE (VS Code, PyCharm, etc.)</li> <li>Version control your code with git</li> <li>Run scripts in CI/CD pipelines</li> <li>Debug with local tools</li> </ul> <p>Databricks Connect lets you do all of this while still using a remote Databricks cluster for compute.</p>