https://degangwang.com/tags/rwd/Recent content in RWD on Degang WangHugoen-usSun, 10 May 2026 00:00:00 -0500https://degangwang.com/2026/05/10/databricks-rwd-workflow/Sun, 10 May 2026 00:00:00 -0500https://degangwang.com/2026/05/10/databricks-rwd-workflow/<h2 id="why-not-just-load-everything-locally">Why Not Just Load Everything Locally?</h2> <p>Real-world data (RWD) — claims, EHR, registries — is massive. A single claims database can have billions of rows across diagnosis, procedure, and pharmacy tables. You cannot <code>pd.read_csv()</code> your way through it.</p> <p>The workflow that works:</p> <ol> <li><strong>Manipulate data in Databricks</strong> — filter, join, aggregate using SQL on distributed compute</li> <li><strong>Create an analytic-ready table</strong> — a focused cohort with only the columns and rows you need</li> <li><strong>Bring the result to your analytics tool</strong> — Python, R, or SAS for statistical analysis and visualization</li> </ol> <p>This post walks through that workflow using a diabetes cohort as an example. While we use Databricks here, the same pattern applies to any big data computing engine — Snowflake, Impala, AWS Athena, or Google BigQuery. The principle is the same: reduce data remotely, then bring the result to your analytics tool.</p>https://degangwang.com/2026/05/08/mgb-ehr-data/Sun, 10 May 2026 00:00:00 -0500https://degangwang.com/2026/05/08/mgb-ehr-data/<h2 id="what-is-mass-general-brigham">What Is Mass General Brigham?</h2> <p>Mass General Brigham (MGB), formerly Partners HealthCare, is one of the largest academic health systems in the United States. It includes Massachusetts General Hospital, Brigham and Women&rsquo;s Hospital, and several other hospitals and community health centers across New England. The system serves millions of patients and generates rich longitudinal electronic health record (EHR) data.</p>