Cleaning Berkeley Earth’s BEST Gridded Daily Temperature Data

You may have recently seen air quality maps produced by the Berkeley Earth group, especially in the wake of the horrific Camp Fire whose death toll now exceeds 80. For example, here's their real-time visualization of PM2.5 concentrations. For several years, the Berkeley Earth team had primarily worked on producing gridded weather datasets with solid … Continue reading Cleaning Berkeley Earth’s BEST Gridded Daily Temperature Data

Large Stata Datasets and False Errors about ‘Duplicates’

Variable storage types exercise more importance when working with larger datasets, and variables with more digits. I'm reminded of this because of an error message Stata threw while trying to perform a long reshape, claiming duplicate entries of the ID variable. That was obviously not the case, since the _n id was uniquely created, and … Continue reading Large Stata Datasets and False Errors about ‘Duplicates’

Identify nth-Degree Neighbors Using R’s Simple Features Package, Simply

You're more likely to complain about the neighbors upstairs who are making noise after midnight than those in an apartment two buildings away. Proximity matters and that's patently obvious, but oftentimes it takes a bit of work to identify who is close and who isn't. While raster data is packaged in a consistent gridded format for … Continue reading Identify nth-Degree Neighbors Using R’s Simple Features Package, Simply

Converting .TXT/.GRD Climate Data Files to netCDF Format

Climate data is packaged and distributed in too many file formats.  Under ideal circumstances, you could easily convert data from formats you're not familiar with (and don't have scripts to handle), to those that you do. This is why analogous tools like Stat/Transfer for statistical databases often used by social scientists, are so helpful.  If a stranger … Continue reading Converting .TXT/.GRD Climate Data Files to netCDF Format

Effortlessly Merging 1,000s of Raw Data Files with Stata

I frequently have to consolidate 100s or 1,000s of raw data files into Stata, that are potentially stored in numerous and potentially unknown folders and subfolders, and have developed a workflow that I think is useful.  This approach means I don't need to know file names or paths, and can instead assign search parameters that … Continue reading Effortlessly Merging 1,000s of Raw Data Files with Stata

Stata-Latex esttab Regression Table Output Streamlining

Researchers spend an excessive amount of time getting up to speed with a field's chosen tools and methods, excessive because there is often a consensus on best practice and yet those best practices are not made common knowledge.  I think the CS and statistics communities have this right in their pushing for open data, transparency, and reproducibility … Continue reading Stata-Latex esttab Regression Table Output Streamlining