Rerun regressions from reproduction packages and store all relevant information in a database that allows to quickly re-run particular regressions and modify them, e.g. use different standard error.
An initial reproduction, mainly by repboxStata and repboxRegStata. Generates basic regression infos and important parcels like stata_run_cmd.Rds. The regression info is stored as variant so
DRF (Direct Replication Format), mrb_stata.do and sb_raw. Stata reproduction by metaregBase. Using repboxDRF we determine (guess) all relevant data modification commands in stata_run_cmd.Rds and make another Stata replication run (sb_raw) that only uses the data modification commands and stores again regression infos. The generated code is in /metareg/base/stata_code/mrb_stata.do
Note that we have not yet the full Stata base variant sb, since to generate all needed parcels like regvar, we need to analyze the data set in R, e.g. to expand Stata abbreviations.
Re-run data preparation in R. Main translation via stata2r package. Generate all parcels that need to analyse the data set used in the regression. If successful, we have a complet sb variant.
Translate the actual regression using the regtranslate package. If also successful we have the rb variant.
regcheck is the main parcel that notes whether the whole chain lead consistent results or which variants failed to run or had errors. Check if run, and whether coefficients / standard errors are same. We allow for smaller numerical differences though and state the maximum difference. We then can generate different reports, e.g. using repboxReportDo package.
Often not all regressions will be perfectly replicated in the first mrb run. That is due to the fact that not all Stata commands are translated to R. The next section describes the trade-offs and different repair strategies.
Key for the metareg pipeline is that for every regression we can get the actually prepared data set for this regression that results from applying all original data preparation steps that we run in the do file.
Most but not all types of data preparation Stata code will be translated to R. Cache files generated by Stata can allow R replications of regressions even if not all data preparation commands are translated. Consider two extreme scenarios:
Full translation: Every command between the data loading and regression that might possibly influence the data set used in the regression will be translated from Stata to R.
Full caching: For every regression we let Stata store the data set used in the regression (already incorporating if or in filters in the regression command).
Full caching is the most robust approach, but needs most disc space. Some reproduction packages have > 1000 regressions and large underlying data set. You might run out of disc storage quickly with a few reproduction packages.
Full translation can be very time intensive and hard to develop and debug. In particular not all commands that modify data are really needed for the data set of a particular regression.
The best practical solution is likely a trade-off. But the best strategy is not obvious. Consider the following example:
use mydata.dta
foreach xvar of varlist xvars {
replace wA = `xvar`^2
regress y `xvar' wA-wK
predict y_hat, xb
generate y_hat_`xvar' = y_hat
drop y_hat
}
While the predict, generate, drop commands modify the data set, these modifications are actually not needed here to run later regressions in the loop. But it is hard to assess which previous data modifications are needed for a regression or not.
Assume the loop is large and 100 regressions are run. Under the full translation approach we would need to translate and run all previous 99 regressions to compute the data set for 100th regression. Super time intensive and error prone. The full caching approach would save 100 data sets, even though essentially not needed here.
Our actual approach can be described as following:
Try to translate all data preparation commands unless there is an exception.
The commands for which default caches will be generated (see 2.) will not be translated.
stata2r shall not translate regression commands or post-regression commands, even if they are required for later regressions. One example is the predict function that creates a new column from the previous regression. Our default guess that such commands are not needed as data preparation for subsequent regressions may be reasonable. If not ok, our repair strategies may solve it.
One exception is when the e(sample) of a regression is used in the next regression. Here stata2r provides translation code that generates the e(sample) info. That might work without the complete regression spec. But not clear, actually how well it works. Caching e(sample) might be the better idea... actually.
Also regressions with xi: prefixes should be dealt in a suitable fashion if the later regressions use the generated variables... Not so clear how. Currently we cache xi commands. But a cache at the regression position also caches filters... Maybe we just need to have a cache before the subsequent regression...
Consider multiple regressions in a loop like:
foreach xvar of varlist xvars {
replace wA = `xvar`^2
regress y `xvar' wA-wK
predict y_hat, xb
generate y_hat_`xvar' = y_hat
drop y_hat
}
We make the general assumption that the i'th iteration does not depend on the commands in any earlier iteration. This does not always have to be the case, but it seems very reasonable that this assumption captures perhaps 99% of cases where regressions are run in a loop.
The file drf_loop_paths.R contains functions to automatically shorten the path_df in DRF consistent with this example. In the example above this means that for the i'th iteration the code path consists only of the code before the loop and inside the loop only the i'th iteration code before the regression command (here: replace wA = ...).
TO DO: Implement and think whether in mrb_stata.do we should keep the complete loop code as in natural order. This means we should have original path_df for mrb_stata.do and reduced ones later for R translation.
Currently, we add caches by default after a few commands.
The function repboxStata::repbox_always_cache_cmd specifies commands after which a cache will be always generated in the original reproduction run by repboxStata. Currently it is merge and joinby. Also our mrb_stata.do will directly use these caches. Reason: there are reproduction packages that merge with temporary files, it is hard to correctly store these temporary files for merge. So just caching after every merge and joinby seemed easier.
In addition our MRB stata run generates caches after the commands listed in metaregBase::mrb_stata_always_cache_commands(). Currently that is only xi. A clean R translation of xi seems very hard, as we don't know which default factor levels will be dropped by Stata and our current translation in stata2r often drops different levels than Stata.
We can have some heuristics to write additional caches in mrb_stata.do to shorten very long data preparation sequences. That can speed up the R analysis and also reduce the scope of translation errors. The exact heuristic should be parametrizable. Factors could be the number of regressions covered, no of steps eliminated and possibly data set size. Accounting for run time would make the algo not-deterministic which may be undesirable...
The current implementation will sometimes lead to errors in the R translation pipeline. Recall our loop example:
foreach xvar of varlist xvars {
regress y `xvar' wA-wK
predict y_hat, xb
generate y_hat_`xvar' = y_hat
drop y_hat
}
Consider the data preparation code for the regression in the 2nd iteration. Since we don't translate the predict command, the R translation of the generate y_hat_ ... command from the 1st iteration will throw an error because y_hat was not generated in our R pipeline.
If such an error occurs in a data preparation command with runid = err_runid, we store err_runid as a file in drf/r_err_runids and stop the current execution immediately. If we try to reproduce another regression, we check whether its data preparation path contains a runid that is marked as error. If yes, completely skip the replication of that regression.
So in our example. The 1st regression can be replication, the 2nd regression fails, because predict is not translated and the translated generate y_hat_ ... throws an error. The replications of all later regressions in the loop are then directly skipped since the generate y_hat_ ... from the first iteration is on the data preparation path of all regressions in the loop and was marked as throwing an error. So replication is halfway fast here, but only 1 regression is actually replicated.
So our main run, by construction can leave some regressions not replicated even though that should be possible. The idea is to have a sensible repair heuristic that balances speed / storage / success rate. We try different heuristics.
NOTE: Not yet implemented as described below
The first heuristic is to simply ignore failed translation commands.
Trade-off: May not work always. Want to fail early if e.g. in a loop the ignore does not work for earlier iterations, we can expect that it won't work for later iterations too? How can detect loops? If multiple runids share the same line and do file we are in a loop. I.e. we also know the loop length.
Try to add caches in a clever fashion. Trade-off: we don't want too many caches. If we would need a 100 caches or so in a loop to make the replication work, we may rather just leave those regression commands untranslated.
Finding the optimal cache position: a cache might be reusable by multiple regression command. Best cache at the latest position in the path after the last error that still serves the same number of regression PIDs than earlier positions after the last error. The function mrb_determine_repair_cache_runid already tries to implement that heuristic.
A repair is successful if regcheck is happy, i.e. the R regression (variant rb) yields the same coefficients than the mrb Stata (sb) and also sb and so are identical. However, even if we re-implemented the correct data set for the R regression regcheck might fail because our R translation via regtranslate just was not able to replicate Stata's regression results. E.g. for probit regression currently the R translation using fixest typically yields somewhat different coefficients than Stata's probit command.
So basing our test just on regcheck we might incorrectly think we are not able to repair Stata data preparation just because regression translation does not work. One way to check repair success of data preparation pipeline is to generate a cache directly for the regression command and the compare whether the R regresion using our repaired pipeline yields the same regression results than if using the direct cache. (Data sets might differ by columns that are irrelevant for the regression). If yes, we can delete the cache again. But of course that testing is a bit time intensive.
Assume we have a regression that can be well replicated via MRB. Still the data preparation pipeline may take very long and possibly contain commands not actually necessary to prepare the data set for the regression. It might be worthwhile to add profiling steps, e.g.
Strategically add some caches that a have good trade-off in required additional disk space vs saved time.
drf_cache_suggest.R in repboxDRF.Try to determine data preparation steps that can be left out without changing the regression results. Trade-off: complexity and time to optimize. Yet, as this can be done in R purely and we can save intermediate data sets in memory that might be doable and worthwhile sometimes.
Our ideal setting for the relevant modification commands in path_df that will be translated to R via the stata2r package:
Currently these ideas are not fully implemented. Current state:
The functions in repboxDRF drf_deps.R already mark dependencies of different sorts and store in drf$dep_df:
xi commands in front of regressions that modify the data setif filter in a regression command.Handling so far:
scalars are already all stored in the original repboxStata reproduction and scalar definitions will be added to the R or Stata code when needed. This is handled fine by repboxDRF and needs no change.
if a regression command line is needed because of a xi: prefix in the regression that changes the data in a way needed by later regressions then stata2r probably already replaces the regression by the xi command. Don't know how it is detected currently: does stata2r get the info from repboxDRF or does it replicate the xi detection? Preferred solution: just use dependency info drf$dep_df from repboxDRF as generated in drf_deps.R. The xi dependencies don't violate save caching, as the xi: prefix directly changes the data set which would be respected by a cache that saves the data.
Dependencies via e() or r() variables are currently handled such that stata2r tries to replicate the generation of the e() and r() variables, e.g. if e(sample) is used later in a regression or data modification command, stata2r tries to generate code that creates the e(sample). However I am not convinced by that approach anymore. It has two drawbacks: i) stata2r is considerably more complex and ii) it violates "save caching": if we just create a cache at some runid, previously generated e() variables will not be used.
Proposed changes: Needed e() or r() values, according to drf_deps.R and only the needed ones, will be systematically stored in the mrb Stata run. This means like the caches, we can add code to mrb_stata.do that saves e() or r() values in a systematic fashion. Ideally generate a DRF subdirectory, stata_e_r and use the following filename convention. E.g. if e(sample) after runid 23 should be stored, call the file e_23.sample.dta (possibly a different file type depending on the type of info, like simple scalar numbers or strings values should rather be stored in a .txt file). To save the data from Stata, generate some helper .ado commands like, repbox_write_e_r_value.ado for saving a specific single value entry like e(rmse) and repbox_write_e_sample to save the whole vector that specifies the e_sample.
Now repboxDRF drf_get_data, shall ensure that if some e() or r() value is needed according to drf$dep_df then that value is loaded into a variable, named like e.sample for e(sample). That is similar to how needed scalar values are added to the code. The difference is that one looks at the existing files in the stata_e_r subfolder. If a dependency file is missing that should be noted with a repbox_problem.
Our package stata2r shall not create any e() or r() anymore itself but just expect that variables following the naming convention like e.sample for e(sample) exist in the execution environment. So we remove a lot of stuff from stata2r. Also the whole dependency check currently still in stata2r, which duplicates ideas from drf_deps.R shall be removed from stata2r.
keep if activeD==0|[activeD==1&consip==1]
xi: areg Lprice199 i.consip i.goodid*Lquantity i.year i.goodid*trend `controls' if activeD==0|[activeD==1&consip==1], a(id) robust
Here the [] are used like (). This somehow works with the Stata parser but is not documented. Usually [] are used to index elements. Trying to duplicate this parser quirk seems too complicated as we don't want to corrupt by accident correct [] use.
The regtranslate translation for regressions with iw, pw or fw weights will often yield different standard errors than Stata. Details are in weights_explained.md. We keep this issue as it is, not trying to repair.
NA handling: check where idiosyncratic NA handling in Stata changes results compared to proper NA handling (if proper can be well defined)
xtreg random effects vs fixed effects. xtreg without options uses random effects estimator. Are users aware of that or did they actually want to run a fixed effects estimator? Needs also AI to assess whether FE was intendet.