Skip to contents

Papercheck is designed modularly, so you can add modules to check for anything. It comes with a set of pre-defined modules, and we hope people will share more modules.

Module List

You can see the list of built-in modules with the function below.

  • all_p_values: List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.

Note that this will not catch p-values reported like “the p-value is 0.03” because that results in a ton of false positives when papers discuss p-value thresholds. If you need to detect text like that, use search_text() function and a custom pattern like “\bp(-| )?values?\s+.{1,20}\s+[0-9\.]+”

This will catch most comparators like =<>~≈≠≤≥≪≫ and most versions of scientific notation like 5.0 x 10^-2 or 5.0e-2. If you find any formats that are not correctly handled by this function, please contact the author. * all_urls: List all the URLs in the main text. * effect_size: Detect t-tests and F-tests with missing effect sizes * exact_p: List any p-values reported with insufficient precision (e.g., p < .05 or p = n.s.) * marginal: List all sentences that describe an effect as ‘marginally significant’. * osf_check: List all OSF links and whether they are open, closed, or do not exist. * ref_consistency: Check if all references are cited and all citations are referenced * retractionwatch: Flag any cited papers in the RetractionWatch database * statcheck: Check consistency of p-values and test statistics

Use module_help("module_name") for help with a specific module

Built-in Modules

Below, we will demonstrate the use of each built-in module, first on a single paper and then a list of papers, the psychsci list of 250 open-access papers from Psychological Science.

paper <- psychsci$`0956797620955209`

all_p_values

List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.

all_p <- module_run(paper, "all_p_values")

all_p$table # print table
#> # A tibble: 20 × 9
#>    text        section header           div     p     s id        p_comp p_value
#>    <chr>       <chr>   <chr>          <dbl> <dbl> <int> <chr>     <chr>    <dbl>
#>  1 "p = .003"  method  Questionnaires     6     2     3 09567976… =        0.003
#>  2 "p = .08"   method  Questionnaires     6     2     3 09567976… =        0.08 
#>  3 "p < .001 " results Results            9     2     1 09567976… <        0.001
#>  4 "p < .025"  results Results            9     2     3 09567976… <        0.025
#>  5 "p = .040"  results Results            9     2     3 09567976… =        0.04 
#>  6 "p = .173"  results Results            9     2     3 09567976… =        0.173
#>  7 "p = .006"  results Results            9     2     4 09567976… =        0.006
#>  8 "p = .02"   results Results            9     2     4 09567976… =        0.02 
#>  9 "p = .691"  results Results            9     2     5 09567976… =        0.691
#> 10 "p = .303"  results Results            9     2     5 09567976… =        0.303
#> 11 "p = .023"  results Results            9     3     3 09567976… =        0.023
#> 12 "p < .001"  results Results            9     3     3 09567976… <        0.001
#> 13 "p = .006"  results Results            9     4     2 09567976… =        0.006
#> 14 "p = .037"  results Results            9     4     2 09567976… =        0.037
#> 15 "p = .038"  results Results            9     4     2 09567976… =        0.038
#> 16 "p = .358"  results Results            9     4     2 09567976… =        0.358
#> 17 "p < .001"  results Results            9     4     3 09567976… <        0.001
#> 18 "p = .127"  results Results            9     4     3 09567976… =        0.127
#> 19 "p = .062"  results Results            9     4     3 09567976… =        0.062
#> 20 "p = .047"  results Results            9     4     3 09567976… =        0.047

If you run this module on all 250 papers, you will get more rows than you probably want to print in the full table one row for every p-value in each paper), so you can print the summary table, which gives you one row per paper.

all_p_ps <- module_run(psychsci, "all_p_values")

all_p_ps$summary
#>                    id p_values
#> 1    0956797613520608        6
#> 2    0956797614522816       39
#> 3    0956797614527830       13
#> 4    0956797614557697       27
#> 5    0956797614560771        4
#> 6    0956797614566469        0
#> 7    0956797615569001       25
#> 8    0956797615569889       26
#> 9    0956797615583071       24
#> 10   0956797615588467       21
#> 11   0956797615603702        8
#> 12   0956797615615584       26
#> 13   0956797615617779       34
#> 14   0956797615620784        9
#> 15   0956797615625973       11
#> 16   0956797616631990        8
#> 17   0956797616634654       22
#> 18   0956797616634665       18
#> 19   0956797616636631       26
#> 20   0956797616647519        8
#> 21   0956797616657319       17
#> 22   0956797616661199       13
#> 23   0956797616663878        5
#> 24   0956797616665351        8
#> 25   0956797616667447       17
#> 26   0956797616669994        2
#> 27   0956797616671327       37
#> 28   0956797616671712       25
#> 29   0956797617692000       39
#> 30   0956797617693326       37
#> 31   0956797617694867       51
#> 32   0956797617702501        0
#> 33   0956797617702699        3
#> 34   0956797617705391       16
#> 35   0956797617705667       31
#> 36   0956797617707270        8
#> 37   0956797617710785        8
#> 38   0956797617714811        1
#> 39   0956797617716922       14
#> 40   0956797617716929       24
#> 41   0956797617724435        1
#> 42   0956797617736886       12
#> 43   0956797617737129       37
#> 44   0956797617739368        6
#> 45   0956797617740685        9
#> 46   0956797617744542       19
#> 47   0956797618755322        5
#> 48   0956797618760197        3
#> 49   0956797618772822       49
#> 50   0956797618773095        1
#> 51   0956797618785899       26
#> 52   0956797618795679       17
#> 53   0956797618796480        7
#> 54   0956797618804501        0
#> 55   0956797618815482        4
#> 56   0956797618815488       28
#> 57   0956797618823540        0
#> 58   0956797619830326       17
#> 59   0956797619830329       32
#> 60   0956797619831964       21
#> 61   0956797619833325        0
#> 62   0956797619835147       41
#> 63   0956797619837981        5
#> 64   0956797619841265        7
#> 65   0956797619842261       58
#> 66   0956797619842550       39
#> 67   0956797619844231       37
#> 68   0956797619851753       21
#> 69   0956797619866625       25
#> 70   0956797619866627        3
#> 71   0956797619869905       24
#> 72   0956797619876260        9
#> 73   0956797619881134       48
#> 74   0956797619890619        0
#> 75   0956797620903716        9
#> 76   0956797620904450       14
#> 77   0956797620904990        0
#> 78   0956797620915887        1
#> 79   0956797620916521       28
#> 80   0956797620916782        4
#> 81   0956797620927648       34
#> 82   0956797620927967       11
#> 83   0956797620929297       16
#> 84   0956797620929302       21
#> 85   0956797620931108        8
#> 86   0956797620939054       21
#> 87   0956797620941840        0
#> 88   0956797620948821       17
#> 89   0956797620951115        0
#> 90   0956797620954815       39
#> 91   0956797620955209       20
#> 92   0956797620957625       62
#> 93   0956797620958638       95
#> 94   0956797620958650       38
#> 95   0956797620959014       18
#> 96   0956797620959594        4
#> 97   0956797620960011       20
#> 98   0956797620963615       25
#> 99   0956797620965520       19
#> 100  0956797620965536       32
#> 101  0956797620967261        7
#> 102  0956797620968789       15
#> 103  0956797620970548       32
#> 104  0956797620970559       10
#> 105  0956797620971298       41
#> 106  0956797620971652        0
#> 107  0956797620972116       61
#> 108  0956797620972688        6
#> 109  0956797620975781       29
#> 110  0956797620984464       52
#> 111  0956797620985832       27
#> 112 09567976211001317        0
#> 113 09567976211005465       33
#> 114 09567976211005767        4
#> 115 09567976211007414       10
#> 116 09567976211007788       18
#> 117 09567976211010718       15
#> 118 09567976211011969       33
#> 119 09567976211013045       11
#> 120 09567976211015941        1
#> 121 09567976211015942       32
#> 122 09567976211016395        6
#> 123 09567976211016410       11
#> 124 09567976211017870       17
#> 125 09567976211018618       22
#> 126 09567976211019950        0
#> 127 09567976211024259       12
#> 128 09567976211024260        2
#> 129 09567976211024535       23
#> 130 09567976211026983       40
#> 131 09567976211028978       10
#> 132 09567976211030630       44
#> 133 09567976211032224       17
#> 134 09567976211032676       15
#> 135 09567976211037971       21
#> 136 09567976211040491        8
#> 137 09567976211040803        5
#> 138 09567976211043426        0
#> 139 09567976211043428        6
#> 140 09567976211046884        0
#> 141 09567976211048485       17
#> 142 09567976211049439       13
#> 143 09567976211051272        2
#> 144 09567976211052476        2
#> 145 09567976211055375       24
#> 146 09567976211059801       32
#> 147 09567976211061321       39
#> 148 09567976211068045        0
#> 149 09567976211068070       27
#> 150 09567976211068880       41
#> 151  0956797621991137       32
#> 152  0956797621991548       14
#> 153  0956797621995197       47
#> 154  0956797621995202       33
#> 155  0956797621996660       23
#> 156  0956797621996667       46
#> 157  0956797621997350        6
#> 158  0956797621997366        2
#> 159  0956797621998312        0
#> 160 09567976221079633       18
#> 161 09567976221082637        2
#> 162 09567976221082938       31
#> 163 09567976221082941        6
#> 164 09567976221083219       27
#> 165 09567976221086513       66
#> 166 09567976221089599       15
#> 167 09567976221094036       29
#> 168 09567976221094782       48
#> 169 09567976221101045        0
#> 170 09567976221114055       39
#> 171 09567976221116816        1
#> 172 09567976221116892       34
#> 173 09567976221116893        1
#> 174 09567976221119391       14
#> 175 09567976221121348       55
#> 176 09567976221131519       15
#> 177 09567976221131520       30
#> 178 09567976221134476       19
#> 179 09567976221139496       16
#> 180 09567976221140326        0
#> 181 09567976221140341       24
#> 182 09567976221145316        5
#> 183 09567976221147258        8
#> 184 09567976221147259       29
#> 185 09567976221150616       33
#> 186 09567976231151581       27
#> 187 09567976231154804       24
#> 188 09567976231156413       61
#> 189 09567976231156793        4
#> 190 09567976231158288        6
#> 191 09567976231158570       24
#> 192 09567976231160098       30
#> 193 09567976231160702        5
#> 194 09567976231161565        1
#> 195 09567976231164553        0
#> 196 09567976231165267       32
#> 197 09567976231170878       16
#> 198 09567976231172500       29
#> 199 09567976231173900       57
#> 200 09567976231173902       14
#> 201 09567976231177968       27
#> 202 09567976231179378       31
#> 203 09567976231180578        8
#> 204 09567976231180588        9
#> 205 09567976231180881       16
#> 206 09567976231184887        5
#> 207 09567976231185127        8
#> 208 09567976231185129        1
#> 209 09567976231188107       23
#> 210 09567976231188124        0
#> 211 09567976231190546       31
#> 212 09567976231192241       28
#> 213 09567976231194221       14
#> 214 09567976231194590       25
#> 215 09567976231196145        7
#> 216 09567976231198194       21
#> 217 09567976231198435       30
#> 218 09567976231199440       13
#> 219 09567976231203139        0
#> 220 09567976231204035       47
#> 221 09567976231207095       27
#> 222 09567976231213572        3
#> 223 09567976231217508        0
#> 224 09567976231218640       35
#> 225 09567976231220902        8
#> 226 09567976231221789       31
#> 227 09567976231222288        0
#> 228 09567976231222836        1
#> 229 09567976231223130       70
#> 230 09567976231223410       18
#> 231 09567976241227411       45
#> 232 09567976241228504       23
#> 233 09567976241232891       15
#> 234 09567976241235931       28
#> 235 09567976241235932        0
#> 236 09567976241239932       12
#> 237 09567976241239935       17
#> 238 09567976241242105        3
#> 239 09567976241243370       11
#> 240 09567976241245695        0
#> 241 09567976241246561       39
#> 242 09567976241249183       21
#> 243 09567976241254312       20
#> 244 09567976241258149       20
#> 245 09567976241260247        8
#> 246 09567976241263344       25
#> 247 09567976241263347       53
#> 248 09567976241266516       37
#> 249 09567976241267854       46
#> 250 09567976241279291        3

You can still access the full table for further processing.

all_p_ps$table |>
  count(text, sort = TRUE) |>
  head()
#> # A tibble: 6 × 2
#>   text          n
#>   <chr>     <int>
#> 1 p < .001   1503
#> 2 p < .01     137
#> 3 p < .05     135
#> 4 p = .001    120
#> 5 p = .002     93
#> 6 p < .0001    88

all_urls

List all the URLs in the main text. There will, of course, be a few false positives when text in the paper is formatted as a valid URL.

all_urls <- module_run(paper, "all_urls")

all_urls$table
#> # A tibble: 5 × 7
#>   text                                    section header   div     p     s id   
#>   <chr>                                   <chr>   <chr>  <dbl> <dbl> <int> <chr>
#> 1 3.9.1.7                                 method  Parti…     3     1     5 0956…
#> 2 https://osf.io/k2dbf                    method  Analy…     8     1     1 0956…
#> 3 https://osf.io/k2dbf                    funding Open …    14     1     1 0956…
#> 4 https://osf.io/k2dbf                    funding Open …    14     2     1 0956…
#> 5 http://www.psychologicalscience.org/pu… funding Open …    14     2     3 0956…
all_urls_ps <- module_run(psychsci, "all_urls")

all_urls_ps$summary
#>                    id urls
#> 1    0956797613520608    0
#> 2    0956797614522816    0
#> 3    0956797614527830    1
#> 4    0956797614557697    6
#> 5    0956797614560771    0
#> 6    0956797614566469    5
#> 7    0956797615569001    7
#> 8    0956797615569889    2
#> 9    0956797615583071    4
#> 10   0956797615588467    2
#> 11   0956797615603702    0
#> 12   0956797615615584    2
#> 13   0956797615617779    1
#> 14   0956797615620784    4
#> 15   0956797615625973    4
#> 16   0956797616631990    6
#> 17   0956797616634654    2
#> 18   0956797616634665    1
#> 19   0956797616636631    5
#> 20   0956797616647519    7
#> 21   0956797616657319    3
#> 22   0956797616661199    5
#> 23   0956797616663878    4
#> 24   0956797616665351    5
#> 25   0956797616667447    1
#> 26   0956797616669994    1
#> 27   0956797616671327    2
#> 28   0956797616671712    1
#> 29   0956797617692000    6
#> 30   0956797617693326    1
#> 31   0956797617694867    8
#> 32   0956797617702501    6
#> 33   0956797617702699    4
#> 34   0956797617705391    3
#> 35   0956797617705667    5
#> 36   0956797617707270    2
#> 37   0956797617710785    4
#> 38   0956797617714811    1
#> 39   0956797617716922    3
#> 40   0956797617716929   10
#> 41   0956797617724435    9
#> 42   0956797617736886    1
#> 43   0956797617737129    9
#> 44   0956797617739368    9
#> 45   0956797617740685    3
#> 46   0956797617744542    3
#> 47   0956797618755322    5
#> 48   0956797618760197    4
#> 49   0956797618772822    5
#> 50   0956797618773095    1
#> 51   0956797618785899    8
#> 52   0956797618795679    3
#> 53   0956797618796480    5
#> 54   0956797618804501    1
#> 55   0956797618815482    0
#> 56   0956797618815488    3
#> 57   0956797618823540    2
#> 58   0956797619830326   17
#> 59   0956797619830329    9
#> 60   0956797619831964    5
#> 61   0956797619833325    2
#> 62   0956797619835147    8
#> 63   0956797619837981    1
#> 64   0956797619841265    8
#> 65   0956797619842261    6
#> 66   0956797619842550    5
#> 67   0956797619844231    7
#> 68   0956797619851753    3
#> 69   0956797619866625    6
#> 70   0956797619866627    8
#> 71   0956797619869905    5
#> 72   0956797619876260   11
#> 73   0956797619881134    7
#> 74   0956797619890619    7
#> 75   0956797620903716   21
#> 76   0956797620904450    2
#> 77   0956797620904990   17
#> 78   0956797620915887   20
#> 79   0956797620916521    2
#> 80   0956797620916782   12
#> 81   0956797620927648    4
#> 82   0956797620927967    8
#> 83   0956797620929297    1
#> 84   0956797620929302    6
#> 85   0956797620931108    4
#> 86   0956797620939054   14
#> 87   0956797620941840    6
#> 88   0956797620948821    8
#> 89   0956797620951115    6
#> 90   0956797620954815    0
#> 91   0956797620955209    5
#> 92   0956797620957625    8
#> 93   0956797620958638    2
#> 94   0956797620958650    3
#> 95   0956797620959014    2
#> 96   0956797620959594   16
#> 97   0956797620960011    5
#> 98   0956797620963615   10
#> 99   0956797620965520    2
#> 100  0956797620965536    5
#> 101  0956797620967261    5
#> 102  0956797620968789    2
#> 103  0956797620970548    3
#> 104  0956797620970559    3
#> 105  0956797620971298    7
#> 106  0956797620971652    4
#> 107  0956797620972116    4
#> 108  0956797620972688    2
#> 109  0956797620975781    5
#> 110  0956797620984464    5
#> 111  0956797620985832    9
#> 112 09567976211001317    6
#> 113 09567976211005465    3
#> 114 09567976211005767    7
#> 115 09567976211007414   16
#> 116 09567976211007788   14
#> 117 09567976211010718    9
#> 118 09567976211011969    9
#> 119 09567976211013045    0
#> 120 09567976211015941    5
#> 121 09567976211015942    4
#> 122 09567976211016395    1
#> 123 09567976211016410    8
#> 124 09567976211017870    9
#> 125 09567976211018618   16
#> 126 09567976211019950   11
#> 127 09567976211024259   17
#> 128 09567976211024260    4
#> 129 09567976211024535   10
#> 130 09567976211026983    7
#> 131 09567976211028978    3
#> 132 09567976211030630    4
#> 133 09567976211032224    6
#> 134 09567976211032676    4
#> 135 09567976211037971    8
#> 136 09567976211040491   14
#> 137 09567976211040803   12
#> 138 09567976211043426    6
#> 139 09567976211043428    4
#> 140 09567976211046884    3
#> 141 09567976211048485    1
#> 142 09567976211049439   16
#> 143 09567976211051272    8
#> 144 09567976211052476    9
#> 145 09567976211055375   13
#> 146 09567976211059801    5
#> 147 09567976211061321   14
#> 148 09567976211068045    2
#> 149 09567976211068070    1
#> 150 09567976211068880    3
#> 151  0956797621991137    2
#> 152  0956797621991548    6
#> 153  0956797621995197   10
#> 154  0956797621995202   16
#> 155  0956797621996660    7
#> 156  0956797621996667    9
#> 157  0956797621997350    4
#> 158  0956797621997366   12
#> 159  0956797621998312    5
#> 160 09567976221079633    1
#> 161 09567976221082637    7
#> 162 09567976221082938   11
#> 163 09567976221082941   11
#> 164 09567976221083219   10
#> 165 09567976221086513    4
#> 166 09567976221089599    7
#> 167 09567976221094036   12
#> 168 09567976221094782    9
#> 169 09567976221101045    8
#> 170 09567976221114055    9
#> 171 09567976221116816    2
#> 172 09567976221116892    6
#> 173 09567976221116893    5
#> 174 09567976221119391    6
#> 175 09567976221121348    5
#> 176 09567976221131519    4
#> 177 09567976221131520    9
#> 178 09567976221134476    8
#> 179 09567976221139496    1
#> 180 09567976221140326    1
#> 181 09567976221140341   16
#> 182 09567976221145316    4
#> 183 09567976221147258    2
#> 184 09567976221147259    6
#> 185 09567976221150616    5
#> 186 09567976231151581    1
#> 187 09567976231154804    3
#> 188 09567976231156413    6
#> 189 09567976231156793    1
#> 190 09567976231158288    7
#> 191 09567976231158570    4
#> 192 09567976231160098    2
#> 193 09567976231160702   10
#> 194 09567976231161565    4
#> 195 09567976231164553    2
#> 196 09567976231165267    1
#> 197 09567976231170878    2
#> 198 09567976231172500    4
#> 199 09567976231173900   15
#> 200 09567976231173902    2
#> 201 09567976231177968    2
#> 202 09567976231179378    1
#> 203 09567976231180578   10
#> 204 09567976231180588    2
#> 205 09567976231180881    5
#> 206 09567976231184887    8
#> 207 09567976231185127    3
#> 208 09567976231185129    6
#> 209 09567976231188107   11
#> 210 09567976231188124    0
#> 211 09567976231190546    7
#> 212 09567976231192241    1
#> 213 09567976231194221    0
#> 214 09567976231194590   18
#> 215 09567976231196145    5
#> 216 09567976231198194   11
#> 217 09567976231198435    3
#> 218 09567976231199440    3
#> 219 09567976231203139    1
#> 220 09567976231204035   12
#> 221 09567976231207095    7
#> 222 09567976231213572    9
#> 223 09567976231217508    1
#> 224 09567976231218640    2
#> 225 09567976231220902    6
#> 226 09567976231221789    1
#> 227 09567976231222288    4
#> 228 09567976231222836    4
#> 229 09567976231223130    6
#> 230 09567976231223410    9
#> 231 09567976241227411    6
#> 232 09567976241228504    4
#> 233 09567976241232891    3
#> 234 09567976241235931    4
#> 235 09567976241235932    0
#> 236 09567976241239932    3
#> 237 09567976241239935    9
#> 238 09567976241242105    5
#> 239 09567976241243370    4
#> 240 09567976241245695    8
#> 241 09567976241246561    2
#> 242 09567976241249183    5
#> 243 09567976241254312    3
#> 244 09567976241258149    9
#> 245 09567976241260247    4
#> 246 09567976241263344    8
#> 247 09567976241263347    7
#> 248 09567976241266516    4
#> 249 09567976241267854    5
#> 250 09567976241279291   11

exact_p

List any p-values that may have been reported with insufficient precision (e.g., p < .05 or p = n.s.).

imprecise <- module_run(paper, "exact_p")

imprecise$table # print table
#> # A tibble: 1 × 9
#>   text     p_comp p_value section header    div     p     s id              
#>   <chr>    <chr>    <dbl> <chr>   <chr>   <dbl> <dbl> <int> <chr>           
#> 1 p < .025 <        0.025 results Results     9     2     3 0956797620955209

You can use the expand_text() function to get the full sentence or paragraph for context. Here you can see that “p < .025” was not an imprecisely reported p-value, but a description of the preregistered alpha threshold.

exp <- expand_text(imprecise$table, paper)

exp$expanded # print expanded text
#> [1] "The main effect of illness recency did not meet our preregistered threshold (p < .025)-recently ill: M = 661 ms, SD = 197; not recently ill: M = 626 ms, SD = 153, F(1, 400) = 4.23, η p 2 = .010, 90% CI = [.000, .039], p = .040-nor did the interaction between illness recency and face type (disfigured vs. typical), F(1, 400) = 1.87, η p 2 = .005, 90% CI = [.000, .027], p = .173."

We can investigate the most common imprecise p-values in the PsychSci set. “p < .01” and “p < .05” are probably often describing figures or tables, but what is the deal with “p > .25”?

imprecise_ps <- module_run(psychsci, "exact_p")

imprecise_ps$table |>
  count(text, sort = TRUE) |>
  head()
#> # A tibble: 6 × 2
#>   text           n
#>   <chr>      <int>
#> 1 "p < .01"    137
#> 2 "p < .05"    135
#> 3 "p < .05 "    35
#> 4 "p > .250"    17
#> 5 "p < .005"    16
#> 6 "p < .01 "    15

We can expand the text to check the context for “p > .25”.

gt.25 <- imprecise_ps$table |>
  filter(grepl("\\.25", text)) |>
  expand_text(paper = psychsci)

gt.25$expanded[1:3] # look at the first 3
#> [1] "There was a significant interactive effect of time and political orientation, b = -0.09, SE = 0.04, 95% CI = [-0.16, -0.02], t(1922) = -2.49, p = .013, on the endorsement of the in-group foundation (see Table S2 in CI = [-0.14, -0.04], t(1922) = -3.59, p < .001, disappeared after 7/7, b = 0.004, SE = 0.02, 95% CI = [-0.04, 0.05], t(1922) = 0.17, p > .250 (see Fig."                    
#> [2] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."
#> [3] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."

marginal

List all sentences that describe an effect as ‘marginally significant’.

marginal <- module_run(paper, "marginal")

marginal # print table

No effects were described as marginally/borderline/close to significant.

Let’s check how many are in the full set.

marginal_ps <- module_run(psychsci, "marginal")

marginal_ps$table # print table
#> # A tibble: 99 × 7
#>    text                                   section header   div     p     s id   
#>    <chr>                                  <chr>   <chr>  <dbl> <dbl> <int> <chr>
#>  1 Although the PTSD group showed a sign… results Detai…    11     1     4 0956…
#>  2 A marginally significant negative cor… results Postt…    13     1     4 0956…
#>  3 When we more closely matched depressi… results The e…    14     3     1 0956…
#>  4 In that analysis, the group differenc… results Condi…    15     1     2 0956…
#>  5 The Congruency × Alignment interactio… results Resul…     7     2     3 0956…
#>  6 The twoway interactions between knowl… results Resul…     6     2     3 0956…
#>  7 An omnibus 3 (sex ratio) × 2 (partici… results Resul…    15     2     1 0956…
#>  8 Further, we observed a marginally sig… fig     NA         3    NA     2 0956…
#>  9 The dagger and asterisks indicate mar… fig     Fig. …     5    NA     7 0956…
#> 10 Given the unexpected nature of the fi… annex   Open …    20     1    21 0956…
#> # ℹ 89 more rows

osf_check

List all OSF links and whether they are open, closed, or do not exist. This requires an internet connection to check the status of the link.

osf_check <- module_run(paper, "osf_check")
#> Starting OSF retrieval for 1 files...
#> * Retrieving info from k2dbf...
#> ...OSF retrieval complete!
osf_check$summary
#>                 id osf.open
#> 1 0956797620955209        3

The returned table contains additional information about the links if they are found and open.

osf_check$table[1,] |> t()
#>              [,1]                                                                                                                          
#> text         "osf.io/k2dbf"                                                                                                                
#> section      "method"                                                                                                                      
#> header       "Analyses"                                                                                                                    
#> div          "8"                                                                                                                           
#> p            "1"                                                                                                                           
#> s            "1"                                                                                                                           
#> id           "0956797620955209"                                                                                                            
#> osf_id       "k2dbf"                                                                                                                       
#> name         "Preregistered replication of “Sick body, vigilant mind: The biological immune system activates the behavioral immune system”"
#> description  ""                                                                                                                            
#> osf_type     "nodes"                                                                                                                       
#> public       "TRUE"                                                                                                                        
#> category     "project"                                                                                                                     
#> registration "FALSE"                                                                                                                       
#> preprint     "TRUE"                                                                                                                        
#> parent       NA                                                                                                                            
#> status       "open"

You can also run this module on a batch of papers. You can only make 100 API requests per hour, unless you authorise your requests, when you can make 10K requests per day. The OSF functions in papercheck often make several requests per URL to get all of the info, so it’s worthwhile setting your PAT. You can authorise them by creating an OSF token at https://osf.io/settings/tokens and including the following line in your .Renviron file (which you can open using usethis::edit_r_environ()):

OSF_PAT="replace-with-your-token-string"

This would take a while to run for the 333 unique OSF links found in the PsychSci set (usually a few links per second), so we will skip it here (but can reveal we found 10 closed links).

osf_check_ps <- module_run(psychsci, "osf_check")

ref_consistency

Check if all references are cited and all citations are referenced.

ref_consistency <- module_run(paper, "ref_consistency")

ref_consistency$table
#>   bib_id
#> 1    b16
#> 2    b17
#> 3     b3
#> 4    b31
#>                                                                                                                                                                                                                                                                               ref
#> 1                                                       Jones BC, Hahn AC, Fisher CI, Wang H, Kandrik M, Tybur JM, Debruine LM (2018). “Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis.” _Evolution and Human Behavior_, *39*, 166-169.
#> 2 Jones BC, Hahn AC, Fisher CI, Wang H, Kandrik M, Tybur JM, Debruine LM (2018b). “Reply to Fleischman and Fessler's (2018) comment on "Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis.” _Evolution and Human Behavior_, *39*, 470-471.
#> 3                                                                                                                                                           Curtis VA (2014). “Infection-avoidance behaviour in humans and other animals.” _Trends in Immunology_, *35*, 457-464.
#> 4       Tybur JM, Inbar Y, Aarøe L, Barclay P, Barlow FK, De Barra M, Žeželj .., I (2016). “Parasite stress and pathogen avoidance relate to distinct dimensions of political ideology across 30 nations.” _Proceedings of the National Academy of Sciences_, *113*, 12408-12413.
#>    doi bibtype
#> 1 <NA> Article
#> 2 <NA> Article
#> 3 <NA> Article
#> 4 <NA> Article
#>                                                                                                                                           title
#> 1                                                      Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis
#> 2 Reply to Fleischman and Fessler's (2018) comment on "Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis
#> 3                                                                                     Infection-avoidance behaviour in humans and other animals
#> 4                                  Parasite stress and pathogen avoidance relate to distinct dimensions of political ideology across 30 nations
#>                                           journal  year
#> 1                    Evolution and Human Behavior  2018
#> 2                    Evolution and Human Behavior 2018b
#> 3                            Trends in Immunology  2014
#> 4 Proceedings of the National Academy of Sciences  2016
#>                                                                         authors
#> 1   B C Jones, A C Hahn, C I Fisher, H Wang, M Kandrik, J M Tybur, L M Debruine
#> 2   B C Jones, A C Hahn, C I Fisher, H Wang, M Kandrik, J M Tybur, L M Debruine
#> 3                                                                    V A Curtis
#> 4 J M Tybur, Y Inbar, L Aarøe, P Barclay, F K Barlow, M De Barra, . . Žeželj, I
#>                 id  missing
#> 1 0956797620955209 citation
#> 2 0956797620955209 citation
#> 3 0956797620955209 citation
#> 4 0956797620955209 citation

It looks like there are some references with missing citations. The first one doesn’t look like a reference, and grobid often parses tables oddly. You’d need to look at the original PDF to see if the others are actually missing or false positives. Here, they are all false positives, based on grobid not being able to match the in-text citation to the reference list.

retractionwatch

Flag any cited papers in the RetractionWatch database. It takes a substantial time to make calls to crossref for each reference, so we provide a summary of the database in papercheck, which is updated regularly (last 2025-02-28).

rw <- module_run(paper, "retractionwatch")

rw$table
#>  [1] bib_id          ref             doi             bibtype        
#>  [5] title           journal         year            authors        
#>  [9] id              retractionwatch
#> <0 rows> (or 0-length row.names)

In the full PsychSci set, we find three papers in the RetractionWatch database cited, one with a retraction and two with corrections.

rw_ps <- module_run(psychsci, "retractionwatch")

rw_ps$table
#>   bib_id
#> 1    b15
#> 2    b25
#> 3    b36
#> 4    b25
#> 5    b25
#> 6    b25
#>                                                                                                                                                                                                                                                                                                                                                                                            ref
#> 1 Kok BE, Coffey KA, Cohn MA, Catalino LI, Vacharkulksemsuk T, Algoe SB, Brantley M, Fredrickson BL (2013). “How positive emotions build physical health: Perceived positive social connections account for the upward spiral between positive emotions and vagal tone.” _Psychological Science_, *24*(7), 1123-1132. doi:10.1177/0956797612470827 <https://doi.org/10.1177/0956797612470827>.
#> 2                                                                                                                                                                                                      Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*(10), R115. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 3                                                                                                                                                                                                                                                                                                         (????). doi:10.1038/s41562-023-01749-9 <https://doi.org/10.1038/s41562-023-01749-9>.
#> 4                                                                                                                                                                                                          Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 5                                                                                                                                                                                                          Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 6                                                                                                                                                                                                          Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#>                          doi bibtype
#> 1   10.1177/0956797612470827 Article
#> 2 10.1186/gb-2013-14-10-r115 Article
#> 3 10.1038/s41562-023-01749-9    Misc
#> 4 10.1186/gb-2013-14-10-r115 Article
#> 5 10.1186/gb-2013-14-10-r115 Article
#> 6 10.1186/gb-2013-14-10-r115 Article
#>                                                                                                                                                       title
#> 1 How positive emotions build physical health: Perceived positive social connections account for the upward spiral between positive emotions and vagal tone
#> 2                                                                                                       DNA methylation age of human tissues and cell types
#> 3                                                                                                                                                      <NA>
#> 4                                                                                                       DNA methylation age of human tissues and cell types
#> 5                                                                                                       DNA methylation age of human tissues and cell types
#> 6                                                                                                       DNA methylation age of human tissues and cell types
#>                 journal year
#> 1 Psychological Science 2013
#> 2        Genome Biology 2013
#> 3             Behaviour <NA>
#> 4        Genome Biology 2013
#> 5        Genome Biology 2013
#> 6        Genome Biology 2013
#>                                                                                                   authors
#> 1 B E Kok, K A Coffey, M A Cohn, L I Catalino, T Vacharkulksemsuk, S B Algoe, M Brantley, B L Fredrickson
#> 2                                                                                               S Horvath
#> 3                                                                                                      NA
#> 4                                                                                               S Horvath
#> 5                                                                                               S Horvath
#> 6                                                                                               S Horvath
#>                  id retractionwatch
#> 1 09567976221150616      Correction
#> 2 09567976231194221      Correction
#> 3 09567976241249183      Retraction
#> 4 09567976241260247      Correction
#> 5 09567976241260247      Correction
#> 6 09567976241260247      Correction
#>                                                                                                                                                                                                                                                   text
#> 1 The vagus nerve controls respiratory sinus arrhythmia (RSA), the covariation of heart rate and respiration, known to be an index of cardiac health and predictor of enhanced capacity for emotion regulation (Butler et al., 2006;Kok et al., 2013).
#> 2                                                                                                                                                              Epigenetic aging is one such indicator of accelerated biological aging (Horvath, 2013).
#> 3                                                                                                                                                                                                                                                 <NA>
#> 4                                         Discrepancies between someone's biological and chronological age translate into "age acceleration," with older biological age relative to chronological age indicating greater acceleration (Horvath, 2013).
#> 5                                                                                                                        EAA from the Horvath PanTissue clock (Horvath, 2013) was calculated using the methylclock package (Pelegí-Sisó et al., 2021).
#> 6                                                            The replication of results across the PedBE and Horvath PanTissue clocks increases our confidence in study findings because these tools share no overlapping DNAm sites (Horvath, 2013;L.

Check the context of the citations in the text column:

rw_ps$table$text
#> [1] "The vagus nerve controls respiratory sinus arrhythmia (RSA), the covariation of heart rate and respiration, known to be an index of cardiac health and predictor of enhanced capacity for emotion regulation (Butler et al., 2006;Kok et al., 2013)."
#> [2] "Epigenetic aging is one such indicator of accelerated biological aging (Horvath, 2013)."                                                                                                                                                             
#> [3] NA                                                                                                                                                                                                                                                    
#> [4] "Discrepancies between someone's biological and chronological age translate into \"age acceleration,\" with older biological age relative to chronological age indicating greater acceleration (Horvath, 2013)."                                      
#> [5] "EAA from the Horvath PanTissue clock (Horvath, 2013) was calculated using the methylclock package (Pelegí-Sisó et al., 2021)."                                                                                                                       
#> [6] "The replication of results across the PedBE and Horvath PanTissue clocks increases our confidence in study findings because these tools share no overlapping DNAm sites (Horvath, 2013;L."

statcheck

Check consistency of p-values and test statistics using functions from statcheck.

statcheck <- module_run(paper, "statcheck")

statcheck$table
#>   test_type df1   df2 test_comp test_value p_comp reported_p computed_p
#> 1         t  NA 248.4         =       2.01      =      0.023 0.04551244
#>                         raw error decision_error one_tailed_in_txt apa_factor
#> 1 t(248.4) = 2.01, p = .023  TRUE          FALSE             FALSE          1
#>                                                                                                                                                                                                                                                                                                text
#> 1 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (d z = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (d z ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p < .001.
#>   section  header div p s               id
#> 1 results Results   9 3 3 0956797620955209

Here we see a false positive, where the paper reported the results of an equivalence test, which are meant to be one-tailed, but statcheck did not detect that this was one-tailed.

In the full PsychSci set, there are more than 27K sentences with numbers to check, so this takes about a minute to run.

statcheck_ps <- module_run(psychsci, "statcheck")

There will be, of course, some false positives in the full set of 151 flagged values. Let’s look just at the flagged values where the computed p-value is about double the reported p-value, and this changes the significance decision (at an alpha of 0.05).

statcheck_ps$table |>
  filter(decision_error, 
         round(computed_p/reported_p, 1) == 2.0) |>
  select(reported_p, computed_p, raw) |>
  mutate(computed_p = round(computed_p, 4))
#>   reported_p computed_p                        raw
#> 1     0.0290     0.0589 F(1, 361) = 3.59, p = .029
#> 2     0.0470     0.0947     t(24) = 1.74, p = .047
#> 3     0.0270     0.0547     t(24) = 2.02, p = .027
#> 4     0.0400     0.0797     t(24) = 1.83, p = .040
#> 5     0.0480     0.0962    t(240) = 1.67, p = .048
#> 6     0.0460     0.0915     t(32) = 1.74, p = .046
#> 7     0.0420     0.0846     t(21) = 1.81, p = .042
#> 8     0.0343     0.0686    t(10) = 2.04, p = .0343
#> 9     0.0330     0.0654     t(55) = 1.88, p = .033

Chaining Modules

Modules return a summary table as well as the detailed results table, which is automatically added to the summary if you chain modules.

ps_metascience <- psychsci[1:10] |>
  module_run("all_p_values") |>
  module_run("exact_p") |>
  module_run("marginal")

ps_metascience$summary
#>                  id p_values exact_p imprecise_p marginal
#> 1  0956797613520608        6       6           0        0
#> 2  0956797614522816       39      39           0        0
#> 3  0956797614527830       13      10           3        0
#> 4  0956797614557697       27      16          11        0
#> 5  0956797614560771        4       3           1        0
#> 6  0956797614566469        0       0           0        0
#> 7  0956797615569001       25       2          23        0
#> 8  0956797615569889       26      24           2        4
#> 9  0956797615583071       24      21           3        0
#> 10 0956797615588467       21      13           8        0