Papercheck is designed modularly, so you can add modules to check for anything. It comes with a set of pre-defined modules, and we hope people will share more modules.
Module List
You can see the list of built-in modules with the function below.
- all_p_values: List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.
Note that this will not catch p-values reported like “the p-value is
0.03” because that results in a ton of false positives when papers
discuss p-value thresholds. If you need to detect text like that, use
search_text() function and a custom pattern like “\bp(-|
)?values?\s+.{1,20}\s+[0-9\.]+”
This will catch most comparators like =<>~≈≠≤≥≪≫ and most versions of scientific notation like 5.0 x 10^-2 or 5.0e-2. If you find any formats that are not correctly handled by this function, please contact the author. * all_urls: List all the URLs in the main text. * effect_size: Detect t-tests and F-tests with missing effect sizes * exact_p: List any p-values reported with insufficient precision (e.g., p < .05 or p = n.s.) * marginal: List all sentences that describe an effect as ‘marginally significant’. * osf_check: List all OSF links and whether they are open, closed, or do not exist. * ref_consistency: Check if all references are cited and all citations are referenced * retractionwatch: Flag any cited papers in the RetractionWatch database * statcheck: Check consistency of p-values and test statistics
Use module_help("module_name") for help with a specific
module
Built-in Modules
Below, we will demonstrate the use of each built-in module, first on
a single paper and then a list of papers, the psychsci list
of 250 open-access papers from Psychological Science.
paper <- psychsci$`0956797620955209`all_p_values
List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.
all_p <- module_run(paper, "all_p_values")
all_p$table # print table#> # A tibble: 20 × 9
#> text section header div p s id p_comp p_value
#> <chr> <chr> <chr> <dbl> <dbl> <int> <chr> <chr> <dbl>
#> 1 "p = .003" method Questionnaires 6 2 3 09567976… = 0.003
#> 2 "p = .08" method Questionnaires 6 2 3 09567976… = 0.08
#> 3 "p < .001 " results Results 9 2 1 09567976… < 0.001
#> 4 "p < .025" results Results 9 2 3 09567976… < 0.025
#> 5 "p = .040" results Results 9 2 3 09567976… = 0.04
#> 6 "p = .173" results Results 9 2 3 09567976… = 0.173
#> 7 "p = .006" results Results 9 2 4 09567976… = 0.006
#> 8 "p = .02" results Results 9 2 4 09567976… = 0.02
#> 9 "p = .691" results Results 9 2 5 09567976… = 0.691
#> 10 "p = .303" results Results 9 2 5 09567976… = 0.303
#> 11 "p = .023" results Results 9 3 3 09567976… = 0.023
#> 12 "p < .001" results Results 9 3 3 09567976… < 0.001
#> 13 "p = .006" results Results 9 4 2 09567976… = 0.006
#> 14 "p = .037" results Results 9 4 2 09567976… = 0.037
#> 15 "p = .038" results Results 9 4 2 09567976… = 0.038
#> 16 "p = .358" results Results 9 4 2 09567976… = 0.358
#> 17 "p < .001" results Results 9 4 3 09567976… < 0.001
#> 18 "p = .127" results Results 9 4 3 09567976… = 0.127
#> 19 "p = .062" results Results 9 4 3 09567976… = 0.062
#> 20 "p = .047" results Results 9 4 3 09567976… = 0.047If you run this module on all 250 papers, you will get more rows than you probably want to print in the full table one row for every p-value in each paper), so you can print the summary table, which gives you one row per paper.
all_p_ps <- module_run(psychsci, "all_p_values")
all_p_ps$summary#> id p_values
#> 1 0956797613520608 6
#> 2 0956797614522816 39
#> 3 0956797614527830 13
#> 4 0956797614557697 27
#> 5 0956797614560771 4
#> 6 0956797614566469 0
#> 7 0956797615569001 25
#> 8 0956797615569889 26
#> 9 0956797615583071 24
#> 10 0956797615588467 21
#> 11 0956797615603702 8
#> 12 0956797615615584 26
#> 13 0956797615617779 34
#> 14 0956797615620784 9
#> 15 0956797615625973 11
#> 16 0956797616631990 8
#> 17 0956797616634654 22
#> 18 0956797616634665 18
#> 19 0956797616636631 26
#> 20 0956797616647519 8
#> 21 0956797616657319 17
#> 22 0956797616661199 13
#> 23 0956797616663878 5
#> 24 0956797616665351 8
#> 25 0956797616667447 17
#> 26 0956797616669994 2
#> 27 0956797616671327 37
#> 28 0956797616671712 25
#> 29 0956797617692000 39
#> 30 0956797617693326 37
#> 31 0956797617694867 51
#> 32 0956797617702501 0
#> 33 0956797617702699 3
#> 34 0956797617705391 16
#> 35 0956797617705667 31
#> 36 0956797617707270 8
#> 37 0956797617710785 8
#> 38 0956797617714811 1
#> 39 0956797617716922 14
#> 40 0956797617716929 24
#> 41 0956797617724435 1
#> 42 0956797617736886 12
#> 43 0956797617737129 37
#> 44 0956797617739368 6
#> 45 0956797617740685 9
#> 46 0956797617744542 19
#> 47 0956797618755322 5
#> 48 0956797618760197 3
#> 49 0956797618772822 49
#> 50 0956797618773095 1
#> 51 0956797618785899 26
#> 52 0956797618795679 17
#> 53 0956797618796480 7
#> 54 0956797618804501 0
#> 55 0956797618815482 4
#> 56 0956797618815488 28
#> 57 0956797618823540 0
#> 58 0956797619830326 17
#> 59 0956797619830329 32
#> 60 0956797619831964 21
#> 61 0956797619833325 0
#> 62 0956797619835147 41
#> 63 0956797619837981 5
#> 64 0956797619841265 7
#> 65 0956797619842261 58
#> 66 0956797619842550 39
#> 67 0956797619844231 37
#> 68 0956797619851753 21
#> 69 0956797619866625 25
#> 70 0956797619866627 3
#> 71 0956797619869905 24
#> 72 0956797619876260 9
#> 73 0956797619881134 48
#> 74 0956797619890619 0
#> 75 0956797620903716 9
#> 76 0956797620904450 14
#> 77 0956797620904990 0
#> 78 0956797620915887 1
#> 79 0956797620916521 28
#> 80 0956797620916782 4
#> 81 0956797620927648 34
#> 82 0956797620927967 11
#> 83 0956797620929297 16
#> 84 0956797620929302 21
#> 85 0956797620931108 8
#> 86 0956797620939054 21
#> 87 0956797620941840 0
#> 88 0956797620948821 17
#> 89 0956797620951115 0
#> 90 0956797620954815 39
#> 91 0956797620955209 20
#> 92 0956797620957625 62
#> 93 0956797620958638 95
#> 94 0956797620958650 38
#> 95 0956797620959014 18
#> 96 0956797620959594 4
#> 97 0956797620960011 20
#> 98 0956797620963615 25
#> 99 0956797620965520 19
#> 100 0956797620965536 32
#> 101 0956797620967261 7
#> 102 0956797620968789 15
#> 103 0956797620970548 32
#> 104 0956797620970559 10
#> 105 0956797620971298 41
#> 106 0956797620971652 0
#> 107 0956797620972116 61
#> 108 0956797620972688 6
#> 109 0956797620975781 29
#> 110 0956797620984464 52
#> 111 0956797620985832 27
#> 112 09567976211001317 0
#> 113 09567976211005465 33
#> 114 09567976211005767 4
#> 115 09567976211007414 10
#> 116 09567976211007788 18
#> 117 09567976211010718 15
#> 118 09567976211011969 33
#> 119 09567976211013045 11
#> 120 09567976211015941 1
#> 121 09567976211015942 32
#> 122 09567976211016395 6
#> 123 09567976211016410 11
#> 124 09567976211017870 17
#> 125 09567976211018618 22
#> 126 09567976211019950 0
#> 127 09567976211024259 12
#> 128 09567976211024260 2
#> 129 09567976211024535 23
#> 130 09567976211026983 40
#> 131 09567976211028978 10
#> 132 09567976211030630 44
#> 133 09567976211032224 17
#> 134 09567976211032676 15
#> 135 09567976211037971 21
#> 136 09567976211040491 8
#> 137 09567976211040803 5
#> 138 09567976211043426 0
#> 139 09567976211043428 6
#> 140 09567976211046884 0
#> 141 09567976211048485 17
#> 142 09567976211049439 13
#> 143 09567976211051272 2
#> 144 09567976211052476 2
#> 145 09567976211055375 24
#> 146 09567976211059801 32
#> 147 09567976211061321 39
#> 148 09567976211068045 0
#> 149 09567976211068070 27
#> 150 09567976211068880 41
#> 151 0956797621991137 32
#> 152 0956797621991548 14
#> 153 0956797621995197 47
#> 154 0956797621995202 33
#> 155 0956797621996660 23
#> 156 0956797621996667 46
#> 157 0956797621997350 6
#> 158 0956797621997366 2
#> 159 0956797621998312 0
#> 160 09567976221079633 18
#> 161 09567976221082637 2
#> 162 09567976221082938 31
#> 163 09567976221082941 6
#> 164 09567976221083219 27
#> 165 09567976221086513 66
#> 166 09567976221089599 15
#> 167 09567976221094036 29
#> 168 09567976221094782 48
#> 169 09567976221101045 0
#> 170 09567976221114055 39
#> 171 09567976221116816 1
#> 172 09567976221116892 34
#> 173 09567976221116893 1
#> 174 09567976221119391 14
#> 175 09567976221121348 55
#> 176 09567976221131519 15
#> 177 09567976221131520 30
#> 178 09567976221134476 19
#> 179 09567976221139496 16
#> 180 09567976221140326 0
#> 181 09567976221140341 24
#> 182 09567976221145316 5
#> 183 09567976221147258 8
#> 184 09567976221147259 29
#> 185 09567976221150616 33
#> 186 09567976231151581 27
#> 187 09567976231154804 24
#> 188 09567976231156413 61
#> 189 09567976231156793 4
#> 190 09567976231158288 6
#> 191 09567976231158570 24
#> 192 09567976231160098 30
#> 193 09567976231160702 5
#> 194 09567976231161565 1
#> 195 09567976231164553 0
#> 196 09567976231165267 32
#> 197 09567976231170878 16
#> 198 09567976231172500 29
#> 199 09567976231173900 57
#> 200 09567976231173902 14
#> 201 09567976231177968 27
#> 202 09567976231179378 31
#> 203 09567976231180578 8
#> 204 09567976231180588 9
#> 205 09567976231180881 16
#> 206 09567976231184887 5
#> 207 09567976231185127 8
#> 208 09567976231185129 1
#> 209 09567976231188107 23
#> 210 09567976231188124 0
#> 211 09567976231190546 31
#> 212 09567976231192241 28
#> 213 09567976231194221 14
#> 214 09567976231194590 25
#> 215 09567976231196145 7
#> 216 09567976231198194 21
#> 217 09567976231198435 30
#> 218 09567976231199440 13
#> 219 09567976231203139 0
#> 220 09567976231204035 47
#> 221 09567976231207095 27
#> 222 09567976231213572 3
#> 223 09567976231217508 0
#> 224 09567976231218640 35
#> 225 09567976231220902 8
#> 226 09567976231221789 31
#> 227 09567976231222288 0
#> 228 09567976231222836 1
#> 229 09567976231223130 70
#> 230 09567976231223410 18
#> 231 09567976241227411 45
#> 232 09567976241228504 23
#> 233 09567976241232891 15
#> 234 09567976241235931 28
#> 235 09567976241235932 0
#> 236 09567976241239932 12
#> 237 09567976241239935 17
#> 238 09567976241242105 3
#> 239 09567976241243370 11
#> 240 09567976241245695 0
#> 241 09567976241246561 39
#> 242 09567976241249183 21
#> 243 09567976241254312 20
#> 244 09567976241258149 20
#> 245 09567976241260247 8
#> 246 09567976241263344 25
#> 247 09567976241263347 53
#> 248 09567976241266516 37
#> 249 09567976241267854 46
#> 250 09567976241279291 3You can still access the full table for further processing.
#> # A tibble: 6 × 2
#> text n
#> <chr> <int>
#> 1 p < .001 1503
#> 2 p < .01 137
#> 3 p < .05 135
#> 4 p = .001 120
#> 5 p = .002 93
#> 6 p < .0001 88all_urls
List all the URLs in the main text. There will, of course, be a few false positives when text in the paper is formatted as a valid URL.
all_urls <- module_run(paper, "all_urls")
all_urls$table#> # A tibble: 5 × 7
#> text section header div p s id
#> <chr> <chr> <chr> <dbl> <dbl> <int> <chr>
#> 1 3.9.1.7 method Parti… 3 1 5 0956…
#> 2 https://osf.io/k2dbf method Analy… 8 1 1 0956…
#> 3 https://osf.io/k2dbf funding Open … 14 1 1 0956…
#> 4 https://osf.io/k2dbf funding Open … 14 2 1 0956…
#> 5 http://www.psychologicalscience.org/pu… funding Open … 14 2 3 0956…
all_urls_ps <- module_run(psychsci, "all_urls")
all_urls_ps$summary#> id urls
#> 1 0956797613520608 0
#> 2 0956797614522816 0
#> 3 0956797614527830 1
#> 4 0956797614557697 6
#> 5 0956797614560771 0
#> 6 0956797614566469 5
#> 7 0956797615569001 7
#> 8 0956797615569889 2
#> 9 0956797615583071 4
#> 10 0956797615588467 2
#> 11 0956797615603702 0
#> 12 0956797615615584 2
#> 13 0956797615617779 1
#> 14 0956797615620784 4
#> 15 0956797615625973 4
#> 16 0956797616631990 6
#> 17 0956797616634654 2
#> 18 0956797616634665 1
#> 19 0956797616636631 5
#> 20 0956797616647519 7
#> 21 0956797616657319 3
#> 22 0956797616661199 5
#> 23 0956797616663878 4
#> 24 0956797616665351 5
#> 25 0956797616667447 1
#> 26 0956797616669994 1
#> 27 0956797616671327 2
#> 28 0956797616671712 1
#> 29 0956797617692000 6
#> 30 0956797617693326 1
#> 31 0956797617694867 8
#> 32 0956797617702501 6
#> 33 0956797617702699 4
#> 34 0956797617705391 3
#> 35 0956797617705667 5
#> 36 0956797617707270 2
#> 37 0956797617710785 4
#> 38 0956797617714811 1
#> 39 0956797617716922 3
#> 40 0956797617716929 10
#> 41 0956797617724435 9
#> 42 0956797617736886 1
#> 43 0956797617737129 9
#> 44 0956797617739368 9
#> 45 0956797617740685 3
#> 46 0956797617744542 3
#> 47 0956797618755322 5
#> 48 0956797618760197 4
#> 49 0956797618772822 5
#> 50 0956797618773095 1
#> 51 0956797618785899 8
#> 52 0956797618795679 3
#> 53 0956797618796480 5
#> 54 0956797618804501 1
#> 55 0956797618815482 0
#> 56 0956797618815488 3
#> 57 0956797618823540 2
#> 58 0956797619830326 17
#> 59 0956797619830329 9
#> 60 0956797619831964 5
#> 61 0956797619833325 2
#> 62 0956797619835147 8
#> 63 0956797619837981 1
#> 64 0956797619841265 8
#> 65 0956797619842261 6
#> 66 0956797619842550 5
#> 67 0956797619844231 7
#> 68 0956797619851753 3
#> 69 0956797619866625 6
#> 70 0956797619866627 8
#> 71 0956797619869905 5
#> 72 0956797619876260 11
#> 73 0956797619881134 7
#> 74 0956797619890619 7
#> 75 0956797620903716 21
#> 76 0956797620904450 2
#> 77 0956797620904990 17
#> 78 0956797620915887 20
#> 79 0956797620916521 2
#> 80 0956797620916782 12
#> 81 0956797620927648 4
#> 82 0956797620927967 8
#> 83 0956797620929297 1
#> 84 0956797620929302 6
#> 85 0956797620931108 4
#> 86 0956797620939054 14
#> 87 0956797620941840 6
#> 88 0956797620948821 8
#> 89 0956797620951115 6
#> 90 0956797620954815 0
#> 91 0956797620955209 5
#> 92 0956797620957625 8
#> 93 0956797620958638 2
#> 94 0956797620958650 3
#> 95 0956797620959014 2
#> 96 0956797620959594 16
#> 97 0956797620960011 5
#> 98 0956797620963615 10
#> 99 0956797620965520 2
#> 100 0956797620965536 5
#> 101 0956797620967261 5
#> 102 0956797620968789 2
#> 103 0956797620970548 3
#> 104 0956797620970559 3
#> 105 0956797620971298 7
#> 106 0956797620971652 4
#> 107 0956797620972116 4
#> 108 0956797620972688 2
#> 109 0956797620975781 5
#> 110 0956797620984464 5
#> 111 0956797620985832 9
#> 112 09567976211001317 6
#> 113 09567976211005465 3
#> 114 09567976211005767 7
#> 115 09567976211007414 16
#> 116 09567976211007788 14
#> 117 09567976211010718 9
#> 118 09567976211011969 9
#> 119 09567976211013045 0
#> 120 09567976211015941 5
#> 121 09567976211015942 4
#> 122 09567976211016395 1
#> 123 09567976211016410 8
#> 124 09567976211017870 9
#> 125 09567976211018618 16
#> 126 09567976211019950 11
#> 127 09567976211024259 17
#> 128 09567976211024260 4
#> 129 09567976211024535 10
#> 130 09567976211026983 7
#> 131 09567976211028978 3
#> 132 09567976211030630 4
#> 133 09567976211032224 6
#> 134 09567976211032676 4
#> 135 09567976211037971 8
#> 136 09567976211040491 14
#> 137 09567976211040803 12
#> 138 09567976211043426 6
#> 139 09567976211043428 4
#> 140 09567976211046884 3
#> 141 09567976211048485 1
#> 142 09567976211049439 16
#> 143 09567976211051272 8
#> 144 09567976211052476 9
#> 145 09567976211055375 13
#> 146 09567976211059801 5
#> 147 09567976211061321 14
#> 148 09567976211068045 2
#> 149 09567976211068070 1
#> 150 09567976211068880 3
#> 151 0956797621991137 2
#> 152 0956797621991548 6
#> 153 0956797621995197 10
#> 154 0956797621995202 16
#> 155 0956797621996660 7
#> 156 0956797621996667 9
#> 157 0956797621997350 4
#> 158 0956797621997366 12
#> 159 0956797621998312 5
#> 160 09567976221079633 1
#> 161 09567976221082637 7
#> 162 09567976221082938 11
#> 163 09567976221082941 11
#> 164 09567976221083219 10
#> 165 09567976221086513 4
#> 166 09567976221089599 7
#> 167 09567976221094036 12
#> 168 09567976221094782 9
#> 169 09567976221101045 8
#> 170 09567976221114055 9
#> 171 09567976221116816 2
#> 172 09567976221116892 6
#> 173 09567976221116893 5
#> 174 09567976221119391 6
#> 175 09567976221121348 5
#> 176 09567976221131519 4
#> 177 09567976221131520 9
#> 178 09567976221134476 8
#> 179 09567976221139496 1
#> 180 09567976221140326 1
#> 181 09567976221140341 16
#> 182 09567976221145316 4
#> 183 09567976221147258 2
#> 184 09567976221147259 6
#> 185 09567976221150616 5
#> 186 09567976231151581 1
#> 187 09567976231154804 3
#> 188 09567976231156413 6
#> 189 09567976231156793 1
#> 190 09567976231158288 7
#> 191 09567976231158570 4
#> 192 09567976231160098 2
#> 193 09567976231160702 10
#> 194 09567976231161565 4
#> 195 09567976231164553 2
#> 196 09567976231165267 1
#> 197 09567976231170878 2
#> 198 09567976231172500 4
#> 199 09567976231173900 15
#> 200 09567976231173902 2
#> 201 09567976231177968 2
#> 202 09567976231179378 1
#> 203 09567976231180578 10
#> 204 09567976231180588 2
#> 205 09567976231180881 5
#> 206 09567976231184887 8
#> 207 09567976231185127 3
#> 208 09567976231185129 6
#> 209 09567976231188107 11
#> 210 09567976231188124 0
#> 211 09567976231190546 7
#> 212 09567976231192241 1
#> 213 09567976231194221 0
#> 214 09567976231194590 18
#> 215 09567976231196145 5
#> 216 09567976231198194 11
#> 217 09567976231198435 3
#> 218 09567976231199440 3
#> 219 09567976231203139 1
#> 220 09567976231204035 12
#> 221 09567976231207095 7
#> 222 09567976231213572 9
#> 223 09567976231217508 1
#> 224 09567976231218640 2
#> 225 09567976231220902 6
#> 226 09567976231221789 1
#> 227 09567976231222288 4
#> 228 09567976231222836 4
#> 229 09567976231223130 6
#> 230 09567976231223410 9
#> 231 09567976241227411 6
#> 232 09567976241228504 4
#> 233 09567976241232891 3
#> 234 09567976241235931 4
#> 235 09567976241235932 0
#> 236 09567976241239932 3
#> 237 09567976241239935 9
#> 238 09567976241242105 5
#> 239 09567976241243370 4
#> 240 09567976241245695 8
#> 241 09567976241246561 2
#> 242 09567976241249183 5
#> 243 09567976241254312 3
#> 244 09567976241258149 9
#> 245 09567976241260247 4
#> 246 09567976241263344 8
#> 247 09567976241263347 7
#> 248 09567976241266516 4
#> 249 09567976241267854 5
#> 250 09567976241279291 11exact_p
List any p-values that may have been reported with insufficient precision (e.g., p < .05 or p = n.s.).
imprecise <- module_run(paper, "exact_p")
imprecise$table # print table#> # A tibble: 1 × 9
#> text p_comp p_value section header div p s id
#> <chr> <chr> <dbl> <chr> <chr> <dbl> <dbl> <int> <chr>
#> 1 p < .025 < 0.025 results Results 9 2 3 0956797620955209You can use the expand_text() function to get the full
sentence or paragraph for context. Here you can see that “p < .025”
was not an imprecisely reported p-value, but a description of the
preregistered alpha threshold.
exp <- expand_text(imprecise$table, paper)
exp$expanded # print expanded text#> [1] "The main effect of illness recency did not meet our preregistered threshold (p < .025)-recently ill: M = 661 ms, SD = 197; not recently ill: M = 626 ms, SD = 153, F(1, 400) = 4.23, η p 2 = .010, 90% CI = [.000, .039], p = .040-nor did the interaction between illness recency and face type (disfigured vs. typical), F(1, 400) = 1.87, η p 2 = .005, 90% CI = [.000, .027], p = .173."We can investigate the most common imprecise p-values in the PsychSci set. “p < .01” and “p < .05” are probably often describing figures or tables, but what is the deal with “p > .25”?
imprecise_ps <- module_run(psychsci, "exact_p")
imprecise_ps$table |>
count(text, sort = TRUE) |>
head()#> # A tibble: 6 × 2
#> text n
#> <chr> <int>
#> 1 "p < .01" 137
#> 2 "p < .05" 135
#> 3 "p < .05 " 35
#> 4 "p > .250" 17
#> 5 "p < .005" 16
#> 6 "p < .01 " 15We can expand the text to check the context for “p > .25”.
gt.25 <- imprecise_ps$table |>
filter(grepl("\\.25", text)) |>
expand_text(paper = psychsci)
gt.25$expanded[1:3] # look at the first 3#> [1] "There was a significant interactive effect of time and political orientation, b = -0.09, SE = 0.04, 95% CI = [-0.16, -0.02], t(1922) = -2.49, p = .013, on the endorsement of the in-group foundation (see Table S2 in CI = [-0.14, -0.04], t(1922) = -3.59, p < .001, disappeared after 7/7, b = 0.004, SE = 0.02, 95% CI = [-0.04, 0.05], t(1922) = 0.17, p > .250 (see Fig."
#> [2] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."
#> [3] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."marginal
List all sentences that describe an effect as ‘marginally significant’.
marginal <- module_run(paper, "marginal")
marginal # print tableNo effects were described as marginally/borderline/close to significant.
Let’s check how many are in the full set.
marginal_ps <- module_run(psychsci, "marginal")
marginal_ps$table # print table#> # A tibble: 99 × 7
#> text section header div p s id
#> <chr> <chr> <chr> <dbl> <dbl> <int> <chr>
#> 1 Although the PTSD group showed a sign… results Detai… 11 1 4 0956…
#> 2 A marginally significant negative cor… results Postt… 13 1 4 0956…
#> 3 When we more closely matched depressi… results The e… 14 3 1 0956…
#> 4 In that analysis, the group differenc… results Condi… 15 1 2 0956…
#> 5 The Congruency × Alignment interactio… results Resul… 7 2 3 0956…
#> 6 The twoway interactions between knowl… results Resul… 6 2 3 0956…
#> 7 An omnibus 3 (sex ratio) × 2 (partici… results Resul… 15 2 1 0956…
#> 8 Further, we observed a marginally sig… fig NA 3 NA 2 0956…
#> 9 The dagger and asterisks indicate mar… fig Fig. … 5 NA 7 0956…
#> 10 Given the unexpected nature of the fi… annex Open … 20 1 21 0956…
#> # ℹ 89 more rowsosf_check
List all OSF links and whether they are open, closed, or do not exist. This requires an internet connection to check the status of the link.
osf_check <- module_run(paper, "osf_check")#> Starting OSF retrieval for 1 files...#> * Retrieving info from k2dbf...#> ...OSF retrieval complete!
osf_check$summary#> id osf.open
#> 1 0956797620955209 3The returned table contains additional information about the links if they are found and open.
osf_check$table[1,] |> t()#> [,1]
#> text "osf.io/k2dbf"
#> section "method"
#> header "Analyses"
#> div "8"
#> p "1"
#> s "1"
#> id "0956797620955209"
#> osf_id "k2dbf"
#> name "Preregistered replication of “Sick body, vigilant mind: The biological immune system activates the behavioral immune system”"
#> description ""
#> osf_type "nodes"
#> public "TRUE"
#> category "project"
#> registration "FALSE"
#> preprint "TRUE"
#> parent NA
#> status "open"You can also run this module on a batch of papers. You can only make
100 API requests per hour, unless you authorise your requests, when you
can make 10K requests per day. The OSF functions in papercheck often
make several requests per URL to get all of the info, so it’s worthwhile
setting your PAT. You can authorise them by creating an OSF token at https://osf.io/settings/tokens and including the
following line in your .Renviron file (which you can open using
usethis::edit_r_environ()):
OSF_PAT="replace-with-your-token-string"This would take a while to run for the 333 unique OSF links found in the PsychSci set (usually a few links per second), so we will skip it here (but can reveal we found 10 closed links).
osf_check_ps <- module_run(psychsci, "osf_check")ref_consistency
Check if all references are cited and all citations are referenced.
ref_consistency <- module_run(paper, "ref_consistency")
ref_consistency$table#> bib_id
#> 1 b16
#> 2 b17
#> 3 b3
#> 4 b31
#> ref
#> 1 Jones BC, Hahn AC, Fisher CI, Wang H, Kandrik M, Tybur JM, Debruine LM (2018). “Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis.” _Evolution and Human Behavior_, *39*, 166-169.
#> 2 Jones BC, Hahn AC, Fisher CI, Wang H, Kandrik M, Tybur JM, Debruine LM (2018b). “Reply to Fleischman and Fessler's (2018) comment on "Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis.” _Evolution and Human Behavior_, *39*, 470-471.
#> 3 Curtis VA (2014). “Infection-avoidance behaviour in humans and other animals.” _Trends in Immunology_, *35*, 457-464.
#> 4 Tybur JM, Inbar Y, Aarøe L, Barclay P, Barlow FK, De Barra M, Žeželj .., I (2016). “Parasite stress and pathogen avoidance relate to distinct dimensions of political ideology across 30 nations.” _Proceedings of the National Academy of Sciences_, *113*, 12408-12413.
#> doi bibtype
#> 1 <NA> Article
#> 2 <NA> Article
#> 3 <NA> Article
#> 4 <NA> Article
#> title
#> 1 Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis
#> 2 Reply to Fleischman and Fessler's (2018) comment on "Hormonal correlates of pathogen disgust: Testing the compensatory prophylaxis hypothesis
#> 3 Infection-avoidance behaviour in humans and other animals
#> 4 Parasite stress and pathogen avoidance relate to distinct dimensions of political ideology across 30 nations
#> journal year
#> 1 Evolution and Human Behavior 2018
#> 2 Evolution and Human Behavior 2018b
#> 3 Trends in Immunology 2014
#> 4 Proceedings of the National Academy of Sciences 2016
#> authors
#> 1 B C Jones, A C Hahn, C I Fisher, H Wang, M Kandrik, J M Tybur, L M Debruine
#> 2 B C Jones, A C Hahn, C I Fisher, H Wang, M Kandrik, J M Tybur, L M Debruine
#> 3 V A Curtis
#> 4 J M Tybur, Y Inbar, L Aarøe, P Barclay, F K Barlow, M De Barra, . . Žeželj, I
#> id missing
#> 1 0956797620955209 citation
#> 2 0956797620955209 citation
#> 3 0956797620955209 citation
#> 4 0956797620955209 citationIt looks like there are some references with missing citations. The first one doesn’t look like a reference, and grobid often parses tables oddly. You’d need to look at the original PDF to see if the others are actually missing or false positives. Here, they are all false positives, based on grobid not being able to match the in-text citation to the reference list.
retractionwatch
Flag any cited papers in the RetractionWatch database. It takes a substantial time to make calls to crossref for each reference, so we provide a summary of the database in papercheck, which is updated regularly (last 2025-02-28).
rw <- module_run(paper, "retractionwatch")
rw$table#> [1] bib_id ref doi bibtype
#> [5] title journal year authors
#> [9] id retractionwatch
#> <0 rows> (or 0-length row.names)In the full PsychSci set, we find three papers in the RetractionWatch database cited, one with a retraction and two with corrections.
rw_ps <- module_run(psychsci, "retractionwatch")
rw_ps$table#> bib_id
#> 1 b15
#> 2 b25
#> 3 b36
#> 4 b25
#> 5 b25
#> 6 b25
#> ref
#> 1 Kok BE, Coffey KA, Cohn MA, Catalino LI, Vacharkulksemsuk T, Algoe SB, Brantley M, Fredrickson BL (2013). “How positive emotions build physical health: Perceived positive social connections account for the upward spiral between positive emotions and vagal tone.” _Psychological Science_, *24*(7), 1123-1132. doi:10.1177/0956797612470827 <https://doi.org/10.1177/0956797612470827>.
#> 2 Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*(10), R115. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 3 (????). doi:10.1038/s41562-023-01749-9 <https://doi.org/10.1038/s41562-023-01749-9>.
#> 4 Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 5 Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> 6 Horvath S (2013). “DNA methylation age of human tissues and cell types.” _Genome Biology_, *14*, 3156. doi:10.1186/gb-2013-14-10-r115 <https://doi.org/10.1186/gb-2013-14-10-r115>.
#> doi bibtype
#> 1 10.1177/0956797612470827 Article
#> 2 10.1186/gb-2013-14-10-r115 Article
#> 3 10.1038/s41562-023-01749-9 Misc
#> 4 10.1186/gb-2013-14-10-r115 Article
#> 5 10.1186/gb-2013-14-10-r115 Article
#> 6 10.1186/gb-2013-14-10-r115 Article
#> title
#> 1 How positive emotions build physical health: Perceived positive social connections account for the upward spiral between positive emotions and vagal tone
#> 2 DNA methylation age of human tissues and cell types
#> 3 <NA>
#> 4 DNA methylation age of human tissues and cell types
#> 5 DNA methylation age of human tissues and cell types
#> 6 DNA methylation age of human tissues and cell types
#> journal year
#> 1 Psychological Science 2013
#> 2 Genome Biology 2013
#> 3 Behaviour <NA>
#> 4 Genome Biology 2013
#> 5 Genome Biology 2013
#> 6 Genome Biology 2013
#> authors
#> 1 B E Kok, K A Coffey, M A Cohn, L I Catalino, T Vacharkulksemsuk, S B Algoe, M Brantley, B L Fredrickson
#> 2 S Horvath
#> 3 NA
#> 4 S Horvath
#> 5 S Horvath
#> 6 S Horvath
#> id retractionwatch
#> 1 09567976221150616 Correction
#> 2 09567976231194221 Correction
#> 3 09567976241249183 Retraction
#> 4 09567976241260247 Correction
#> 5 09567976241260247 Correction
#> 6 09567976241260247 Correction
#> text
#> 1 The vagus nerve controls respiratory sinus arrhythmia (RSA), the covariation of heart rate and respiration, known to be an index of cardiac health and predictor of enhanced capacity for emotion regulation (Butler et al., 2006;Kok et al., 2013).
#> 2 Epigenetic aging is one such indicator of accelerated biological aging (Horvath, 2013).
#> 3 <NA>
#> 4 Discrepancies between someone's biological and chronological age translate into "age acceleration," with older biological age relative to chronological age indicating greater acceleration (Horvath, 2013).
#> 5 EAA from the Horvath PanTissue clock (Horvath, 2013) was calculated using the methylclock package (Pelegí-Sisó et al., 2021).
#> 6 The replication of results across the PedBE and Horvath PanTissue clocks increases our confidence in study findings because these tools share no overlapping DNAm sites (Horvath, 2013;L.Check the context of the citations in the text column:
rw_ps$table$text#> [1] "The vagus nerve controls respiratory sinus arrhythmia (RSA), the covariation of heart rate and respiration, known to be an index of cardiac health and predictor of enhanced capacity for emotion regulation (Butler et al., 2006;Kok et al., 2013)."
#> [2] "Epigenetic aging is one such indicator of accelerated biological aging (Horvath, 2013)."
#> [3] NA
#> [4] "Discrepancies between someone's biological and chronological age translate into \"age acceleration,\" with older biological age relative to chronological age indicating greater acceleration (Horvath, 2013)."
#> [5] "EAA from the Horvath PanTissue clock (Horvath, 2013) was calculated using the methylclock package (Pelegí-Sisó et al., 2021)."
#> [6] "The replication of results across the PedBE and Horvath PanTissue clocks increases our confidence in study findings because these tools share no overlapping DNAm sites (Horvath, 2013;L."statcheck
Check consistency of p-values and test statistics using functions from statcheck.
statcheck <- module_run(paper, "statcheck")
statcheck$table#> test_type df1 df2 test_comp test_value p_comp reported_p computed_p
#> 1 t NA 248.4 = 2.01 = 0.023 0.04551244
#> raw error decision_error one_tailed_in_txt apa_factor
#> 1 t(248.4) = 2.01, p = .023 TRUE FALSE FALSE 1
#> text
#> 1 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (d z = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (d z ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p < .001.
#> section header div p s id
#> 1 results Results 9 3 3 0956797620955209Here we see a false positive, where the paper reported the results of an equivalence test, which are meant to be one-tailed, but statcheck did not detect that this was one-tailed.
In the full PsychSci set, there are more than 27K sentences with numbers to check, so this takes about a minute to run.
statcheck_ps <- module_run(psychsci, "statcheck")There will be, of course, some false positives in the full set of 151 flagged values. Let’s look just at the flagged values where the computed p-value is about double the reported p-value, and this changes the significance decision (at an alpha of 0.05).
statcheck_ps$table |>
filter(decision_error,
round(computed_p/reported_p, 1) == 2.0) |>
select(reported_p, computed_p, raw) |>
mutate(computed_p = round(computed_p, 4))#> reported_p computed_p raw
#> 1 0.0290 0.0589 F(1, 361) = 3.59, p = .029
#> 2 0.0470 0.0947 t(24) = 1.74, p = .047
#> 3 0.0270 0.0547 t(24) = 2.02, p = .027
#> 4 0.0400 0.0797 t(24) = 1.83, p = .040
#> 5 0.0480 0.0962 t(240) = 1.67, p = .048
#> 6 0.0460 0.0915 t(32) = 1.74, p = .046
#> 7 0.0420 0.0846 t(21) = 1.81, p = .042
#> 8 0.0343 0.0686 t(10) = 2.04, p = .0343
#> 9 0.0330 0.0654 t(55) = 1.88, p = .033Chaining Modules
Modules return a summary table as well as the detailed
results table, which is automatically added to the summary
if you chain modules.
ps_metascience <- psychsci[1:10] |>
module_run("all_p_values") |>
module_run("exact_p") |>
module_run("marginal")
ps_metascience$summary#> id p_values exact_p imprecise_p marginal
#> 1 0956797613520608 6 6 0 0
#> 2 0956797614522816 39 39 0 0
#> 3 0956797614527830 13 10 3 0
#> 4 0956797614557697 27 16 11 0
#> 5 0956797614560771 4 3 1 0
#> 6 0956797614566469 0 0 0 0
#> 7 0956797615569001 25 2 23 0
#> 8 0956797615569889 26 24 2 4
#> 9 0956797615583071 24 21 3 0
#> 10 0956797615588467 21 13 8 0