Why we need to clean the Augean stables of ecology – the case of InsectChange

In my first review of this paper, I wrote "... I recommend publication after minor revision." Given that, I do not feel that another round of review or revision is required, and the handling editor has all of the information needed from me to make a decision that considers my review.

For what it's worth, I have read the response from the authors and feel they did a good job of addressing my rather minor concerns from the original version of the paper. Therefore, I again recommend publication.

Dear authors, 
 
first of all, thanks for your thorough work on the revision. I found the new manuscript version to be significantly improved. Figures are more intuitive, and most importantly, I believe the new analyses and information presented in the manuscript provide a more objective measure of the magnitude of the impact of errors and inconsistencies in InsectChange on the estimation of insects’ temporal trends.
 
I went through the manuscript once again and annotated some minor points that, in my opinion, could be addressed to further improve clarity. As you will see, these are all minor comments. 
 
Specific comments:
 
Table 1: second row, column ‘Definition’: a white space is missing between the words ‘often’ and ‘macroinvertebrates’. Second row, column ‘Consequences and risks’: shouldn’t it be “the consequence of which depends on the ..”?. Fourth row, column ‘Definition’: there is a period missing at the end of the sentence. You may want to double-check the table to fix these types of typos. 
It looks like ‘Inadequate geographic coordinates..’ and ‘Studies with internal drivers’ were put together under the ‘methodological issues’ section. This is the only instance where two ‘problem types’ have been grouped together. 
Finally, I suggest to always indicate if a file/table/metadata belongs to the InsectChange paper or if it is supplementary material related to your manuscript. As an example, in the ‘Definition’ column under ‘inconsistencies’ and ‘inadequate temporal resolution’, you refer to the ‘MetadataS1’, but it is unclear whether this supplementary material belongs to your article or to the InsectChange paper. I am aware that at the bottom of the table, you mention that some ‘files’ are from InsectChange. However, this refers only to .csv files.
 
Table 1: I apologise, but it is not entirely clear to me what is the difference between ‘Overlapping studies/plots’ and ‘Inflation of studies/plots’. The resulting consequences also appear very similar. Would it be possible to merge the two types for simplicity?
 
Lines 148-151: the parenthesis at line 148 is never closed.
 
Lines 190-192: why not include the ‘unstandardized abundance measures’ among the problem types listed in Table 1. Especially if this issue affects the whole dataset.
 
Line 262: ‘Italia’ -> ‘Italy’.
 
Lines 469-472: I might be missing something here, but 486 + 496 is not equal to 985.
 
As a side note: I apologise as I probably did not express myself correctly in my first comment letter to your manuscript. I never intended to say that improving InsectChange was your task. Rather, I meant that your work provided an invaluable source of information to enable the original creators of InsectChange to improve the database.

Summary

This paper by Gaume and Desquilbet is an important piece of work and I recommend publication after minor revision.

Gaume and Desquilbet take a deep dive into the InsectChange database that has been assembled by other researchers and used to publish several peer-reviewed papers in high-profile journals like Science. Some of these papers have claimed to find trends in insect abundance that run counter to existing knowledge. For example, authors who have used the InsectChange database have published meta-analyses claiming that while terrestrial insect abundances are in decline, freshwater insect abundances are increasing. The latter trend runs counter to a considerable body of prior research.

Inferences from meta-analyses are only as good as the data they are based on (e.g., garbage in-garbage out). And datasets like InsectChange are only useful when the information contained in them is accurate, reliable, and appropriate for testing the proposed hypotheses.

In the current paper, Gaume and Desquilbet show that the InsectChange database is riddled with data errors, inconsistencies, methodological problems, and information gaps showing this dataset did not receive the quality assurance/quality control needed for use by scientific studies. The authors go on to show how these errors, inconsistencies, problems, and information gaps undermine results of publications that have used this database and cast doubt on their conclusions. As just one example, conclusions about freshwater “insects” from analyses of the InsectChange database appear to be incorrect, as they are driven by increasing abundances of non-insect invertebrates that tend to increase as water quality declines (e.g., oligochaeta, turbellaria, amphipoda). These non-insect invertebrates were apparently added to the dataset by accident, as those assembling the InsectChange database did not properly differentiate between insects and other types of non-insect invertebrates.

I am highly supportive of publication of the critique by Gaume and Desquilbet. It took a huge amount of work to review the quality of InsectChange, and the authors have done the scientific community a huge service in doing so. I wish more databases like InsectChange received the scrutiny they should prior to being used for numerous publications.

I hope my suggestions below are helpful:

Editorial work. The manuscript contains several English language errors and would benefit from more editing before publication. I kept track of some issues during my reading of the first part of the paper, and mention these below under ‘Suggested edits’. However, there were enough editorial errors that I stopped noting them about a third of the way through the paper so that I could focus on the more substantive issues. I do want to emphasize that none of the editorial issues affect the analyses, conclusions, or message of this paper. But the paper would be a bit easier to read with a tad more editorial work.

Reliance on examples. In Figures 2-4, the authors rely on use of select examples pulled from the InsectChange dataset to illustrate their points. Yet, it is not always clear whether these select examples are broadly representative of the dataset as a whole. For example, the text starting line 72 reads “Among the errors, 35 datasets considered taxa other than insects, arachnids or entognaths (hereafter 73 collectively referred to as “insects” for brevity), most often including the entire invertebrate assemblage 74 instead of insects only, sometimes biasing the trends to the point of reversal (Figure 2a).” Figure 2a might lead the reader to believe that these errors in 35 datasets alter conclusions about time-series. However, upon closer inspection it looks as if the authors have chosen just one plot from a singles study to illustrate the problem (Study 1435 – Plot 448).

There are other places in the text / figures where it is clear that Gaume and Desquilbet have assessed the entire InsectChange database and concluded that errors are pervasive (e.g., Figure 3b). Even so, the text could be more clear about when select examples are being used to illustrate a problem vs. when analyses are illustrating a collection of errors (i.e. 35 datasets where non-insects were considered) that lead to conclusions different from the original papers.

Problems and their impact. It was not always clear to me from the text how select problem types shown in Figure 1a might impact conclusions from the InsectChange database. For example, I did not initially understand how problem type 1 – Inadequate geographical coordinates -- might influence conclusions from the database. It wasn’t until I read Appendix 1 describing the consequences and risks of each problem type that it became apparent to me.

The manuscript seems to rely heavily on readers going into the Appendices to get more information before they can understand the main body of the text. The authors may want to consider taking essential elements out of the Appendices and putting them into the main body of the paper to make the reading easier. For example, Appendix 1 could be a Table in the main body of the paper … or, if that is too cumbersome … perhaps be more clear when describing the consequences of each problem type in the main body of text.

Suggested edits.

Line 22. Change “… allow one to study”

Line 23. Change to “… and extreme vigilance in use of the InsectChange database”

Line 34. Change to “… with trend assessments hampered by lack of data …”

Line 41. Change to “… than reported by prior authors, and they further proposed that the diversity of freshwater insects is increasing rather than decreasing.”

Line 61. Change ‘totalizing’ to totaling.

Lines 74-77 (Figure 2b). I was unable to tell whether the sentence was referencing the blue line in Figure 2b (insects) or the red line (moths+beetles reported as insects). It was difficult to understand the error tilted ‘Misreporting of select insect groups’ based on the text and figure.

Lines 79-80 (Figure 2f). Based on the text, I was not able to understand what was meant by ‘datasets or plots had overlapping data’.

Lines 80-82. It was unclear to me why erroneous or imprecise geographic coordinates in 88 datasets might lead to inaccurate conclusions about insect time trends.

Line 90. Should this line be referring to Figure 2e or a different panel?

The manuscript “InsectChange: Comment” presents an exhaustive review of InsectChange, a large database of long-term time-series on insects, arachnids and entognaths’ assemblages. The authors made an extraordinarily thorough work in scanning InsectChange (dataset per dataset). They detected (and reported on) data errors and inconsistencies that were inherited from other databases (e.g., BioTIME and Global Population Dynamics Database) or introduced by the processing of the original datasets. Importantly, the authors made the effort to compare the datasets included in InsectChange (in their current status) with the corresponding original data sources. Beside tracing back the origin of data errors and inconsistencies, this allowed identifying other methodological issues affecting InsectChange. Examples are the lack of information on whether data were collected under some experimental treatment or the undocumented inclusion of invertebrates other than insects in assemblages thought to consist only of insects.

I personally agree with the authors that some of the issues they found in InsectChange could potentially bias estimation of temporal change of insects. I believe that their review is an important basis of improvement of InsectChange. Considering (and addressing) all errors and inconsistencies highlighted by the authors will surely make InsectChange a better database to investigate insects’ trends across wide spatiotemporal scales!

My main concern is not about the review itself, which, once again, is an important piece of work. Several times in the comment, the authors claim that the issues they found (fully) undermine or invalid previous findings based on InsectChange (often referring to the global analyses carried out by van Klink et al. 2020). I agree that addressing all errors, inconsistencies and other issues in InsectChange, and re-running analyses with a ‘better’ version of the database, would likely change the ‘numbers’ estimated in previous works (e.g., measures of effect size or estimates of temporal trend in insects’ abundance). Yet, the authors did not explicitly test that in their comment. At least, I did not find a comparison between findings from previous studies and those obtained using an updated version of InsectChange. Rather, the authors made a large series of examples of datasets affected by data-related errors or methodological issues, and of the potential bias they could introduce when estimating temporal trends in insect change. However, these dataset-specific examples do not allow quantifying/guessing the real impact of the different issues on previous research (e.g., reversing of trends in insect change previously described in studies implemented at large spatial scales). This is just to say that I would personally avoid stating that errors found in InsectChange fully undermine previous research (as stated in the conclusions section), as the authors never explicitly quantified how much previous findings based on InsectChange deviate from those potentially obtained addressing all errors and issues. I see this review as an invaluable basis for improvement rather than as a means to debate on the validity of previous work (at least as long as no explicit comparison on that is provided). 

A somehow similar concern is that authors made several statements about the danger posed by some methodological issues affecting InsectChange (in the main text as well as in the appendices), but it was not always clear to me what was the real impact of these issues on the database. As an example, section #4 of the comment discusses the accuracy of local cropland cover derived from remote sensing data (ESA-CCI), which is provided in InsectChange as a ready-to-use driver to analyze insect change. In short, based on a comparison of cropland cover as reported in InsectChange with information from the original data sources (and the visual interpretation of satellite images), the authors conclude that ESA-CCI data are not adequate to estimate local cropland cover, as these data consistently under/over estimate it. Although the authors present a detailed comparison in the CroplandCover.xlsx appendix, they do not report in the main text what is the average difference between the cropland cover reported in InsectChange and the correct cover reported either in the original source or visually estimated from satellite imagery. I would suggest to provide (whenever possible) simple measures of the difference between data provided in InsectChange vs. original or alternative sources. This would greatly help readers to understand what is the real impact of a given issue on the database.

List of detailed comments:

Line 67 (Figure 1 caption): what does the ‘*’ refers to in the figure?

Line 70 (Figure 1 caption): does ‘studies’ mean the same as ‘datasets’ here? The terms ‘studies’ and ‘datasets’ are used interchangeably in the text, but it is not always clear whether they always mean the same thing.

Line 74: would it be possible to quantify (and report) how many times temporal trends were reversed if exclusively considering insects? This could provide a clearer idea of the potential effect of this error on the temporal trends estimated using InsectChange.

Lines 80-82: what does ‘erroneous coordinates’ exactly mean here? Does it mean that, for example, a plot located somewhere in Spain was instead erroneously located in Greece? Or rather that longitude and latitude were swapped? Similarly, I would clarify what is meant by ‘not precise enough coordinates’? Does this refer to the number of decimal places of the coordinates pair? I believe these details are important to allow readers understand what is the problem with these data. Finally, it is not clear whether errors with coordinates were introduced by InsectChange or somehow inherited from the original data sources.

Figure 2: In the figure caption, I would mention that ‘blue’ is used to represent insect change as obtained through a ‘correct’ use of the data, while ‘red’ is used to represent insect change as obtained using data in their current status. Also, the background of the plots could be colored to better discriminate between examples of errors and inconsistencies (for instance using the same color palette used in Figure 1a).

Line 90: note that Figure 2e is about ‘non consideration of sampling effort’ (and not about 'inconsistencies of taxa between plots of a same dataset’).

Line 114: note that the Excel file is named ‘FreshNoInsects’ and not ‘FreshNonInsects’.

Figure S1 (Appendix S2): what does ‘computable albeit not considered’ mean in the title of Figure S1? Also, I would avoid referring to other appendices (or to the main text) to find details about a figure or table. As an example, in Figure S1 (Appendix S1) authors refer to the caption of Figure 3 (main text) to read about what the colors of the pie chart are associated with. I strongly suggest to avoid that as the comment (main text and appendices) is very dense and one can get easily lost while looking for information across appendices.

Line 122: it should be ‘in almost half of the plots’ here, right?

Figure 3: not entirely clear to me what ‘insect % sometimes computable’ means? Also, I do not understand what ‘insects inferred to be dominant’ means.

Lines 135-146: I understand (and agree) that ‘internal drivers’ (i.e. those originally considered/imposed in the source studies) may better explain insect change than ‘external drivers’ (i.e. those derived from external databases and not necessarily linked to the experimental, study-specific context). However, I am wondering how (and if) one could really account for all study-specific contingencies in analyses focused on large spatiotemporal scales, such as that carried out by van Klink et al. (2020). Should we refrain from doing large scale analyses if we can’t control for all local-scale drivers possibly affecting context-specific changes? Should we break the database in small pieces to analyze separately and then try to put together inferences drawn from these single analyses? Isn’t the whole idea of large scale analyses about finding emerging patterns rather than explain local contingencies? 

I am saying this as the authors (rightfully) mention several case-specific studies where trends of insect change deviated from those expected by the original authors of the study (or from those reported by van Klink et al.). However, I am wondering how these issues should be accounted for when analyzing all data together with the aim of finding general/global trends of insect change (and/or possible correlates of the change to make predictions). 

Figure 4g: I would use a different palette for this plot, as the same colors are also used in panels 4a to 4f but are associated with different meanings.

Lines 187-188: not entirely clear to me what is meant by ‘plots with distinct geographical coordinates’. Are these the plots that do not belong to groups of plots that were assigned a unique pair of coordinates? Please, clarify.

#—————

Appendix S4 (no line numbering)

Point #1 of the list of possible causes of the incorrect assessment of local cropland cover: the authors highlight a mismatch between the cropland cover reported in original studies and that estimated from remote sensing. I would not be surprised by this mismatch if in the original studies cropland cover was estimated at a truly local scale. ESA-CCI remote sensing data provide worldwide cropland cover at a (not that coarse) spatial resolution (300 m x 300 m), and note that InsectChange aggregates data across an even larger spatial window. I believe this sort of mismatches are inevitable when comparing (truly) local vs. global remote sensing data. Also, I am wondering why the visual interpretation of satellite images (mentioned in the first sentence of point #1) should provide a better estimation of cropland cover than that provided by ESA-CCI, given the difficulty of distinguishing between grasslands and croplands acknowledged also by the authors.

In point #1, I missed a more rigorous (and objective) quantification of the mismatch between locally estimated cropland cover and that derived from ESA-CCI (e.g., average difference between cropland cover reported in original studies or visually estimated from satellite images vs. that reported in InsectChange). This could help readers understanding what is real magnitude (and potential impact) of this issue on InsectChange. Finally, last sentence of point #1 sounds to me more as a critique to the ESA-CCI land cover product than to InsectChange.

Point #2: once again, it is not clear to me what the authors mean by ‘inaccurate location’. Please clarify.

As already pointed out above, I would avoid reporting details about figures in the main text in an appendix (and vice-versa).

#—————

Figure 5: I would include information to interpret Figure 5b in the caption. Concerning Figure 5a, what are the numbers on the y-axis about? 

Figure 5 reports a rather extreme case of mismatch between cropland cover reported in InsectChange and what estimated by the authors (or found in the original source). I think that it would be good to show what is the average mismatch between the information derived from ESA-CCI and that estimated by the authors. This could be done using data reported in the CroplandCover Excel file. I believe such a comparison would more objectively demonstrate the magnitude of the issue.

Line 215: I would be more cautious here and state that errors, inconsistencies and methodological issues found in InsectChange could bias estimation of temporal trends in insect change (rather than claim they fully undermine what found by, e.g., van Klink et al. 2020). Indeed, although several concerning examples on the misuse of data included in InsectChange are highlighted in the comment, authors mostly focus on dataset-specific issues and do not quantify their overall impact on the conclusions of previous works based on InsectChange.

#—————

Appendix S1 (no line numbering)

Minor issue: I would move the “Presentation of the different appendices”, which describes all appendices, outside Appendix S1.

I would move the section about “Other problems not included in our analysis” at the end of Appendix S1.

Study 70*, line 15: shouldn’t be ‘beyond 1996’ here?

Study 300: a reference for ESA-CCI is missing.

Study 375*, last sentence of the paragraph: this statement is vague as no measure of the difference between cropland cover from ESA-CCI and that visually estimated from satellite images is reported.

#—————