How to Share Data with a Statistician

This English version was machine-translated from the Chinese original. Technical terms may require verification.

This post is from a guest article published by Jeff Leek in BMC Blog. How to Share Data with a Statistician, the detailed guidance is derived from GitHub Repository。

It is a practical guide for researchers, students and data compilers who need to work with statisticians. It describes the extent to which data should be collated before they are handed over to statistical analysts in order to reduce communication costs and speed up the analysis process.

The questions in this article can also be addressedScientific theory and practical experience、The Statistical Crisis in ScienceHow the concept of a relatively close read together is developed in different contexts.

How to share data with statisticians

This is a guide for those who need to share data with statisticians. It is possible that you can benefit from this, whether you are a scientific partner, a student or a doctorate who needs the assistance of statisticians in analysing the data, or a primary statistical student who is responsible for organizing/cleaning the data.

The objective of the guidelines is to provide best practices for data sharing and to avoid the pitfalls and delays that are common in the transition from data collection to data analysis. The Leek team, together with a large number of collaborators, found that the most important factor in the difference in output speed is the state of the data when they reach us. According to my interaction with other statisticians, it's almost a universal truth.

I think statisticians should be able to process data in any state. It is important to view raw data, understand each step of the process and include implicit sources of variability in data analysis. However, for many data types, processing steps have been documented and standardized. Therefore, the conversion of data from the original form to a form that can be analysed directly could be accomplished well before seeking the assistance of statisticians. This would significantly accelerate turnaround time, as statisticians do not need to process all pre-treatment steps first.

What are you supposed to offer to statisticians?

To speed up analysis, you should provide the following information to statisticians:

1. Source Data (The Raw Data)
2. Clean Data Set (A dy data set)
3. Code book (A code book): Describes each variable in the clean data set and its values.
4. Processing of records: a clear and precise recipe that records how you moved from step 1 to step 2 and 3.

Each part of the package is presented in turn.

1. Raw data (The Raw data)

To provide statisticians with what you can get.OriginalData is important. This ensures that data trace across the entire workflow.

The following are examples of some raw data forms:

• The odd binary files that the measuring machine spit out.
• The company signed the Excel file containing 10 sheets.
• The complex JSON data that you can retrieve from Twitter API.
• The numbers are observed through microscopes and manually recorded.

You can judge whether the original data is in the correct format by the following criteria:

1. No software was run for the data.
2. No data values have been modified.
3. No data from the data collection was removed.
4. Data are not aggregated in any way.

If you make any changes to the original data, it is no longer in original form.Reporting of modified data as raw data is a common cause of slowing down the analysis processBecause analysts usually have to do “cord-in” your data to figure out why the original data look strange. (And imagine what would happen if new data arrived?

2. Clean data set (A dy data set)

The general principle of clean data (Tidy Data) is that Hadley Wickham is a man who is a man of the world.This paper.Submitted by: Although, as with all truths, these principles are common in R:

1. Each variableYou should measure a column.
2. Every different observation.should be a row under this variable.
3. Each type. The variable should have a table.
4. If you have more than one table, they should contain a column that allows them to be connected or merged.

Although these are mandatory, there are other things that will make your dataset easier to process:

• The first line of each data sheet/ spreadsheet contains the full row name. For example, if you measure the age of the patient at the time of diagnosis, name the column as AgeAtDiagnosis♪ Not like ♪ ADx This way others may find it difficult to understand the acronym.

Examples of genomics

Suppose you have 20 individual RNA-Seq genetic expression measurements. You also collect demographic and clinical information on patients, including age, treatment and diagnosis.

• You should have a table/ spreadsheet containing clinical/demographic information. It will have 4 columns (patient ID, age, treatment, diagnosis) and 21 rows (one line variable, followed by one row for each patient).
• You should have a summary genome data spreadsheet. Usually, such data are aggregated at the count level of each external item. If you have 100,000 externals, you should have a table/ spreadsheet containing 21 rows (one genetic name per patient, one row) and 100,001 columns (one patient ID, one column per data type).

Format Recommendations

If you share data with partners in Excel, clean data should beOne Excel file per tableMedium. They should not have multiple sheets, should not apply macros to data, and should not highlight any columns/cells. Alternatively, share data using a CSV or TAB-separated text file. (But be careful when reading CSV files into Excel, which sometimes leads to the unrecurring processing of date and time variables.

3. Code book (A code book)

For almost any data set, the measurements you calculate require more detailed descriptions than you can plug in in your spreadsheet. The code book is used to contain this information. It's like a “telegraph” for your data.

At the very least, this note should enable statisticians to understand:

1. Information on variables not contained in the clean data (Including units!）。
2. Information about the sum of options you made.
3. Information about the experimental research design you used.

In our genomics case, analysts would like to know what the unit of measurement is for each clinical/demographic variable (age is year? Is the treatment a name or a dose? What is the level of diagnosis and its heterogeneity? They would also like to know how you chose the outsiders to summarize the genome data (UCSC/Ensembl, etc.). They would also like to know any other information on how you are conducting the data collection/research design. For example, are these the first 20 patients who went into the clinic? Are they carefully selected 20 patients by certain characteristics, such as age? Are they randomly assigned to treatment?

The common format for this document is Word (in 2026 in this Rept, perhaps.md has become a close-mainstream format). There should be a chapter called “Research Design” detailing how you collect data. There is also a chapter called "Codebook " describing each variable and its unit.

Variable code recommendations

When you put variables in spreadsheets, you encounter several main types:

1. Continuous Variables (Continuous): Anything measured on a quantitative scale may be any fraction (e.g. weightkg).
2. Order Variable (Ordinal): with fixed, small (<100) Level but sequenced data (e.g. survey responses: poor, general, good).
3. Categorical Variables• There are several categories, but no order of data (e.g. sex: men, women).
4. Missing Data (Missing): No data observed. Should be coded NA。
5. Deleted Data: You know that there is some lack of data on the mechanisms (e.g., the measurement values are below the test limit or the patient is missing).

Avoid Encoding Classification or Order Variables into NumbersI'm sorry. In the clean data, the value for gender should read "male" or "female", instead of one or two. The orderly value should read "poor", "fair", "good", not one, two, three. This avoids potential confusion about the direction of effects and helps identify coding errors.

4. Processing records (The processing list/script)

You may have heard of it,Repetitivity is a big thing in computing science.I'm sorry. This means that when you present your paper, the reviewers and others around the world should be able to fully recreate the analysis from the original data to the end result.

If you try to improve efficiency, you are likely to implement some of the aggregation/data analysis steps before the data are considered “clean”. The ideal thing is to create aComputer Script(in R, Python or other languages) The script is entered with the original data and the exact data you share. You can try running scripts several times to see if the code produces the same output.

If you can't write code, you should give a statistician a name for it.HypocriteSomething. It should look like this:

1. Step 1 - Get original files, summarised software for running version 3.1.2, with a = 1, b = 2, c = 3.
2. Step 2 - Run the software for each sample.
3. Step 3 - The third column of the outputfile.out for each sample is the corresponding row in the output data set.

You should also include information on the system you use (Mac/Windows/Linux) and the software version.

What do you expect from an analyst?

When you transfer a properly organized data set, it significantly reduces the workload of statisticians. So I hope they can get back to you faster. But most careful statisticians will check your processing records, ask you about the steps and try to ascertain whether they can obtain the same neat data as you through a spot check.

Then you should expect from statisticians:

1. I'm gonna do every analysis.Analyse Script(not just an indication).
2. They're used for running analysis.The exact computer code.。
3. Everything they produce.Output file/chart。

These are the information you will use to supplement the material to create the repetitivity and accuracy of the results. Every step of the analysis should be clearly explained, and if you do not understand what the analyst has done, you should ask. Understanding statistical analysis is a shared responsibility of statisticians and scientists. You may not be able to do the exact analysis without the code of a statistician, but you should be able to explain to your lab partner/your chief researcher why the statisticians do every step.