Frequently Asked Questions

1. Project title
2. A brief description of the systematic review (project objectives) and investigative team
3. Prospero registration link (if available)
4. RIS file of your citation list
5. Inclusion and exclusion criteria for abstract and full-text screening
6. On your project page, you may also wish to include: (i) a description of the PI and research team; (ii) tips and advice for reviewers; (iii) whether you are offering any incentive(s) to reviewers; and (iv) upload the systematic review protocol and search strategy

To upload your citation list to the project, you will need an RIS file. This is an export option on reference manager software.  The citation list can be uploaded to your project in two ways:

1. On the Project Creation Page: Under the heading “Title and Abstracts Upload”. Click on the “Choose File” button, select your RIS file and upload.
2. On the Project Management Page: Click on the “Upload Abstracts” button near the top of the page, select your RIS file, then press Submit.

Numbers of assessments: This is how you indicate how many assessments you want for each citation. For example, if you want your citations to be reviewed in duplicate, you would indicate “2”.

Single Retain Moves Forward: If this is turned on, a citation will move forward to the next screening stage once one reviewer has assessed the citation as “Retain”. Project leads might turn this function on in order to allow the screening process to be more sensitive to eligible studies, however, this might increase the amount of work required at the next screening level.

The Expert feature can be used to ensure that each citation is assessed by a team member that you have designated as an expert. See FAQ: How do I manage which tasks test members can perform? to find out how to designate team members as experts.

This feature allows you to sort retained citations into categories, and export a list of the retained citations in each category (see FAQ: How do I export my data?). To create categories, click on “Add More+” and then type the category title in the text box that appears. If you set up this feature, then during citation screening, reviewers will be prompted to select all categories in which a retained citation belongs. Please note that conflicts are not flagged for the Retain Allocation Category feature.

The Project Management Page is where you go to manage your project. To access this page, click on “Manage” beside your project title. From the Project Management page, you can edit your project, create a Test Set, download Test Set data, invite team members to join your project, upload citation lists, email your team, manage individual citations, manage user permissions, review statistics about each team member, review who has started and completed the Test Set, review Test Set results for each reviewer, and approve reviewers onto the project following Test Set completion.

Yes, you can set up a Test Set of citations to assess a potential reviewer’s ability to perform citation screening before granting them access to the project. Usually 50-100 citations are enough for a Test Set. It is also important to include True Positives (i.e. fully eligible citations) in your Test Set. We recommend including at least 5-10 True Positives. Once your project is set up, you can set up your Test Set as follows:

1. Click on “Manage” beside your project title. Then click on “Edit Test Set”
2. On the Edit Test Set page, assess the citations that you wish to include in your Test Set. Each citation that you assess on this page will be added to the Test Set. The assessment that you enter is considered the correct response. When potential reviewers take the Test Set, they will receive feedback on their responses based on the answers you have entered here.
3. You can use the search bar at the top of the screen if you have specific abstracts you wish to include in your Test Set. Make sure to adjust the dropdown menu for the search bar as required.
4. When you are done assessing the citations you wish to include in your Test Set, click on “Close” (top right of screen).

You will receive an email notification when someone completes the Test Set. To review their results, click on “Manage”. Then click on “Reviewers Applying to the Project”. The results for each reviewer are presented as True Negative (TN), True Positive (TP), False Negative (FN), False Positive (FP). This is based on the assessments that you entered when you created the Test Set. For example, if the reviewer retained a citation that you had also retained when you created the Test Set, this would be classified as a True Positive. If a reviewer excluded a citation that you also excluded when you created the Test Set, this would be classified as a True Negative. If the reviewer retained a citation that you had excluded when you created the Test Set, this would be classified as a False Positive. If the reviewer excluded a citation that you had retained when you created the Test Set, this would be classified as a False Negative.

If you are satisfied with the reviewer’s performance on the Test Set, and would like to give them access to the project, select “Yes” and then press “Submit”. The reviewer will receive an email notification that they have been granted access to the project. Note: You will also need to adjust the reviewers Access Permissions before they can start screening. See FAQ: How do I manage which tasks test members can perform? to find out learn how to manage Access Permissions.

To see Test Set results for each individual citation, click on “Edit Test Set”. Here you can download the Test Set data in a CSV file. Data is presented by reviewer for each citation.  You can also use the TICC (Test Set Identification of Challenging Citations) function to identify citations that were particularly challenging. See FAQ: What is TICC?

TICC stands for Test Set Identification of Challenging Citations. From the Edit Test Set page, click on the “TICC” button. You will see a list of citations in the Test Set, along with the answer you entered during Test Set creation (PI’s Selection) and the number of times that crowd members assessed that citation as Retain or Exclude. If the majority of the crowd members’ assessments were different from the PI’s Selection, we suggest that you review this citation to ensure that the assessment you entered during Test Set creation is correct. If, after review, you decide that the assessment you entered during Test Set creation was incorrect, you can use the Citation Management button to reset the citation. See FAQ: How do I manage (reset, delete assessments) for individual citations?

You can bring your own team rather than using the InsightScope crowd. During project set up, set your project to “Private”. This way it will not be visible to the crowd. Once your project is set up, you can invite your team members to join the project (see FAQs: How can I invite people to take the Test Set? and How can I give someone access to my project without taking the Test Set?)

Click on “Manage”. On the Project Management Page, click on “Invite Reviewers”. Make sure the radio button for “Invite to Test Set” is selected. Then enter the email address of the person whom you wish to invite. They will receive an email with an invitation link to the Test Set.

Click on “Manage”. On the Project Management Page, click on “Invite Reviewers”. Make sure the radio button for “Invite to Title & Abstract” is selected. Then enter the email address of the person whom you wish to invite. They will receive an email with an invitation link to the project. Note: Once they accept the invitation, they will be given access to Title & Abstract screening only. You can adjust their Access Permissions to allow them to perform additional project tasks (see FAQ: How do I manage which tasks team members can perform?).

Click on “Manage”. On the Project Management Page, click on “Access Permissions”. You can elect to give each team member access to the following project tasks:

1. PI – Designates the team member as a Principal Investigator. This will allow them to access the Project Management and Edit Project Page, and to access all project tasks.
2. Edit Test Set – Gives the team member access to the Edit Test Set page so that they can create or edit the Test Set, and download the Test Set data
3. Title/Abstract – Gives the team member access to Title & Abstract Screening
4. Upload – Gives the team member access to Document Upload
5. Full Text – Gives the team member access to Full Text Screening
6. DA – Gives the team member access to Data Abstraction (DA)
7. Conflict – Gives the team member access to Conflict Resolution
8. Expert – Designates the team member as an “Expert”. If the Expert feature is turned on during project set up, each citation will need to receive at least one assessment from a team member who is designated as an expert.
9. Disable – Removes the team member’s project access, but does not delete any citation assessment they have performed
10. Delete – Deletes the team member from the project, including any assessments they have performed

Click on “Manage”. Click on “Citation Management”. You will see a list of each citation in your project, along with the citation’s current status. From here, you can reset a citation back to a specific screening level. You can use the search bar at the top of the page to search for a specific citation. Make sure to adjust the dropdown menu on the search bar as required.

Click on the “Project Data” icon. Here you will see an overview of each screening level, including: total citations, total number citations that have been fully assessed, number of citations that have been retained or excluded, number of citations that are pending, and number of conflicts.

You will be notified of conflicts on your project in two ways: (1) When you login to insightScope, you will see a number under the Conflicts column informing you of the number of conflicts you have to resolve. (2) You will receive a daily email notification when there are pending conflicts that require resolution.

To resolve conflicts, click on the number in the Conflicts column. This will take you to the Conflict Resolution page. Conflicts are presented by screening level (use the tabs at the top of the page to navigate between levels). To resolve, click on a citation. You will see the citation information, along with each reviewer’s assessment. Enter the final assessment for the citation and press “Submit”.

Click on the “Project Data” icon. In the Download Project Output section of this page, click the button for the type of data that you wish to export.

1. Project Output – Full Text: List of all citations at the Full Text screening level, with the following information for each citation: reviewer assessments, whether or not there was a conflict, final paper status
2. Project Output – Abstract and Title: List of all citations at the Title and Abstract screening level, with the following information for each citation: reviewer assessments, whether or not there was a conflict, final paper status
3. PRISMA Outcome – Full Text: Count of exclusion reasons during Full Text screening for your PRISMA diagram
4. PRISMA Outcome – Abstract and Title: Count of exclusion reasons during Title and Abstract screening for your PRISMA diagram
5. Extract Retain Data: List of citations in each Retain Allocation Category (see FAQ: What is Retain Allocation Category?)
6. Download All PDF: A link to a Dropbox folder with all PDFs that have been uploaded to your project. PDFs are renamed as follows: InsightScope ID Number_First Author Last Name_Publication Year.
7. Project Overview: Number of citations (total, retained, excluded) and Kappa for each screening level
8. Full Text Retained Papers: RIS file of the citations that were retained during Full Text Screening

Each project on insightScope has a leaderboard. You can access the leaderboard by logging in and clicking on Leaderboard in the website header. The leaderboard lists reviewers in order by their insightScope score. For each reviewer, it also shows the number of citations they have assessed during Title and Abstract and during Full Text screening.

The insightScope score considers both the amount of work the reviewer has performed, as well as their accuracy.

Please note that the leaderboard is updated at the end of each calendar day, so the work that you or your reviewers do will not be reflected in the leaderboard data until the following calendar day.