Frequently Asked Questions
Answers for Principal Investigators
- Project title
- A brief description of the systematic review (project objectives) and investigative team
- Prospero registration link (if available)
- RIS file of your citation list
- Inclusion and exclusion criteria for abstract and full-text screening
- On your project page, you may also wish to include: (i) a description of the PI and research team; (ii) tips and advice for reviewers; (iii) whether you are offering any incentive(s) to reviewers; and (iv) upload the systematic review protocol and search strategy
To upload your citation list to the project, you will need an RIS file. This is an export option on reference manager software. The citation list can be uploaded to your project in two ways:
- On the Project Creation Page: Under the heading “Title and Abstracts Upload”. Click on the “Choose File” button, select your RIS file and upload.
- On the Project Management Page: Click on the “Upload Abstracts” button near the top of the page, select your RIS file, then press Submit.
Numbers of assessments: This is how you indicate how many assessments you want for each citation. For example, if you want your citations to be reviewed in duplicate, you would indicate “2”.
Single Retain Moves Forward: If this is turned on, a citation will move forward to the next screening stage once one reviewer has assessed the citation as “Retain”. Project leads might turn this function on in order to allow the screening process to be more sensitive to eligible studies, however, this might increase the amount of work required at the next screening level.
The Expert feature can be used to ensure that each citation is assessed by a team member that you have designated as an expert. See FAQ: How do I manage which tasks test members can perform? to find out how to designate team members as experts.
This feature allows you to sort retained citations into categories, and export a list of the retained citations in each category (see FAQ: How do I export my data?). To create categories, click on “Add More+” and then type the category title in the text box that appears. If you set up this feature, then during citation screening, reviewers will be prompted to select all categories in which a retained citation belongs. Please note that conflicts are not flagged for the Retain Allocation Category feature.
The Project Management Page is where you go to manage your project. To access this page, click on “Manage” beside your project title. From the Project Management page, you can edit your project, create a Test Set, download Test Set data, invite team members to join your project, upload citation lists, email your team, manage individual citations, manage user permissions, review statistics about each team member, review who has started and completed the Test Set, review Test Set results for each reviewer, and approve reviewers onto the project following Test Set completion.
Yes, you can set up a Test Set of citations to assess a potential reviewer’s ability to perform citation screening before granting them access to the project. Usually 50-100 citations are enough for a Test Set. It is also important to include True Positives (i.e. fully eligible citations) in your Test Set. We recommend including at least 5-10 True Positives. Once your project is set up, you can set up your Test Set as follows:
- Click on “Manage” beside your project title. Then click on “Edit Test Set”
- On the Edit Test Set page, assess the citations that you wish to include in your Test Set. Each citation that you assess on this page will be added to the Test Set. The assessment that you enter is considered the correct response. When potential reviewers take the Test Set, they will receive feedback on their responses based on the answers you have entered here.
- You can use the search bar at the top of the screen if you have specific abstracts you wish to include in your Test Set. Make sure to adjust the dropdown menu for the search bar as required.
- When you are done assessing the citations you wish to include in your Test Set, click on “Close” (top right of screen).
You will receive an email notification when someone completes the Test Set. To review their results, click on “Manage”. Then click on “Reviewers Applying to the Project”. The results for each reviewer are presented as True Negative (TN), True Positive (TP), False Negative (FN), False Positive (FP). This is based on the assessments that you entered when you created the Test Set. For example, if the reviewer retained a citation that you had also retained when you created the Test Set, this would be classified as a True Positive. If a reviewer excluded a citation that you also excluded when you created the Test Set, this would be classified as a True Negative. If the reviewer retained a citation that you had excluded when you created the Test Set, this would be classified as a False Positive. If the reviewer excluded a citation that you had retained when you created the Test Set, this would be classified as a False Negative.
If you are satisfied with the reviewer’s performance on the Test Set, and would like to give them access to the project, select “Yes” and then press “Submit”. The reviewer will receive an email notification that they have been granted access to the project. Note: You will also need to adjust the reviewers Access Permissions before they can start screening. See FAQ: How do I manage which tasks test members can perform? to find out learn how to manage Access Permissions.
To see Test Set results for each individual citation, click on “Edit Test Set”. Here you can download the Test Set data in a CSV file. Data is presented by reviewer for each citation. You can also use the TICC (Test Set Identification of Challenging Citations) function to identify citations that were particularly challenging. See FAQ: What is TICC?
TICC stands for Test Set Identification of Challenging Citations. From the Edit Test Set page, click on the “TICC” button. You will see a list of citations in the Test Set, along with the answer you entered during Test Set creation (PI’s Selection) and the number of times that crowd members assessed that citation as Retain or Exclude. If the majority of the crowd members’ assessments were different from the PI’s Selection, we suggest that you review this citation to ensure that the assessment you entered during Test Set creation is correct. If, after review, you decide that the assessment you entered during Test Set creation was incorrect, you can use the Citation Management button to reset the citation. See FAQ: How do I manage (reset, delete assessments) for individual citations?
You can bring your own team rather than using the InsightScope crowd. During project set up, set your project to “Private”. This way it will not be visible to the crowd. Once your project is set up, you can invite your team members to join the project (see FAQs: How can I invite people to take the Test Set? and How can I give someone access to my project without taking the Test Set?)
Click on “Manage”. On the Project Management Page, click on “Invite Reviewers”. Make sure the radio button for “Invite to Test Set” is selected. Then enter the email address of the person whom you wish to invite. They will receive an email with an invitation link to the Test Set.
Click on “Manage”. On the Project Management Page, click on “Invite Reviewers”. Make sure the radio button for “Invite to Title & Abstract” is selected. Then enter the email address of the person whom you wish to invite. They will receive an email with an invitation link to the project. Note: Once they accept the invitation, they will be given access to Title & Abstract screening only. You can adjust their Access Permissions to allow them to perform additional project tasks (see FAQ: How do I manage which tasks team members can perform?).
Click on “Manage”. On the Project Management Page, click on “Access Permissions”. You can elect to give each team member access to the following project tasks:
- PI – Designates the team member as a Principal Investigator. This will allow them to access the Project Management and Edit Project Page, and to access all project tasks.
- Edit Test Set – Gives the team member access to the Edit Test Set page so that they can create or edit the Test Set, and download the Test Set data
- Title/Abstract – Gives the team member access to Title & Abstract Screening
- Upload – Gives the team member access to Document Upload
- Full Text – Gives the team member access to Full Text Screening
- DA – Gives the team member access to Data Abstraction (DA)
- Conflict – Gives the team member access to Conflict Resolution
- Expert – Designates the team member as an “Expert”. If the Expert feature is turned on during project set up, each citation will need to receive at least one assessment from a team member who is designated as an expert.
- Disable – Removes the team member’s project access, but does not delete any citation assessment they have performed
- Delete – Deletes the team member from the project, including any assessments they have performed
Click on “Manage”. Click on “Citation Management”. You will see a list of each citation in your project, along with the citation’s current status. From here, you can reset a citation back to a specific screening level. You can use the search bar at the top of the page to search for a specific citation. Make sure to adjust the dropdown menu on the search bar as required.
Click on the “Project Data” icon. Here you will see an overview of each screening level, including: total citations, total number citations that have been fully assessed, number of citations that have been retained or excluded, number of citations that are pending, and number of conflicts.
You will be notified of conflicts on your project in two ways: (1) When you login to insightScope, you will see a number under the Conflicts column informing you of the number of conflicts you have to resolve. (2) You will receive a daily email notification when there are pending conflicts that require resolution.
To resolve conflicts, click on the number in the Conflicts column. This will take you to the Conflict Resolution page. Conflicts are presented by screening level (use the tabs at the top of the page to navigate between levels). To resolve, click on a citation. You will see the citation information, along with each reviewer’s assessment. Enter the final assessment for the citation and press “Submit”.
Click on the “Project Data” icon. In the Download Project Output section of this page, click the button for the type of data that you wish to export.
- Project Output – Full Text: List of all citations at the Full Text screening level, with the following information for each citation: reviewer assessments, whether or not there was a conflict, final paper status
- Project Output – Abstract and Title: List of all citations at the Title and Abstract screening level, with the following information for each citation: reviewer assessments, whether or not there was a conflict, final paper status
- PRISMA Outcome – Full Text: Count of exclusion reasons during Full Text screening for your PRISMA diagram
- PRISMA Outcome – Abstract and Title: Count of exclusion reasons during Title and Abstract screening for your PRISMA diagram
- Extract Retain Data: List of citations in each Retain Allocation Category (see FAQ: What is Retain Allocation Category?)
- Download All PDF: A link to a Dropbox folder with all PDFs that have been uploaded to your project. PDFs are renamed as follows: InsightScope ID Number_First Author Last Name_Publication Year.
- Project Overview: Number of citations (total, retained, excluded) and Kappa for each screening level
- Full Text Retained Papers: RIS file of the citations that were retained during Full Text Screening
Each project on insightScope has a leaderboard. You can access the leaderboard by logging in and clicking on Leaderboard in the website header. The leaderboard lists reviewers in order by their insightScope score. For each reviewer, it also shows the number of citations they have assessed during Title and Abstract and during Full Text screening.
The insightScope score considers both the amount of work the reviewer has performed, as well as their accuracy.
Please note that the leaderboard is updated at the end of each calendar day, so the work that you or your reviewers do will not be reflected in the leaderboard data until the following calendar day.
Answers for Reviewers
Click on the Sign-Up button and you will be prompted to enter your name, email address and to set up a username and password. Once you have submitted your information, you will receive an email notification. You must confirm your request within the email notification before you are able to log in to the website. Once you set up your login information, you will also be directed to set up your profile.
PI stands for Principal Investigator. This is the individual (or individuals) who are leading the project.
The PI of the project may set up a Test Set to evaluate reviewers before giving them access to their review. The Test Set will consist of a subset of the project citations and will give you a chance to evaluate the citations using the screening criteria. After you assess each citation in the Test Set, you will see how your assessment compares to the PI’s assessment.
Once you login to the website, you will see a “Join Projects” button near the top of the screen. Clicking on this button will bring up a list of existing projects. Clicking on the “Details” button will bring you to the project homepage where you can read details about the systematic review and decide if it is a project you are interested in joining. You can apply to be a reviewer on a project by completing a Test Set (select “Start Test Set” at the bottom of the project’s home page).
Before starting to assess citations, it is recommended that you read over all of the systematic review details on the project home page. Depending on what is provided by the PI, you might be able to see an overall study goal, information about the PI and study team, comments and instructions for reviewers, the systematic review abstract, the full systematic review protocol and/or a Prospero link providing further details on the study.
Depending on the systematic review, there are the 5 stages:
- Test Set: Reviewers complete screening on a subset of citations
- Title and Abstract: Includes screening only the abstract for specific criteria.
- Document Upload: The full text for all citations that were retained during Title and Abstract screening are retrieved and uploaded to the system
- Full-Text Review: Includes screening the complete article (PDF) for specific criteria
- Data Abstraction: Includes abstracting specific data elements from paper that was retained during Full Text screening
Once you have been granted access to a project by a PI, the project will appear on your dashboard when you first log into the insightScope website.
From here, select “Title and Abstract” to work on Title and Abstract screening. Click on “Document Upload” to find and upload the full text document for citations that are retained during Title and Abstract screening. Click on “Full Text Review” to work on Full Text screening.
It is recommended to start screening citations from the top of the table one by one. Citations are sorted to show the ones with the least input first, to make sure all get equal assessments.
Using the “Upload Documents” button on the Dashboard, team members can attach the PDFs files to the citations one by one. Once a PDF is uploaded, the citation will move to the Full Text screening stage, and will be accessible to all team members who have been given Full Text screening privileges. If you are unable to locate a PDF, you can flag the PDF for “Later”. This will move the PDF to the “Later” tab on the Upload Documents page until such time that you are able to locate that PDF.
Only your username will be shown, your full name will only appear to PIs.
This is completely between you and the PI. Some PIs might offer monetary compensation for your work, while other systematic reviews are done on a voluntary basis. In that case, the PI may provide you with a reference letter outlining your involvement in the project. Sometimes the PI will indicate what they are offering to reviewers on the project’s homepage.
If you make a significant contribution during citation screening and are involved in other parts of the research project (e.g. Analysis, writing), you might be given authorship on the project publication.
Once citation screening and data abstraction is complete, there is still a lot of work for the investigative team to complete before the systematic review can be submitted for publication. They need to resolve conflicts, clean the data, analyze the data and prepare the manuscript. This process can take several months, depending on the complexity of the analysis and the competing workload of the investigative team members.
Each project on insightScope has a leaderboard. You can access the leaderboard by logging in and clicking on Leaderboard in the website header. The leaderboard lists reviewers in order by their insightScope score. For each reviewer, it also shows the number of citations they have assessed during Title and Abstract and during Full Text screening.
The insightScope score considers both the amount of work the reviewer has performed, as well as their accuracy.
Please note that the leaderboard is updated at the end of each calendar day, so the work that
Click on the arrow in the top right of the screen, and select View Public Profile. Here you can see your insightScope score (see FAQ: What is the Project Leaderboard and how do I access it? What is the insightScope score?), and the number of citations that you have
Contact Us
Contact us if you have any questions about the insightScope platform that are not answered above.
Email: info@insightscope.ca
Phone: 613-808-4071
Address:
insightScope
810 Walkley Road
Ottawa, ON K1V 6R7
Canada