diff --git a/Import/Shoulder_Database_Importation_Protocol_SOP.md b/Import/Shoulder_Database_Importation_Protocol_SOP.md index 0f317a2..612307e 100644 --- a/Import/Shoulder_Database_Importation_Protocol_SOP.md +++ b/Import/Shoulder_Database_Importation_Protocol_SOP.md @@ -1,240 +1,250 @@ # Shoulder_Database_Importation_Protocol ## Preliminary notes -- The **source directory** is *lbovenus.epfl.ch:/home/shoulder* -- The main Matlab script is *importSCase.m* and should be executed from *shoulder/method/database/Import* (**this directory will change to** ***shoulder/method/database/ImportSCases***) -- a second script needed to -- The **data directory** is *shoulder/data* -- A shoulder case is called ***SCase***. - - Each patient may have been operated on one ore both shoulders, eventually twice on the same shoulder. Therefore, the same patient might appear in two, three, or even four entries of the database. - - Each entry of the database is a different SCase, and uniquely identified by a SCaseID. which refers to the set of CTs from the same patient and shoulder - - Each pending patient folder should be considered as a different SCase. When more than one set of CTs are present for a single SCase, we might find different types of CT images: scout-scans, shoulder CTs, elbow CTs, thorax CTs, opposed shoulder CTs, etc. Not all of them must be imported, however this will be explained later in this document (more information in section *CT types classification*). - - Each *SCase* study is focused in a single shoulder. Here forward we will refer to this shoulder as the **shoulder of interest** or **shoulder side**. -- DICOM (Digital Imaging and Communications in Medicine) is a worldwide-accepted standard for medical images exchange, managed by the Medical Imaging & Technology Alliance – a division of the National Electrical Manufacturers Association (*https://www.dicomstandard.org/current/*). -- The anonymization consists in removing some information from the metadata of the DICOM files that can identify the patient, such as its name or birth date. This can automatically done by the Matlab script or manually with Amira. -- The Matlab interface for anonymization and importation is explained in the second section of this document. We will refer to the numbered steps as step 1, step 2, etc. +1. DICOM (Digital Imaging and Communications in Medicine) is a worldwide-accepted standard for medical images exchange, managed by the Medical Imaging & Technology Alliance – a division of the National Electrical Manufacturers Association (*https://www.dicomstandard.org/current/*). +2. The **source directory** is *lbovenus.epfl.ch:/home/shoulder* +3. Matlab Scripts: + - The main Matlab script is *importSCase.m* and should be executed from *shoulder/method/database/Import* (**this directory will change to** ***shoulder/method/database/ImportSCases***) + - An other script needed at the end of the process is *dicominfoSCase.m*, located in ***shoulder/method/database/***. This script updates the CSV and XLS files *CTdicomInfo*, located in ***shoulder/data/Excel/xlsFromMatlab/***, which are readed by the main Excel ***ShoulderDataBase.xlsx*** +4. The **data directory** is *shoulder/data* +5. This shoulder's CT database contains both normal and pathological clinical cases. Depending on their content, we classify them as is explained in the section **CT types classification** +6. A shoulder case is called ***SCase***. + - Each patient within the pathological group may have been operated on one ore both shoulders, eventually twice on the same shoulder. Therefore, the same patient might appear in two, three, or even four entries of the database. + - Each entry of the database is a different treatment or surgery and consequently a different SCase uniquely identified by a SCaseID. Note that one single SCase may refer to several CT's set from the same patient and shoulder but not from different treatment or surgeries. +7. Importation + - Each patient folder pending to be imported should be initially considered as a different SCase. When more than one set of CTs are present for a single SCase, we might find different types of CT images: scout-scans, shoulder CTs, elbow CTs, thorax CTs, opposed shoulder CTs, etc. Not all of them must be imported, however this will be explained later in this document (more information in section *CT types classification*). + - The anonymization consists in removing some information from the metadata of the DICOM files that can identify the patient, such as its name or birth date. This can automatically done by the Matlab script or manually with Amira. +8. The Matlab interface for anonymization and importation is explained in the section **SOP** of this document. ## CT types classification - The CTs in the database are classify as explained here: - Group of patients: - - **Normal**: CTs from patients without shoulder pathology - - **Pathological**: CTs from patient with shoulder pathology + - **Normal (N)**: CTs from patients without shoulder pathology. Those CT scans are often acquired in order to find multiple traumas after an accident, so they might contain a portion of the patient body, much larger than the area of interest of this database. + - **Pathological (P)**: CTs from patient with shoulder pathology. Those CT scans are aimed to analyze the shoulder anatomy and can contain several CT sets with different acquisition parameters. - Acquisition stage: - - **Preop** is the name we give to the CTs from any patient shoulder before the first shoulder replacement surgery. This means that only CTs without any shoulder prostheses on the shoulder of interest can be classify as *preop*. - - **Postop** is the name we give to the CTs from any patient shoulder that has already been under shoulder replacement surgery. This means that every CT with a shoulder prostheses on the shoulder of interest should be considered as *postop* (even in the eventual case of a shoulder needing a new surgery as a prostheses replacement). + - **Preop** CT sets acquired before the shoulder joint replacement surgery either for diagnostic or surgery planning purposes. Every CT set without prostheses in the shoulder of interest must be classified as *preop*. + - **Postop** The CT sets acquired after the surgery aiming to validate the prostheses placement (just after surgery) or checking their state (some time after the surgery). This means that every CT with a shoulder prostheses on the shoulder of interest should be initially considered as *postop* (unless the responsible clinician informs explicitly about the eventual case of a prostheses replacement). - Body part: - - **Shoulder** is the tag for every CT containing the shoulder of interest. + - **Shoulder** is the tag for every CT containing (at least) the shoulder of interest. - **Elbow** is the tag for CTs containing elbow without shoulder. - **Contralateral** of *Opposed Shoulder* is the tag for CTs with the opposed shoulder to the shoulder of interest. - - **Thorax** is the tag for the Cts containing the patient thorax without any complete shoulder. + - **Thorax** is the tag for the CTs containing the patient thorax without any complete shoulder. - **Other** is the tag for any other body. - Phantoms (**with** or **without**): - These are samples of synthetic bone with accurate and homogeneous mineral density and cylindrical shape. Phantoms are typically placed behind the back of the patient and are used to calibrate the measurement of the bone mineral density. At least 2 phantoms are needed for an accurate calibration. - The following table summarize CT classification adding a **CT series label** as a numerical value for each enumerated class (column "#") | Acquisition | Body | Phantoms | Kernel | # | | - | - | - | - | - | | Preop | Shoulder | Without | Sharp | 1 | | Preop | Shoulder | Without | Smooth | 2 | | Preop | Shoulder | With | Sharp | 3 | | Preop | Shoulder | With | Smooth | 4 | | Preop | Elbow | - | Sharp | 5 | | Preop | Elbow | - | Smooth | 6 | | Preop | Thorax | - | Sharp | 7 | | Preop | Thorax | - | Smooth | 8 | | Preop | Opposed Shoulder | - | Sharp | 9 | | Preop | Opposed Shoulder | - | Smooth | 10 | | Preop | Other | - | | 11 | | - | - | - | - | - | | Postop | Shoulder | Without | Sharp | p1 | | Postop | Shoulder | Without | Smooth | p2 | | Postop | Shoulder | With | Sharp | p3 | | Postop | Shoulder | With | Smooth | p4 | | Postop | Elbow | - | Sharp | p5 | | Postop | Elbow | - | Smooth | p6 | | Postop | Thorax | - | Sharp | p7 | | Postop | Thorax | - | Smooth | p8 | | Postop | Opposed Shoulder | - | Sharp | p9 | | Postop | Opposed Shoulder | - | Smooth | p10 | | Postop | Other | - | | p11 | ## SOP -### 1. CT files receiving and storing before importation -- Extract the archive (.zip) file sent by CHUV and save the unzipped folder in the folder */data/[X]/pending/TSA-update-[YYYY-MM-DD]*, where: - - [X] must be: - - N is for normal shoulders - - or P is for pathological shoulders - - [YYYY-MM-DD] is the date you received the archive. +### CT files receiving and storing before importation +- Extract the archive (.zip) file sent by CHUV and save the unzipped folder in the folder: + - */data/N/pending/N-update-[YYYY-MM-DD]* for SCases of group *N* + - */data/P/pending/TSA-update-[YYYY-MM-DD]* for SCases of group *P* + - where [YYYY-MM-DD] is the date you received the archive. - Inside this main folder, each patient folder: - is named with the *Patient Permanent Identifier*, **IPP** (*Identifiant Permanent du Patient*). - contains at least one sub-folder with a CT reconstruction set of DICOM images. - Those DICOM files are the set of CT images that we will be loaded into the MATLAB interface for importation and anonymization. - General example: */data/P/pending/TSA-update-2018-07-26* -### 2. MATLAB interface for SCase anonymization and importation +### MATLAB interface for SCase anonymization and importation +In this subsection, we will refer to the numbered paragraphs as *step1*, *step2*, etc. 1. Run the Matlab function ***importSCase.m*** located in *shoulder/methods/matlab/database/Import* (**this directory will change to** ***shoulder/method/database/ImportSCases***) 2. Select ***Normal*** or ***Pathologic*** in *Patient Group*, depending on the type of CT that you are about to import. - - If you choose *Pathologic* a message window will be prompted, asking you whether to activate or not the 3D visualization. The recommendation is to avoid the 3D viewer activation at this step, because it may be activated later. For more information read the step 5 of this section. + - If you choose *Pathologic*, a message window will be prompted, asking you whether to activate or not the 3D visualization. The recommendation is to avoid the 3D viewer activation at this step, because it may be activated later. For more information read the *step5* of this section. -3. If you are importing a new patient set of CT's, click on button ***Start new Patient/SCase*** and select the patient folder of your interest. Otherwise, you might skip this step. +3. If you are importing a new patient set of CT's (first CT of a new pending SCase), click on button ***Start new Patient/SCase*** and select the patient folder of your interest. Otherwise, you might skip this step. - IMPORTANT: select the folder that contains the sub-folders with the CTs. For example: - The pending patient directory *lbovenus.epfl.ch:/shoulder/data/P/pending/TSA-update-2017-08-18/97212* contains: - *DICOMDIR* (file) - *LOCKFILE* (file) - *VERSION* (file) - *24NZUR5U* **(Folder to be selected in this step)** - - This folder contains the *sub-folders* *CO1ALS1N*, *CO02LW5N*, *FFNFXVOY*, *FMNFXVOY* ans *RMEJ0LXY*. Each one of which contains a CT series. - - This step: - - finds the number of slices present in each of the CTs of the new patient. - - adds a prefix to the CTs sub-folders with the number of slices contained so they can be prioritized depending in the CT size. - - adds a suffix that help discriminate between potential shoulder CTs, elbow CTs or scout CTs. - - -4. Click on ***Load next CT set*** to load a new CT. - - Navigate inside the patient folder until finding the set of sub-folders that contain the CTs. These sub-folders have now prefixes indicating the number of CT slices inside. Start loading the largest CT (usually shoulders are in the largest CTs) and continue by decreasing order. - - Select all DICOM files (not file “VERSION”) in the folder and click on button “Open”. - - A bar-progress window will be prompted showing the loading progress. Don't close it before the process is finished. - - The CT slices will appear in main viewer (2D) and, if *3D visualization* is activated, in the 3D viewer as well. - - The information box above the 2D viewer shows the current slice full-name. + - This folder contains the *sub-folders* *CO1ALS1N*, *CO02LW5N*, *FFNFXVOY*, *FMNFXVOY* ans *RMEJ0LXY*, each one of which contains a CT series. + - Information: This step... + - ... finds the number of slices present in each of the CTs of the new patient. + - ... adds a prefix to the CTs sub-folders with the number of slices contained so they can be prioritized depending in the CT size. + - ... adds a suffix that help discriminate between potential shoulder CTs, elbow CTs or scout CTs. + + +4. Click on ***Load next CT set*** to load a new CT set. + - Navigate inside the patient folder until finding the set of sub-folders that contain the CTs. These sub-folders have now prefixes indicating the number of CT slices inside. Start loading the largest CT (usually *shoulders* are in the largest CTs) and continue in decreasing order. + - Select all DICOM files (not file “VERSION”) in the folder and click on button *“Open”*. + - A bar-progress window will be prompted showing the loading progress. Don't close it before the process is finished, because this will stop the loading and you will need to start again this *step4*. + - The CT slices will appear in main viewer (2D) and (if *3D visualization* is activated) a 3D rendering will be shown in the 3D viewer as well. + - The information box above the 2D viewer shows the full-name of the current slice. - During loading, some types of errors might appear: - Window prompted with *Error identifier:* ***heterogenousStructAssignment***: - - This means that as least one of the selected slices has a different format, so they cannot be loaded as a single CT set. Although this happens frequently with a single image it is possible to find a few of them with different format. - - Try to identify the problematic images and then load again the CT set but skipping those files. + - This means that at least one of the selected slices has a different format, so they cannot be loaded as a single CT set. + - Try to identify the problematic images and then load again the CT set but skipping those files: - Start by checking in which image the loading progress bar has stopped, because this must be the file producing the error. - - Load the rest of the slices by selecting them all except the problematic one. + - Load the rest of the DICOM images by selecting all of them except the problematic one. - If the problem happens again it means that there are several DICOM files with different format. Hence, repeat the process a few times to avoid all the problematic files. - - An alternative could be opening the full CT set with another software like Amira, which is able to differentiate the problematic slices. Then, saving the set of correct images in DICOM format within a different folder should allow further loading of the new set without the heterogeneous error. - - However this process might divide the CT set in two or more parts (when the problematic file is neither the first nor the last). + - Alternatively, open the full CT set with another software like Amira, which is able to differentiate the problematic slices. Then, saving the set of correct images in DICOM format within a different folder should allow further loading of the new set without the *heterogeneous error*. + - This process might divide the CT set in two or more parts (when the problematic file is neither the first nor the last). - If the error is still present it might be necessary to use Amira to import this CT set, but it could be also due to an error made in the reconstruction step. - Check if it opens correctly in Amira but if it creates several series of images and you cannot identify any of them as relevant it might be better to avoid the CT set. Eventually, shall the CT provider be informed. - Do not import unusual CT image formats (atypical orientation, slices from a 3D rendering, images with superposed grid or measurements, etc). - - Some of these unusual formats are treated by the script in such a way that the containing folder will be automatically renamed adding the suffix wrongOrientation. The user just needs to avoid these sets and continue loading the following one. - - The Matlab script cannot read the DICOM file metadata. - - The alternative is to import CT with **Amira (described later in this document)**. + - Some of these unusual formats are treated by the script in such a way that the containing folder will be automatically renamed adding the suffix *wrongOrientation*. The user just needs to avoid these sets and continue loading the next one. + - If the Matlab script cannot read the DICOM file metadata + - The alternative is to import CT with **Amira (described later in this document)**. 5. (optional) If needed, activate the 3D viewer. - The 3D visualization is useful for shoulder side identification of Pathological series, but rarely needed in Normal patient because Normal CTs usually contain both shoulders. - WARNING: when the CT are very large (over 400 slices) and the computer has a limited memory the loading might slow down or even stop the script. - When activated it shows a *volume rendering* of the CTs. 6. Select ***Left*** or ***Right*** in the radio-buttons group *Shoulder Side*. - The side selected must match with the *shoulder of interest* and not with the *contralateral shoulder* if this is present. (For more information read *Preliminary notes* and *CT types classification* above in the current document) - - For Normal CTs were both shoulder are present, select *Right* shoulder. + - For Normal CTs were both shoulders are present in resting position, select *Both* shoulders (if the importation software doesn't have this option choose *Right* shoulder). 7. Select ***Preop*** or ***Postop*** in the radio-buttons group *Acquisition stage* 8. Click in button ***Auto Check DB*** to launch the database auto-check for IPP coincidences. - The results will be shown in the *DB check result* information box. - - Depending on the auto-check results, an *SCaseID* will be suggested. But if any problem rises, you might always do a manual selection of the *SCaseID*, as described in the next step. + - Depending on the auto-check results, a *SCaseID* will be suggested. But if any problem rises, you might always do a manual selection of the *SCaseID*, as described in the next *step9*. 9. (optional) If the *Suggested SCaseID* does not match with your expectations, you might choose manually the output directory by clicking in button ***Manual SCase***. - Example: *shoulder/data/P/5/5* -10. Verify whether the *Parent output directory* was updated correctly or modify it manually if needed. +10. Verify the *Parent output directory* was updated correctly or modify it manually if needed. - Example: *lbovenus.epfl.ch:/shoulder/data/P/5/5* 11. Choose between the different possibilities in the radio-buttons group *Body part*: - **Shoulder** must be chosen when at least one complete shoulder is present in the CT reconstruction. - **Elbow** must be chosen when the CT contains at least one elbow without any complete shoulder. - **Thorax** must be selected only when the main body in the CT is the *thorax* and no shoulder is present or at least not complete. - **Contralateral** must be selected when the only complete shoulder present in the CT is not the *shoulder of interest*. -12. In *CT Kernel (image filter)*, select ***Sharp (bone)*** when the 2D viewer shows a *sand like* texture for the CT slices or ***Smooth (soft tissue)*** the the image looks soft and smooth. +12. In *CT Kernel (image filter)*, select ***Sharp (bone)*** when the 2D viewer shows a *sand like* texture for the CT slices or ***Smooth (soft tissue)*** the image looks soft and smooth. + 13. Select ***With*** or ***Without*** in the radio-buttons group *Phantoms*. -14. (optional) When a SCase contains many CT sets, it might be very useful adding some extra information to the *README* file, so the the CTs can be differentiated easily. As examples: +14. (optional) When a SCase contains many CT sets, it might be very useful adding some extra information to the *README* file, so the CTs can be differentiated easily. As examples: - When both shoulders are present and complete --> add *both shoulders* - - When several CT series can be classified similarly in terms of body part or CT Kernel, it might be useful adding other CT data as: *number of slices*, *resolution* + - When several CT series can be classified similarly in terms of body part or CT Kernel, it might be useful adding other CT data. - This is also the place where the operator might add any comments -15. Check that all the options selected are correct and click on button **Import to Database**. +15. Check in the readme line below the 2D viewer that all the options selected are correct and click on button **Import to Database**. - If the imported CT is meant to be the first CT of a new SCase in the database, this will create a new folder in the directory indicated in the text field *Parent output directory* with the SCase identification number indicated in *Suggested SCaseID* and the IPP found in the metadata (and shown in the field *IPP*, in the *DICOM Metadata* box). Example: *shoulder/data/P/5/5/P555-918819*. - Inside the SCase folder, a new sub-folder will be created to store the imported CT images: - The name of this sub-folder contains the SCaseID, the IPP and the CT set numeric label. Example: ***CT-[SCaseID]-[IPP]-#*** --> ***CT-P555-229341-1*** - A new line will be added to the *README* line with the name of the new folder followed by some information like the *kernel*, the presence of phantoms, the main body in the CT, etc. - If the CT set label already exists in the SCase folder, then a new folder will be automatically created with *"xxx"* as CT set label. Consequently, the CT series label will need to be updated manually. And the corresponding line in the *README* file will need to be corrected. - The anonymized DICOM images will be finally stored inside the directory like ***shoulder/data/P/5/5/P555-918819/CT-P555-229341-1/DICOM***. -16. If the recently imported CT becomes a new SCase to the DB you still need to update manually the Excel file ***ShoulderDataBase.xlsx***, located in *//lbovenus.epfl.ch/shoulder/data/Excel/*. - - Check the SCaseID of the last entry in the database for your case type (N or P) and increment by 1 to get the current SCaseID (g.e. if the last entry in database is *P284*, current SCaseID will be *P285*”). - - Add a new row to ShoulderDatabase.xlsx with the new SCaseID in the first column. This automatically add a new row to the table, and copy formulas. Then: +16. If the recently imported CT set was a new SCase, the Excel Database need to be updated. +In that case, click on button *Add new SCase to the Excel DB*. This automatically updates the Excel Database with almost all the DICOM information from the imported SCase. However, this parameter (SCaseID) have to be manually added to the Excel Database: + - Open the file ***ShoulderDataBase.xlsx***, located in *//lbovenus.epfl.ch/shoulder/data/Excel/*. + - the SCaseID of the last entry in the database for your case type (N or P) and increment by 1 to get the current SCaseID (g.e. if the last entry in database is *P284*, current SCaseID will be *P285*”). + - Scroll down to find the last SCaseID (last row) and add a new row by taping the new SCaseID in the cell below the last SCaseID (first column). This automatically add a new row to the table, and copy formulas. Then: - In the second column introduce the row number by simply increasing the number from the immediately superior row. - - From the *DICOM Metadata box* copy the following data in their corresponding columns of the excel file ***ShoulderDataBase.xlsx***: + - From the *DICOM Metadata box* check that the following data are correctly in their corresponding columns of the excel file ***ShoulderDataBase.xlsx***: - IPP - Patient Birthday - Patient Gender - Patient Initials - - In the step6, you should have used 2D viewer and 3D viewer (if needed) to figure out which is the anatomical side of the *shoulder of interest*. Accordingly, add now *R* (for Right) or *L* (for Left) to the column *shoulder_side*. + - Check also the shoulder side column *shoulder_side* + - Check that the rest of the columns are filled with data as in the precedents rows. If you find some columns with *NaN* values: + - close the Excel File without saving, + - try to run manually the script *dicominfoSCase.m* without arguments, which is located in *shoulder/methods/matlab/database*. This script updates the new row in the Database with metadata from the imported CT. + - try to solve the errors if some arise + - open the Excel file and start this *step16* again. - Save and close the Excel file. - Run the script ***dicominfoSCase.m***, which is located in *shoulder/methods/matlab/database*. This script updates the new row in the Database with metadata from the imported CT. Once the script has finished open the Excel Database again and save it. -17. If the patient folder contains more CT sets to import, continue from step4. Otherwise, start a new patient folder and continue from step3. +17. If the patient pending folder contains more CT sets to import, continue from *step4*. Otherwise, start a new patient folder and continue from *step3*. ## Alternative SOP using AMIRA ### %Might be simplified by just saving in a correct format,usable by matlab% - When the CT has a non-uniform resolution, the (above) Matlab protocol may not work, but can be replaced by the following Amira protocol. 1. Start Amira and click on *Open Data…* (Ctrl + O). Navigate to *shoulder/data/[X]/pending/[IPP]*. Inside the CT folder, there might be several reconstruction folders. 2. Load the images of the CT folder that you could not import with the Matlab script - Select all files except the file named *VERSION* (if it exists). 3. Check the parameter *series description*, in the prompted window, to know the kernel used for this set in *series description*. Usually it is *OS* for **Sharp** and *STD* for **Smooth**. Click on OK. 4. Attach an *Ortho Slice* or *Slice* display object to the newly added green data modules to visualize the content of the image stacks. 5. Open Excel shoulder database *shoulder/data/Excel/ShoulderDatabase.xlsx* database (*SCase* tab) and compare the IPP, CT date, shoulder side and DICOM information to determine if the new case is already in the database. - If the IPP already exits in the database: - Find the *Readme* file in the SCase folder of the existing SCase and compare the data inside with the metadata of the current CT to decide whether the current CT set already exist in the SCase folder or it have to be imported. - If it does not exist continue with step6. 6. Right-click on the first data module and click on *Export Data As…* and choose type *DICOM*. 8. Go to the correct folder *shoulder/data/[X]/[#1]/[#2]/[CaseID]-[IPP]* , where *#1* and *#2* are the first two digits following ‘P’ or ‘N’ in the current SCaseID. For example: *shoulder/data/P/0/1* for SCase *P13*. If folder [CaseID]-[IPP] does not exist yet, create it. Then create the folder *CT-[CaseID]-[IPP]–#* and inside this folder, create a folder *dicom*. 9. Save the series with the name *[CaseID]-[IPP]_[PatientInitials]-[bone/smooth].0*, e.g. *P123-456789_XY.0*. 10. In the next window, replace the patient’s name by *[CaseID]_[IPP]–[PatientInitials]*. 11. If the error: *Non-uniform coordinates not supported* appears, you will need to convert your data to uniform coordinates before exporting to DICOM. - To do the conversion, attach an *Arithmetic* module to the data object. Select *regular* for the result type. - Specify the resolution (ex: 512x512x267) similar to the original data. - In the *Expression field* put A (the input data object). - Click on *Apply*. - The result is a new uniform coordinate data object. - Save the result as described in points 7 to 10. 12. Rename the folder *CT-###-IPP-xxx* following the classification presented at the beginning of this document. 13. This step is similar to the step16 of the Matlab SOP, described in the previous section *2. Matlab interface for SCase anonymization and importation* 14. Continue with the next CT set.