.. include:: ../macros.hrst .. include:: ../abbreviations.hrst .. _chapter:LoadingData: Loading Data ############ In a visualization pipeline, data sources bring data into the system for processing and visualization. Sources, such as the ``Sphere`` source (accessible from the ``Sources`` menu in |paraview|), programmatically create datasets for processing. Another type of data sources are readers. Readers can read data written out in disk files or other databases and bring it into |ParaView| for processing. |ParaView| includes readers that can read several of the commonly used scientific data formats. It's also possible to write plugins that add support for new or proprietary file formats. |ParaView| provides several sample datasets for you to get started. You can download an archive with several types of data files from the download page at https://www.paraview.org/download under the Data section. Opening data files in |paraview| ================================== To open a data file in |paraview|, you use the ``Open File`` dialog. This dialog can be accessed from the :guilabel:`File > Open` menu or by using the |pqOpen| button in the ``Main Controls`` toolbar. You can also use the keyboard shortcut |CTRL| + :kbd:`O` (or |CMD| + :kbd:`O`) to open this dialog. .. figure:: ../images/OpenFileDialog.png :name: fig-OpenFileDialog :width: 100% :align: center ``Open File`` dialog in |paraview| for opening data (and other) files. The ``Open File`` dialog allows you to browse the file system on the data processing nodes. This will become clear when we look at using |ParaView| for remote visualization. While several of the UI elements in this dialog are obvious such as navigating up the current directory, creating a new directory, and navigating back and forth between directories, which can all be done with the standard system shortcuts like |CTRL| + :kbd:`N` (or |CMD| + :kbd:`N`) to create a directory or |ALT| + |UP| to go to the parent directory, there are a few things to note. * The ``Favorites`` pane contains a custom list of favorite directories that can be customized using the buttons above it, which respectively adds the current directory to the favorites or removes all the favorites. Another way to change which directories are in the favorites is to use the right-click context menus in the list of files. Right-clicking any directory in the main list will display a menu with the option to add it to the favorites. To remove edit an exising favorite, right-click or ctrl-click the favorite. A context menu appears which lets you remove the favorite or rename it. * The ``Locations`` pane shows some platform-specific common locations such as the home directory and desktop. External drives will also appear in this list. * The ``Recent Directories`` pane shows a few of the most recently used directories. You can browse to the directory containing your datasets and either select the file and hit ``Ok`` or simply double click on the file to open it. You can also select multiple files using the |CTRL| (or |CMD|) key. This will open each of the selected files separately. Right-clicking the files list will display a few options, depending on what was right-clicked. .. figure:: ../images/ContextMenuFileDialog.png :width: 100% :align: center * Selecting a directory adds the ``Add to favorites`` option which adds the selected directory to the ``Favorites`` pane on the left. * Selecting a file or a directory adds the ``Rename`` option. * The ``Open in file explorer`` is also always present and either opens in the system file explorer the selected directory if one was right-clicked, or opens the current directory if a file or nothing was selected. * Selecting an empty directory provides the ``Delete empty directory`` option. * The ``Show Hidden Files`` is always visible, and can be checked to display the hidden files and directories. The ``Look in:`` combobox shows the current path in the Open File dialog. Clicking on the combobox down arrow will show the parent directories of the current directory while clicking in the path line edit itself allows you to edit the path. Navigation buttons showing arrows are to the right of the combo box. These enable navigating through previously visited directories as well as moving up to the parent of the current directory. The |pqAdvanced| button to the right of the arrows toggles on or off additional columns in the dialog that show file size and last modified date. When a file is opened, |paraview| will create a reader instance of the type suitable for the selected file based on its extension. The reader will simply be another pipeline module, similar to the source we created in :numref:`chapter:Introduction`. From this point forward, the workflow will be the same as we discussed in :numref:`sec:UnderstandingVisualizationGUI` : You adjust the reader properties, if needed, and hit ``Apply`` . |paraview| will then read the data from the file and render it in the view. If you selected multiple files using the |CTRL| (or |CMD|) key, |paraview| will create multiple reader modules. When you hit ``Apply`` , all of the readers will be executed, and their data will be shown in the view. .. figure:: ../images/OpenFileGroupSelect.png :width: 80% :align: center .. figure:: ../images/OpenFileMultiSelect.png :width: 80% :align: center :name: fig:OpenFileSelectingMultipleFiles The ``Open File`` dialog can be used to select a temporal file series (top) or select multiple files to open separately (bottom). By default, |paraview| groups files that appear to define a time series. These file series have names that contain a number sequence where the number defines the order of the files in time. |paraview| can be told not to do this by clicking on the |pqGroupFiles| button at the top right of the ``Open File`` dialog. .. admonition:: **Did you know?** :class: tip This ability to hit the ``Apply`` button once to accept changes on multiple readers applies to other pipeline modules, including sources and filters. In general, you can change properties for multiple modules in a pipeline, and hit ``Apply`` to accept all of the changes at once. It is possible to override this behavior from |ParaView|'s' ``Settings`` dialog using the ``Auto Apply Active Only`` setting under the ``General`` tab. When this setting is enabled, only the selected source in the ``Pipeline Browser`` will be updated. Reader selection depending on the selected file types ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. figure:: ../images/FileDialogFilter.png :name: fig:FileDialogFilter :width: 100% :align: center File dialog "Files of type" to filter which files to show and to change how the reader will be chosen. When selecting files, the current category for the "Files of type" field (:numref:`fig:FileDialogFilter`) changes the way the reader will be chosen. * If the "Supported Files" value is selected, if only one reader is available for the type of file, it will be selected automatically. If multiple readers can be used, the "Open Data With..." dialog (:numref:`fig:OpenDataWithDialog`) will appear to choose which reader to use. Clicking the "Set reader as default" button tells ParaView to automatically use this reader when one of the file name patterns it says it can support matches a file that could otherwise be opened by multiple readers. When a reader is set as default, it will be used automatically for files that match its patterns. .. figure:: ../images/OpenDataWithDialog.png :name: fig:OpenDataWithDialog :width: 60% :align: center ``Open Data With...`` dialog shown to manually choose the reader to use for a file with multiple available readers. * If the "All Files" value is selected, the same dialog will be displayed with all the existing readers. If you picked an incorrect reader, however, you'll get error messages either when the reader module is instantiated or after you hit ``Apply``. In either case, you can simply ``Delete`` the reader module and try opening the file again, this time choosing a different reader. If you can click the ``Set reader as default`` button, a small window with a line edit will be displayed where you can set the custom pattern to use for this reader (:numref:`fig:DefineCustomPatternDialog`). This is a standard wildcard pattern, and multiple patterns can be used by separating them with spaces. .. figure:: ../images/DefineCustomPatternDialog.png :name: fig:DefineCustomPatternDialog :width: 60% :align: center The ``Default reader details`` setting is used to define file name patterns and which reader should read files that match those patterns. * If a specific reader is selected, for example "PNG Image Files", this reader will be always be used automatically, even if other readers would be available or if the file is matching a pattern present in the "Default reader details" settings. Error messages in |paraview| are shown in the ``Output Messages`` window (:numref:`fig:OutputMessagesWindow`). It is accessible from the :guilabel:`View > Output Messages` menu. Whenever there's a new error message, |paraview| will automatically pop open this window and raise it to the top (as long as the ``Always open for new messages`` option is set. This window can be attached, or docked, in the main window so that it is visible with the other user interface elements without covering them up. .. figure:: ../images/OutputMessagesWindow.png :name: fig:OutputMessagesWindow :width: 100% :align: center The ``Output Messages`` window is used to show errors, warnings, and other messages raised by the application. The default readers settings can be seen and modified in the :guilabel:`Edit > Settings` menu, and ``IO`` tab (:numref:`fig:MiscellaneousSettings`). .. figure:: ../images/MiscellaneousSettings.png :name: fig:MiscellaneousSettings :width: 80% :align: center Default reader settings. Handling temporal file series ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Most datasets produced by scientific simulation runs are temporal in nature. File formats differ in how this information is saved in the file. While several file formats support saving multiple timesteps in the same file, others save them out as a sequence of files, known as a file series The ``Open File`` dialog automatically detects file series and shows them as a grouped element, as shown in :numref:`fig:OpenFileSelectingMultipleFiles`. To load the file series, simply select the group, and hit ``Ok`` . You can also open a single file in the series as a regular file. To do so, open the file group and select the file you want to open. |paraview| automatically detects several of the commonly-known file naming patterns used for indicating a file series. These include: +--------------+--------------+-------------+-------------+ |``fooN.vtk`` |``fooN.vtk`` |``Nfoo.vtk`` |``foo.vtk.N``| +--------------+--------------+-------------+-------------+ |``foo_N.vtk`` |``foo.N.vtk`` |``N.foo.vtk``|``foo.vtksN``| +--------------+--------------+-------------+-------------+ where *foo* could be any filename, *N* is a numeral sequence (with any number of leading zeros), and *vtk* could be any extension. Sometimes this grouping of files with names that follow a numeral sequence into a time series is not desired. The ``Open File`` dialog has a toggle button that controls whether these file sequences are interpreted as time series. It defaults to on, but if toggled off, then |paraview| will not group file series until the button is toggled back on. ParaView also supports a meta file format based on JSON. This format has support for specifying time values. The format (currently version 1.0) looks like the following .. code-block:: json { "file-series-version" : "1.0", "files" : [ { "name" : "foo1.vtk", "time" : 0 }, { "name" : "foo2.vtk", "time" : 5.5 }, { "name" : "foo3.vtk", "time" : 11.2 } ] } Usually, a reader supporting ``extension`` will also supports ``extension.series`` as the file series meta file. .. _sec:DealingWithTime: Dealing with time ^^^^^^^^^^^^^^^^^ When you open a dataset with time, either as a file series or in a file format that natively supports time, |paraview| will automatically set up an animation for you so that you can play through each of the time steps in the dataset by using the |pqVcrPlay| button on the ``VCR Controls`` toolbar (:numref:`fig:VCRControlsToolbar`). You can change or modify this animation and further customize it, as discussed in Chapter :numref:`chapter:Animation`. .. figure:: ../images/VCRControlsToolbar.png :name: fig:VCRControlsToolbar :width: 30% :align: center ``VCR Controls`` toolbar for interacting with an animation. Reopening previously opened files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |paraview| remembers most recently opened files (or file series). Simply use the :guilabel:`File > Recent Files` menu. |paraview| also remembers the reader type selected for files with unknown extensions or for occasions when multiple reader choices were available. Lastly, ParaView also records the server connection used to load a particular file. If you are connected to one server and access a file through the :guilabel:`Recent Files` menu that was on another server, |paraview| will ask if you want to disconnect from the current server and connect to the server on which the recent file resides. Opening files using command line options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |paraview| provides a command line option that can be used to open datasets on startup. .. code-block:: shell > paraview --data=.../ParaViewData/Data/can.ex2 This is equivalent to opening a can.ex2 data file from the ``Open File`` dialog. The same set of follow-up actions happen. For example, |paraview| will try to locate a reader for the file, create that reader, and wait for you to hit ``Apply`` . To open a file series, simply replace the numbers in the file name sequence by a ``.`` For example, to open a file series named ``my0.vtk``, ``my1.vtk`` ... ``myN.vtk``, use ``my..vtk``. .. code-block:: shell > paraview --data=.../ParaViewData/Data/my..vtk Common properties for readers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |ParaView| uses different reader implementations for different file formats. Each of these have different properties available to you to customize how the data is read and can vary greatly depending on the capabilities of the file format itself or the particular reader implementation. Let's look at some of the properties commonly available in readers. .. _sec:SelectingDataArrays: Selecting data arrays --------------------- .. figure:: ../images/ArraySelectionWidget.png :name: fig:ArrayStatusWidget :width: 40% :align: center Array selection widget for selecting array to load from a data file. One of the most common properties on readers is one that allows you to select the data arrays to be loaded, be they cell-centered, point-centered, or otherwise. Often times, loading only the data arrays you know you are going to use in the visualization will save memory, as well as processing time, since the reader does not have to read in those data arrays, and the filters in the pipeline do not have to process them. .. admonition:: **Did you know?** :class: tip You can change |paraview|'s default behavior to load all available data arrays by selecting the ``Load All Variables`` checkbox under ``Settings/Properties Panel Options/Advanced`` . .. figure:: ../images/AdvancedSettings.png :name: fig-AdvancedSettings :width: 80% :align: center The user interface for selecting the arrays to load is simply a list with the names of the arrays and a checkbox indicating whether that array is to be loaded or not (:numref:`fig:ArrayStatusWidget`). Icons, such as |pqCellData| and |pqNodalData| are often used in this widget to give you an indication of whether the array is cell-centered or point-centered, respectively. If you initially de-select an array, but then as you're setting up your visualization pipeline realize that you need that data array, you can always go back to the ``Properties`` page for the reader by making the reader active in the ``Pipeline Browser`` and then changing the array selection. |ParaView| will automatically re-execute any processing pipeline set up on the reader with this new data array. .. admonition:: **Common Errors** :class: error Remember to hit ``Apply`` (or use ``Auto Apply`` ) after changing the array selection for the change to take effect. Sometimes the list of data arrays can get quite large, and it can become cumbersome to find the array for which you are looking. To help with such situations, |paraview| provides a mechanism to search lists. Click inside the widget to make it get the *focus*. Then type |CTRL| + :kbd:`F` (or |CMD| + :kbd:`F`) to get a search widget. Now you can type in the text to search. Matching rows will be highlighted (:numref:`fig:SearchInLists`). .. figure:: ../images/SearchInLists.png :name: fig:SearchInLists :width: 40% :align: center To search through large lists in |paraview|, you can use |CTRL| + :kbd:`F`. .. admonition:: **Did you know?** :class: tip The ability to search for items in an array selection widget also applies to other list and tree widgets in the |paraview| UI. Whenever you see a widget with a large number of entries in a list, table, or tree fashion, try using |CTRL| + :kbd:`F` (or |CMD| + :kbd:`F`). .. % Maybe this is too advanced at this point in the guide? -- UDA %Selecting blocks ----------------- % %Just like selecting of arrays, certain readers such as those for Exodus or Xdmf %files, allow users to the data blocks. .. _sec:OpeningDataFilesInPython: Opening data files in |pvpython| ================================== To open data files using the scripting interface, |ParaView| provides the ``OpenDataFile`` function. .. code-block:: python >>> reader = OpenDataFile(".../ParaViewData/Data/can.ex2") >>> if reader: ... print("Success") ... else: ... print("Failed") ... ``OpenDataFile`` will try to determine an appropriate reader based on the file extension, just like |paraview|. If no reader is determined, ``None`` is returned. If multiple readers can open the file, however, ``OpenDataFile`` simply picks the first reader. If you explicitly want to create a specific reader, you can always create the reader by its name, similar to other sources and filters. .. code-block:: python >>> reader = ExodusIIReader(FileName=".../ParaViewData/Data/can.ex2") To find out information about the reader created and the properties available on it, you can use the ``help`` function. .. code-block:: python >>> reader = ExodusIIReader(FileName=".../ParaViewData/Data/can.ex2") >>> help(reader) Help on ExodusIIReader in module paraview.servermanager object: class ExodusIIReader(ExodusIIReaderProxy) | The Exodus reader loads | Exodus II files and produces an unstructured grid output. | The default file extensions are .g, .e, .ex2, .ex2v2, | .exo, .gen, .exoII, .exii, .0, .00, .000, and .0000. The | file format is described fully at: | http://endo.sandia.gov/SEACAS/Documentation/exodusII.pdf. | ... | | ----------------------------------------------------------------- | Data descriptors defined here: | | AnimateVibrations | If this flag is on and HasModeShapes is also on, then | this reader will report a continuous time range [0,1] and | animate the displacements in a periodic sinusoid. If this | flag is off and HasModeShapes is on, this reader ignores | time. This flag has no effect if HasModeShapes is off. | | ApplyDisplacements | Geometric locations can include displacements. When this | option is on, the nodal positions are 'displaced' by the | standard exodus displacement vector. If displacements are | turned 'off', the user can explicitly add them by applying | a warp filter. | ... .. admonition:: **Did you know?** :class: tip The ``help`` function can be used to get information about properties available on any source or filter instance. It not only lists the properties, but also provides information about how they affect the pipeline module. ``help`` can also be used on functions. For example: .. code-block:: python >>> help(OpenDataFile) Help on function OpenDataFile in module paraview.simple: OpenDataFile(filename, \*\*extraArgs) Creates a reader to read the given file, if possible. This uses extension matching to determine the best reader possible. If a reader cannot be identified, then this returns None. Handling temporal file series ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Unlike |paraview|, |pvpython| does not automatically detect and load file series. There are two ways you can load a file series: * You can explicitly list the filenames in the series and pass that to the ``OpenDataFile`` call. .. code-block:: python # Create a list with the names of all the files in the file series in # correct order. >>> files = [".../Data/multicomb_0.vts", ".../Data/multicomb_1.vts", ".../Data/multicomb_2.vts"] >>> reader = OpenDataFile(files) * You can use globbing by inserting the wildcard **\*** in the file name using the utility **paraview.util.Glob**. This utility runs `fnmatch` python package on the server's file system, so any pattern supported by fnmatch is supported and interpreted by this utility. .. code-block:: python >>> import paraview.util # Create a list of names of all the files in the file series. >>> files = paraview.util.Glob(path = "multicomb_*.vts", rootDir = ".../Data") >>> reader = OpenDataFile(files) Dealing with time ^^^^^^^^^^^^^^^^^ Similar to |paraview|, if you open a time series or a file with multiple timesteps, |pvpython| will automatically set up an animation for you to play through the timesteps. .. code-block:: python >>> files = [".../Data/multicomb_0.vts", ".../Data/multicomb_1.vts", ".../Data/multicomb_2.vts"] >>> reader = OpenDataFile(files) >>> Show() >>> Render() # Get access to the animation scene. >>> scene = GetAnimationScene() # Now you use the API on the scene when doing things such as playing # the animation, stepping through it, etc. # This will simply play through the animation once and stop. Watch # the rendered view after you hit `Enter.' >>> scene.Play() Common properties on readers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Selecting data arrays --------------------- For those properties on readers that allow you to control what to read in from the file such as point data arrays, cell data arrays, or data blocks, |paraview| uses a selection widget, as seen in :numref:`sec:SelectingDataArrays`. Likewise, |pvpython| provides an API that allows you to determine the available options and then select/deselect them. The name of the property that allows you to make such selections depends on the reader itself. When in doubt, use the tracing capabilities in |paraview| (:numref:`sec:PythonTracing`) to figure it out. You can also use ``help`` (:numref:`sec:OpeningDataFilesInPython`). ``ExodusIIReader`` has a ``PointVariables`` property that can be used to select the point data arrays to load. Let's use this as an example. .. code-block:: python # Open an ExodusII data file. >>> reader = OpenDataFile(".../Data/can.ex2") # Alternatively, you can explicitly create the reader instance as: >>> reader = ExodusIIReader(FileName = ".../Data/can.ex2") # To query/print the current status for `PointVariables' property, # we do what we would have done for any other property: >>> print(GetProperty("PointVariables")) ['DISPL', 'VEL', 'ACCL'] # An alternative way of doing the same is as follows: >>> print(reader.PointVariables) ['DISPL', 'VEL', 'ACCL'] # To set the property, simply set it to list containing the names to # enable, e.g., if we want to read only the 'DISPL' array, we do # the following: >>> SetProperties(PointVariables=['DISPL']) # Or using the alternative way for doing the same: >>> reader.PointVariables = ['DISPL'] # Now, the new value for PointVariables is: >>> print(reader.PointVariables) ['DISPL'] # To determine the array available, use: >>> print(reader.PointVariables.Available) ['DISPL', 'VEL', 'ACCL'] # These are the arrays available in the file. Changing ``PointVariables`` only changes the value on the property. The reader does not re-execute until a re-execution is requested either by calling ``Render`` or by explicitly updating the pipeline using ``UpdatePipeline`` . .. code-block:: python >>> reader.PointVariables = ['DISPL', 'VEL', 'ACCL'] # Assuming that the reader is indeed the active source, let's update # the pipeline: >>> UpdatePipeline() # Or you can use the following form if you're unsure of the active # source or just do not want to worry about it. >>> UpdatePipeline(proxy=reader) # Print the list of point arrays read in. >>> print(reader.PointData[:]) [Array: ACCL, Array: DISPL, Array: GlobalNodeId, Array: PedigreeNodeId, Array: VEL] # Change the selection. >>> reader.PointVariables = ['DISPL'] # Print the list of point arrays read in, nothing changes! >>> print(reader.PointData[:]) [Array: ACCL, Array: DISPL, Array: GlobalNodeId, Array: PedigreeNodeId, Array: VEL] # Update the pipeline. >>> UpdatePipeline() # Now the arrays read in has indeed changed as we expected. >>> print(reader.PointData[:]) [Array: DISPL, Array: GlobalNodeId, Array: PedigreeNodeId] We will cover the ``reader.PointData`` API in more details in :numref:`sec:DataInformationInPython`. Reloading files =============== While ParaView is often used after the simulation has generated all the data, it is not uncommon to use ParaView to inspect data files as they are being written out by the simulation. In such cases, the simulation may either be modifying existing file(s) with new timesteps or creating new files for each timestep. In such cases, you may want to *refresh* ParaView to make it aware of the changes. In |paraview|, this can be done using ``Reload Files`` . When the reader is active, you can use the :guilabel:`File > Reload Files` menu to request the reader to refresh. |paraview| will prompt you to choose whether to reload the existing file(s) or look for new files in the file series, as shown in :numref:`fig:ReloadOptions`. Click on ``Reload existing file(s)`` , to force the reader to re-read the files already opened. This is useful in cases where the simulation may have modified existing file(s). Use ``Find new files`` to make the reader aware of any new files in the file series. .. figure:: ../images/ReloadOptions.png :name: fig:ReloadOptions :width: 50% :align: center The ``Reload Options`` dialog allows you to choose how to refresh the reader. Similar to |paraview|, in |pvpython|, you use ``ReloadFiles`` to reload existing files, and ``ExtendFilesSeries`` to look for new files in a file series. .. code-block:: python # For file being modified in place per timestep >>> reader = OpenDataFile(file) ... >>> ReloadFiles(reader) # For files being generated per timestep >>> reader = OpenDataFile(file) ... >>> ExtendFilesSeries(reader) .. |pqAdvanced| image:: ../images/icons/pqAdvanced.svg :width: 20px .. |pqGroupFiles| image:: ../images/icons/pqGroupFiles.svg :width: 20px .. |pqOpen| image:: ../images/icons/pqOpen.svg :width: 20px .. |pqVcrPlay| image:: ../images/icons/pqVcrPlay.svg :width: 20px .. |pqCellData| image:: ../images/icons/pqCellData.svg :width: 20px .. |pqNodalData| image:: ../images/icons/pqNodalData.svg :width: 20px