Uploaded image for project: 'JASMIN CIS'
  1. JASMIN CIS
  2. JASCIS-364

Stop using NetCDF Variables as data proxies

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.5
    • Fix Version/s: 2.0
    • Component/s: None
    • Labels:
      None

      Description

      We currently use NetCDF Variable objects as proxies for deferred data reading. This means we have to manually take care of concatenating data-sets, that files have to remain open the whole time (which is a big problem for lots of files) and perhaps most importantly we can't pickle UngriddedData objects (so no parallelisation).

      Iris uses a Biggus NetCDFProxy object, but XArray uses Dask.Arrays - this might be the better option longer term as it probably has better support and looks like it has some really nice parallelisation stuff built-in too. See http://dask.pydata.org/en/latest/index.html# for more info

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              duncan.watson-parris@physics.ox.ac.uk Duncan Watson-Parris
              Reporter:
              duncan.watson-parris@physics.ox.ac.uk Duncan Watson-Parris
              Participants:
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated: