Uploaded image for project: 'JASMIN CIS'
  1. JASMIN CIS
  2. JASCIS-364

Stop using NetCDF Variables as data proxies

Details

    • Improvement
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 1.5
    • 2.0
    • None
    • None

    Description

      We currently use NetCDF Variable objects as proxies for deferred data reading. This means we have to manually take care of concatenating data-sets, that files have to remain open the whole time (which is a big problem for lots of files) and perhaps most importantly we can't pickle UngriddedData objects (so no parallelisation).

      Iris uses a Biggus NetCDFProxy object, but XArray uses Dask.Arrays - this might be the better option longer term as it probably has better support and looks like it has some really nice parallelisation stuff built-in too. See http://dask.pydata.org/en/latest/index.html# for more info

      Attachments

        Issue Links

          Activity

            People

              duncan.watson-parris@physics.ox.ac.uk Duncan Watson-Parris
              duncan.watson-parris@physics.ox.ac.uk Duncan Watson-Parris
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: