Viewing 11 reply threads
  • Author
    Posts
    • #19771
      jaysantokhi
      Participant

      I made two submissions to the extended challenge and one to the core challenge last week yet they still remain on “pending score”. My first submission (two weeks ago) produced a score after an hour or so and I am curious why my last three submissions have been at “pending score” for a many days now?

      I look forward to hearing from you.

      Jay

       

    • #20886

      @jaysantokhi sorry we did not see your question – does this persist?

    • #20983
      Alabi Bojesomo
      Participant

      Dear Admin,

      I am also experiencing delayed scoring for more than a day as opposed to an average waiting time of one hour that I am used to on this platform.

    • #21003
      moghadas76
      Participant

      I have the same problem…Are we joke to you?

    • #21017
      Moritz Neun
      Moderator

      Thanks again for noticing us and apologies for the delay. The issue has been fixed now and the scores have been updated.

    • #22963
      nusabetslot
      Participant

      ok nice info

    • #24649
      jaysantokhi
      Participant

      Dear @christian and @moritzneun,

      This issue seems to have re-appeared for the Extended Challenge, I have two submissions that have been on “pending score” for nearly 24 hours.

      Jay

    • #24692
      Moritz Neun
      Moderator

      Hi @jaysantokhi ,

      thanks for your note and sorry for the delay the issue is fixed now. We observed a couple of submissions of very large size that caused some delay and sometimes failing jobs. These large h5 files are using float64 as datatype (dtype(<f8)). The evaluation uses uint8 for the golden data comparisons and, hence, we would kindly ask you to also generate the submissions in uint8 (see also https://github.com/iarai/NeurIPS2021-traffic4cast/blob/master/util/h5_util.py#L60 ).

      Thanks and best regards,

      Moritz

    • #25000
      jaysantokhi
      Participant

      Good morning @moritzneun,

      It seems this problem has occurred again on the Extended Challenge. Two submissions I made 16 hours ago are still on “pending score”. All of my submissions have been in the correct format (I make double-y sure before writing my predictions to file that they are in the correct uint8 format).

      Is there a way for submissions that aren’t in the correct format (e.g. float64) to be flagged at point of submission so this issue doesn’t reoccur every few days, especially seeing as we are getting close to the deadline.

      Kind Regards,

      Jay

    • #25027
      tyron4life
      Participant

      People have so much work to do! Respect their time guys! 🙂 My submission will also be available!

    • #25062
      Moritz Neun
      Moderator

      Hi @jaysantokhi,
      we saw the pending score computations yesterday noon and fixed them. However, it seems, that your two pending submissions to the extended challenge, with a size of 270-280MB, were still in float64 (“<f8”) format. See my snippet below for checking the dtype:

      >>> import h5py
      >>> fr = h5py.File(‘<submission_id>/VIENNA/VIENNA_test_spatiotemporal.h5’)
      >>> data = fr.get(“array”)
      >>> print(data)
      <HDF5 dataset “array”: shape (100, 6, 495, 436, 8), type “<f8”>

      Nevertheless, we implemented a fix with force-converts non-uint8 input prior to any other computations. This reduces the load and hence should prevent the issue on the way forward.

      https://github.com/iarai/NeurIPS2021-traffic4cast/pull/9

      Best regards,
      Moritz

    • #51943
      ecesancar
      Participant

      Hi,

      I tried to make new submissions on the Core and Extended Challenge submission boards but their status are still on “pending score” since last week. Did you close the submissions? If you did not can you please check the problem?

      Kind regards.

      • #52082

        Hi @ecesancar @oncelf

        Thanks for your message. There was a problem with the file upload – this is solved now and scoring works again. Can you please re-upload? Please accept our sincere apologies.

        Best regards

        Christian

Viewing 11 reply threads
  • The forum ‘General Questions’ is closed to new topics and replies.

©2022 IARAI - INSTITUTE OF ADVANCED RESEARCH IN ARTIFICIAL INTELLIGENCE

Imprint | Terms and conditions

Log in with your credentials

or    

Forgot your details?

Create Account