Uploaded image for project: 'Help-Coaches-Desk'
  1. Help-Coaches-Desk
  2. HELC-1130

FIWARE.Request.Coach.INCENSe.Heaving problems perssiting data from Cygnus to Cosmos global instanse

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Component/s: INCENSe
    • Labels:
      None

      Description

      I have looked at the log and found the following:
      10 Nov 2015 16:39:57,903 INFO
      [SinkRunner-PollingRunner-DefaultSinkProcessor]
      (com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation:918) -
      [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file
      (def_serv/def_servpath/8_registervalues/8_registervalues.txt), Data
      (

      {"recvTime":"2015-11-10T15:39:39.919Z","fiwareservicepath":"def_servpath","entityId":"8","entityType":"RegisterValues", "value":"1959", "value_md":[], "name":"register number 8", "name_md":[]}

      )
      10 Nov 2015 16:39:58,183 ERROR
      [SinkRunner-PollingRunner-DefaultSinkProcessor]
      (com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence
      error (The /user/ms/def_serv/def_servpath/8_registervalues directory could
      not be created in HDFS. Server response: 503 Service unavailable)
      10 Nov 2015 16:39:58,183 INFO
      [SinkRunner-PollingRunner-DefaultSinkProcessor]
      (com.telefonica.iot.cygnus.sinks.OrionSink.process:307) - Finishing
      transaction (1447169873-477-0000000000)

      Is there anything wrong with the configuration I am using?

      Agent_1.conf:

      1. Copyright 2014 Telefֳ³nica Investigaciֳ³n y Desarrollo, S.A.U
        #
      2. This file is part of fiware-cygnus (FI-WARE project).
        #
      3. fiware-cygnus is free software: you can redistribute it and/or modify it
        under the terms of the GNU Affero General
      4. Public License as published by the Free Software Foundation, either
        version 3 of the License, or (at your option) any
      5. later version.
      6. fiware-cygnus is distributed in the hope that it will be useful, but
        WITHOUT ANY WARRANTY; without even the implied
      7. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
        GNU Affero General Public License for more
      8. details.
        #
      9. You should have received a copy of the GNU Affero General Public License
        along with fiware-cygnus. If not, see
      10. http://www.gnu.org/licenses/.
        #
      11. For those usages not covered by the GNU Affero General Public License
        please contact with iot_support at tid dot es

      #=============================================

      1. To be put in APACHE_FLUME_HOME/conf/agent.conf
        #
      2. General configuration template explaining how to setup a sink of each of
        the available types (HDFS, CKAN, MySQL).

      #=============================================

      1. The next tree fields set the sources, sinks and channels used by Cygnus.
        You could use different names than the
      2. ones suggested below, but in that case make sure you keep coherence in
        properties names along the configuration file.
      3. Regarding sinks, you can use multiple types at the same time; the only
        requirement is to provide a channel for each
      4. one of them (this example shows how to configure 3 sink types at the same
        time). Even, you can define more than one
      5. sink of the same type and sharing the channel in order to improve the
        performance (this is like having
      6. multi-threading).
        cygnusagent.sources = http-source
        cygnusagent.sinks = hdfs-sink
        cygnusagent.channels = hdfs-channel

      #=============================================

      1. source configuration
      2. channel name where to write the notification events
        cygnusagent.sources.http-source.channels = hdfs-channel
      3. source class, must not be changed
        cygnusagent.sources.http-source.type =
        org.apache.flume.source.http.HTTPSource
      4. listening port the Flume source will use for receiving incoming
        notifications
        cygnusagent.sources.http-source.port = 5050
      5. Flume handler that will parse the notifications, must not be changed
        cygnusagent.sources.http-source.handler =
        com.telefonica.iot.cygnus.handlers.OrionRestHandler
      6. URL target
        cygnusagent.sources.http-source.handler.notification_target = /notify
      7. Default service (service semantic depends on the persistence sink)
        cygnusagent.sources.http-source.handler.default_service = def_serv
      8. Default service path (service path semantic depends on the persistence
        sink)
        cygnusagent.sources.http-source.handler.default_service_path = def_servpath
      9. Number of channel re-injection retries before a Flume event is definitely
        discarded (-1 means infinite retries)
        cygnusagent.sources.http-source.handler.events_ttl = 10
      10. Source interceptors, do not change
        cygnusagent.sources.http-source.interceptors = ts gi
      11. TimestampInterceptor, do not change
        cygnusagent.sources.http-source.interceptors.ts.type = timestamp
      12. GroupinInterceptor, do not change
        cygnusagent.sources.http-source.interceptors.gi.type =
        com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
      13. Grouping rules for the GroupingInterceptor, put the right absolute path
        to the file if necessary
      14. See the doc/design/interceptors document for more details
        cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file =
        /usr/cygnus/conf/grouping_rules.conf
      1. ============================================
      2. OrionHDFSSink configuration
      3. channel name from where to read notification events
        cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
      4. sink class, must not be changed
        cygnusagent.sinks.hdfs-sink.type =
        com.telefonica.iot.cygnus.sinks.OrionHDFSSink
        #ֲ true if the grouping feature is enabled for this sink, false otherwise
        cygnusagent.sinks.hdfs-sink.enable_grouping = false
      5. rest if the interaction with HDFS will be WebHDFS/HttpFS-based, binary if
        based on the Hadoop API
        cygnusagent.sinks.hdfs-sink.backend_impl = rest
      6. Comma-separated list of FQDN/IP address regarding the HDFS Namenode
        endpoints
      7. If you are using Kerberos authentication, then the usage of FQDNs instead
        of IP addresses is mandatory
        cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
      8. port of the HDFS service listening for persistence operations; 14000 for
        httpfs, 50070 for webhdfs
        cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
      9. username allowed to write in HDFS
        cygnusagent.sinks.hdfs-sink.hdfs_username = COSMOS_USER
        #ֲ password for the above username; this is only required for Hive
        authentication
        cygnusagent.sinks.hdfs-sink.hdfs_password = COSMOS_PASSWORD
      10. OAuth2 token for HDFS authentication
        #cygnusagent.sinks.hdfs-sink.oauth2_token = TOKEN
      11. how the attributes are stored, available formats are json-row,
        json-column, csv-row and csv-column
        cygnusagent.sinks.hdfs-sink.file_format = json-column
        #ֲ number of notifications to be included within a processing batch
        cygnusagent.sinks.hdfs-sink.batch_size = 100
      12. timeout for batch accumulation
        cygunsagent.sinks.hdfs-sink.batch_timeout = 30
      13. Hive enabling
        cygnusagent.sinks.hdfs-sink.hive = true
      14. Hive server version, 1 or 2 (ignored if hive is false)
        cygnusagent.sinks.hdfs-sink.hive.server_version = 2
      15. Hive FQDN/IP address of the Hive server (ignored if hive is false)
        cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org
      16. Hive port for Hive external table provisioning (ignored if hive is false)
        cygnusagent.sinks.hdfs-sink.hive.port = 10000
      17. Kerberos-based authentication enabling
        cygnusagent.sinks.hdfs-sink.krb5_auth = false
      18. Kerberos username (ignored if krb5_auth is false)
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
      19. Kerberos password (ignored if krb5_auth is false)
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
      20. Kerberos login file (ignored if krb5_auth is false)
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file =
        /usr/cygnus/conf/krb5_login.conf
      21. Kerberos configuration file (ignored if krb5_auth is false)
        cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file =
        /usr/cygnus/conf/krb5.conf

      #=============================================

      1. hdfs-channel configuration
      2. channel type (must not be changed)
        cygnusagent.channels.hdfs-channel.type = memory
      3. capacity of the channel
        cygnusagent.channels.hdfs-channel.capacity = 1000
      4. amount of bytes that can be sent per transaction
        cygnusagent.channels.hdfs-channel.transactionCapacity = 100

      CYGNUS_INSTANCE_1.CONF

      #####
      #

      1. Configuration file for apache-flume
        #
        #####
      2. Copyright 2014 Telefonica Investigaciֳ³n y Desarrollo, S.A.U
        #
      3. This file is part of fiware-cygnus (FI-WARE project).
        #
      4. fiware-cygnus is free software: you can redistribute it and/or modify it
        under the terms of the GNU Affero General
      5. Public License as published by the Free Software Foundation, either
        version 3 of the License, or (at your option) any
      6. later version.
      7. fiware-cygnus is distributed in the hope that it will be useful, but
        WITHOUT ANY WARRANTY; without even the implied
      8. warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
        GNU Affero General Public License for more
      9. details.
        #
      10. You should have received a copy of the GNU Affero General Public License
        along with fiware-cygnus. If not, see
      11. http://www.gnu.org/licenses/.
        #
      12. For those usages not covered by the GNU Affero General Public License
        please contact with iot_support at tid dot es
      1. Who to run cygnus as. Note that you may need to use root if you want
      2. to run cygnus in a privileged port (<1024)
        CYGNUS_USER=cygnus
      1. Where is the config folder
        CONFIG_FOLDER=/usr/cygnus/conf
      1. Which is the config file
        CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
      1. Name of the agent. The name of the agent is not trivial, since it is the
        base for the Flume parameters
      2. naming conventions, e.g. it appears in .sources.http-source.channels=...
        AGENT_NAME=cygnusagent
      1. Name of the logfile located at /var/log/cygnus. It is important to put
        the extension '.log' in order to the log rotation works properly
        LOGFILE_NAME=cygnus.log
      1. Administration port. Must be unique per instance
        ADMIN_PORT=8081
      1. Polling interval (seconds) for the configuration reloading
        POLLING_INTERVAL=30

      Thanks you!

      Menachem Shtalhaim

      menachem.shtalhaim@n-join.com <elkana@n-join.com> | +972-5
      <%2B972-527-213941>47241959 | www.n-join.com

      n-Join – Maximizing industrial potential

      Since January 1st, old domains won't be supported and messages sent to any domain different to @lists.fiware.org will be lost.
      Please, send your messages using the new domain (Fiware-incense-coaching@lists.fiware.org) instead of the old one.
      _______________________________________________
      Fiware-incense-coaching mailing list
      Fiware-incense-coaching@lists.fiware.org
      https://lists.fiware.org/listinfo/fiware-incense-coaching
      [Created via e-mail received from: Menachem Shtalhaim <menachem.shtalhaim@n-join.com>]

        Activity

        Hide
        pandriani Pasquale Andriani added a comment -

        duplicated of HELC-1132

        Show
        pandriani Pasquale Andriani added a comment - duplicated of HELC-1132
        Hide
        mev Manuel Escriche added a comment -

        Pasquale, we've got Duplicate as resolution.
        https://jira.fiware.org/secure/admin/ViewResolutions.jspa

        In case you find it helpful. When you evolve the ticket you can take the Resolve transition, which will allow you to pick up the resolution. I mean we use Finish for 'successful' resolutions, otherwise, we use Resolve.

        Show
        mev Manuel Escriche added a comment - Pasquale, we've got Duplicate as resolution. https://jira.fiware.org/secure/admin/ViewResolutions.jspa In case you find it helpful. When you evolve the ticket you can take the Resolve transition, which will allow you to pick up the resolution. I mean we use Finish for 'successful' resolutions, otherwise, we use Resolve.

          People

          • Assignee:
            pandriani Pasquale Andriani
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: