Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-5946

FIWARE.Request.Tech.Data.BigData-Analysis.cygnus - kafka sink: can't persist data in kafka by remote

    Details

    • Type: extRequest
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: 2021
    • Component/s: FIWARE-TECH-HELP
    • Labels:
      None

      Description

      Hello,

      Is it possible to have Cygnus and kafka running on different machines?

      My kafka instance run over a virtual machine and it is accessible using a public ip.

      Here is my configuration

      1. OrionKafkaSink configuration
      1. sink class, must not be changed

      cygnusagent.sinks.kafka-sink.type = com.telefonica.iot.cygnus.sinks.OrionKafkaSink

      1. channel name from where to read notification events

      cygnusagent.sinks.kafka-sink.channel = kafka-channel

      1. select the Kafka topic type between topic-by-service, topic-by-service-path and topic-by-destination

      cygnusagent.sinks.kafka-sink.topic_type = topic-by-destination

      1. comma-separated list of Kafka brokers (a broker is defined as host:port)

      cygnusagent.sinks.kafka-sink.broker_list = publicIP:9092

      1. Zookeeper endpoint needed to create Kafka topics, in the form of host:port

      cygnusagent.sinks.kafka-sink.zookeeper_endpoint = publicIP:2181

      cygnusagent.sinks.kafka-sink.batch_size = 1

      cygnusagent.sinks.kafka-sink.batch_timeout = 10

      Here is the error that I got in Cygnus: Error connecting to node 0 at quickstart.cloudera:9092

      The quickstart.cloudera is my virtual machine name, can’t figure out why Cygnus is trying to connect to it by name and not by IP address?

      time=2016-02-22T16:38:08.813UTC | lvl=DEBUG | trans= | srv= | subsrv= | function=initiateConnect | comp=Cygnus | msg=org.apache.kafka.clients.NetworkClient[413] : Initiating connection to node 0 at quickstart.cloudera:9092.

      time=2016-02-22T16:38:08.814UTC | lvl=DEBUG | trans= | srv= | subsrv= | function=initiateConnect | comp=Cygnus | msg=org.apache.kafka.clients.NetworkClient[421] : Error connecting to node 0 at quickstart.cloudera:9092:

      java.io.IOException: Can't resolve address: quickstart.cloudera:9092

      at org.apache.kafka.common.network.Selector.connect(Selector.java:138)

      at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:415)

      at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:116)

      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:165)

      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)

      at java.lang.Thread.run(Thread.java:745)

      Caused by: java.nio.channels.UnresolvedAddressException

      at sun.nio.ch.Net.checkAddress(Net.java:101)

      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)

      at org.apache.kafka.common.network.Selector.connect(Selector.java:135)

      ... 5 more

      Thanks and Best regards,

      Sabrine FATNASSI

      Test and Validation Engineer

      Easy Global Market : http://www.eglobalmark.com/ <http://www.eglobalmark.com/>

      Email : sabrine.fatnassi@eglobalmark.com <sabrine.fatnassi@eglobalmark.com>

      [Created via e-mail received from: "Sabrine Fatnassi (EGM)" <sabrine.fatnassi@eglobalmark.com>]

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        15h 6m 1 Francisco Romero 23/Feb/16 9:14 AM
        In Progress In Progress Answered Answered
        9s 1 Francisco Romero 23/Feb/16 9:14 AM
        Answered Answered Closed Closed
        1h 51m 1 Francisco Romero 23/Feb/16 11:06 AM

          People

          • Assignee:
            frb Francisco Romero
            Reporter:
            fw.ext.user FW External User
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: