Uploaded image for project: 'Help-Desk'
  1. Help-Desk
  2. HELP-13279

[fiware-stackoverflow] Using Apache Flume Kafka sink on Cygnus

        Details

          Description

          Created question in FIWARE Q/A platform on 21-10-2017 at 03:10
          Please, ANSWER this question AT https://stackoverflow.com/questions/46859251/using-apache-flume-kafka-sink-on-cygnus

          Question:
          Using Apache Flume Kafka sink on Cygnus

          Description:
          I want to use the Kafka sink that comes with Apache Flume. However, it seems that the cygnus-common package is based on Apache Flume 1.4 whereas the Kafka sink was added around Apache Flume 1.7. If I add the Kafka jar file to the cygnus-common library it fails because the the cygnus-common core is outdated.

          So it is possible to use the latest flume version (1.8) instead of cygnus-common with cygnus-ngsi libraries? In that case I can switch between using Apache Flume Kafka sink and the NGSIKafkaSink?

            Activity

            Ascending order - Click to sort in descending order
            backlogmanager Backlog Manager created issue -
            Hide
            Permalink
            backlogmanager Backlog Manager added a comment -

            2018-01-12 19:48|CREATED monitor | # answers= 0, accepted answer= False

            Show
            backlogmanager Backlog Manager added a comment - 2018-01-12 19:48|CREATED monitor | # answers= 0, accepted answer= False
            backlogmanager Backlog Manager made changes -
            Field Original Value New Value
            Component/s FIWARE-TECH-HELP [ 10278 ]
            fla Fernando Lopez made changes -
            Assignee Alvaro Alonso [ aalonsog ]
            aalonsog Alvaro Alonso made changes -
            Assignee Alvaro Alonso [ aalonsog ] Joaquín Salvachúa [ joaquin.salvachua ]
            joaquin.salvachua Joaquín Salvachúa made changes -
            Assignee Joaquín Salvachúa [ joaquin.salvachua ] Andres Muñoz [ andres.munoza ]
            Hide
            Permalink
            andres.munoza Andres Muñoz added a comment -

            Hello alstack,

            I am Andrés from UPM Fiware Team. You are right pointing out that Fiware Cygnus is based on Apache Flume 1.4 so, Answering your question, exist a high probability that the sinks used in a posterior version of Flume don't be compatible with Cygnus. We are working on an update of Cygnus in order to use the lastest release of flume and also manage the NGSIV2 notifications in a native way but it is a work in progress and takes time. I think that currently the only way to work with Cygnus and kafka sink is using the GE provided by FIWARE in his corresponding versions.

            Show
            andres.munoza Andres Muñoz added a comment - Hello alstack, I am Andrés from UPM Fiware Team. You are right pointing out that Fiware Cygnus is based on Apache Flume 1.4 so, Answering your question, exist a high probability that the sinks used in a posterior version of Flume don't be compatible with Cygnus. We are working on an update of Cygnus in order to use the lastest release of flume and also manage the NGSIV2 notifications in a native way but it is a work in progress and takes time. I think that currently the only way to work with Cygnus and kafka sink is using the GE provided by FIWARE in his corresponding versions.
            andres.munoza Andres Muñoz made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            andres.munoza Andres Muñoz made changes -
            Status In Progress [ 3 ] Answered [ 10104 ]
            andres.munoza Andres Muñoz made changes -
            Resolution Done [ 10000 ]
            Status Answered [ 10104 ] Closed [ 6 ]
            fla Fernando Lopez made changes -
            HD-Enabler Cygnus [ 11304 ]
            HD-Chapter Data [ 10838 ]
            fla Fernando Lopez made changes -
            Fix Version/s 2021 [ 12600 ]
            Transition Time In Source Status Execution Times Last Executer Last Execution Date
            Open Open In Progress In Progress
            		5d 14h 56m 1 Andres Muñoz 18/Jan/18 10:45 AM
            In Progress In Progress Answered Answered
            		2s 1 Andres Muñoz 18/Jan/18 10:45 AM
            Answered Answered Closed Closed
            		1s 1 Andres Muñoz 18/Jan/18 10:45 AM

              People

              • Assignee:
                andres.munoza Andres Muñoz
                Reporter:
                backlogmanager Backlog Manager
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: