Created question in FIWARE Q/A platform on 17-07-2015 at 15:07
Please, ANSWER this question AT https://stackoverflow.com/questions/31477545/httpbadrequestexception-orion-0-19-0-libcurl-7-19-7-user-agent-not-supported-an

Question:
HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported and Cosmos HttpFS response: HTTP/1.1 401 Unauthorized

Description:
Good afternoon,

I'm trying to configure persistence of Orion Context's in the cosmos HDFS. So far I have created a Orion Context Broker Instance (orion-psb-image-R4.2), created a Cosmos user account and the folder 'cardio' inside it (/user/USERNAME/cardio).

I've also configured the cygnus.conf file (/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf) to be the following:

1. APACHE_FLUME_HOME/conf/cygnus.conf
   orionagent.sources = http-source
   orionagent.sinks = hdfs-sink
   orionagent.channels = notifications

1. Flume source, must not be changed
   orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
2. channel name where to write the notification events
   orionagent.sources.http-source.channels = notifications
3. listening port the Flume source will use for receiving incoming notifications
   orionagent.sources.http-source.port = 5050
4. Flume handler that will parse the notifications, must not be changed
   orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler
5. regular expression for the orion version the notifications will have in their headers
   orionagent.sources.http-source.handler.orion_version = 0\.11\.*
6. URL target
   orionagent.sources.http-source.handler.notification_target = /notify

1. channel name from where to read notification events
   orionagent.sinks.hdfs-sink.channel = notifications
2. Flume sink that will process and persist in HDFS the notification events, must not be changed
   orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
3. IP address of the Cosmos deployment where the notification events will be persisted
   orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
4. port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
   orionagent.sinks.hdfs-sink.cosmos_port = 14000
5. username allowed to write in HDFS (/user/myusername)
   orionagent.sinks.hdfs-sink.cosmos_username = USERNAME
6. dataset where to persist the data (/user/myusername/mydataset)
   orionagent.sinks.hdfs-sink.cosmos_dataset = cardio
7. HDFS backend type (webhdfs, httpfs or infinity)
   orionagent.sinks.hdfs-sink.hdfs_api = httpfs

1. channel name
   orionagent.channels.notifications.type = memory
2. capacity of the channel
   orionagent.channels.notifications.capacity = 1000
3. amount of bytes that can be sent per transaction
   orionagent.channels.notifications.transactionCapacity = 100

The issue is that when I start the agent with the following command:

/usr/local/apache-flume-1.4.0-bin/bin/flume-ng agent --conf /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent -Dflume.root.logger=INFO,consoleapache-flume-1.4.0-bin

I get the following error messages:

2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.

And:

org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported

Here is the complete log:

+ exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/local/apache-flume-1.4.0-bin/conf:/usr/local/apache-flume-1.4.0-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent
2015-07-17 15:33:09,858 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-07-17 15:33:09,863 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: hdfs-sink Agent: orionagent
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,873 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [orionagent]
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-07-17 15:33:09,899 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel notifications type memory
2015-07-17 15:33:09,908 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel notifications
2015-07-17 15:33:09,912 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
2015-07-17 15:33:09,961 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: hdfs-sink, type: es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
2015-07-17 15:33:09,967 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel notifications connected to [http-source, hdfs-sink]
2015-07-17 15:33:09,972 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource

}} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@e8234ec counterGroup:{ name:null counters:{} } }} channels:{notifications=org.apache.flume.channel.MemoryChannel{name: notifications}} }
2015-07-17 15:33:09,973 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel notifications
2015-07-17 15:33:10,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: notifications, registered successfully.
2015-07-17 15:33:10,028 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: notifications started
2015-07-17 15:33:10,028 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfs-sink
2015-07-17 15:33:10,029 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source http-source
2015-07-17 15:33:10,069 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-07-17 15:33:10,149 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2015-07-17 15:33:10,306 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started SocketConnector@0.0.0.0:5050
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: http-source started
2015-07-17 15:33:10,456 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.http.HttpClientFactory.<init>(HttpClientFactory.java:68)] Setting max total connections to 500 and default max connections per route to 100
2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&amp;user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.
org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported
at es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler.getEvents(OrionRestHandler.java:93)
at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

What could be the issue?