convert apache logs to json

[Robert Recchia]

Does anyone know of any scripts or tools out there that can convert
apache access logs to json format so they can be inserted into
couchdb.

[Sam Bisbee]

Post by Robert Recchia
Does anyone know of any scripts or tools out there that can convert
apache access logs to json format so they can be inserted into
couchdb.

No, but I just threw together a little something something for you while
cooking and eating dinner: http://github.com/ravidgemole/common2couchdb

As the README says, it takes the common log formatted log and turns it into a
bulk insert that you can throw at your couchdb instance. Needs work, etc.,
especially around running the script on the same log more than once (you'll get
duplicates and will re-read the whole log file).

Cheers,

[Robert Recchia]

thanks very much works like a charm just tried it. much appreicated

Post by Sam Bisbee

Post by Robert Recchia
Does anyone know of any scripts or tools out there that can convert
apache access logs to json format so they can be inserted into
couchdb.

No, but I just threw together a little something something for you while
cooking and eating dinner: http://github.com/ravidgemole/common2couchdb
As the README says, it takes the common log formatted log and turns it into a
bulk insert that you can throw at your couchdb instance. Needs work, etc.,
especially around running the script on the same log more than once (you'll get
duplicates and will re-read the whole log file).
Cheers,
--
Sam Bisbee

[Nicholas Orr]

FYI with nginx/apache you could output a json encoded log and simply read
that straight into couchdb.

I've created the log format I want, I just haven't sorted out the best way
to read each line into couchdb, when I do it'll be simple as pie :)

Being json encoded the log file is larger - so watch out for that.

Nick

Post by Robert Recchia
thanks very much works like a charm just tried it. much appreicated

Post by Sam Bisbee

Post by Robert Recchia
Does anyone know of any scripts or tools out there that can convert
apache access logs to json format so they can be inserted into
couchdb.

No, but I just threw together a little something something for you while
cooking and eating dinner: http://github.com/ravidgemole/common2couchdb
As the README says, it takes the common log formatted log and turns it

into a

Post by Sam Bisbee
bulk insert that you can throw at your couchdb instance. Needs work,

etc.,

Post by Sam Bisbee
especially around running the script on the same log more than once

(you'll get

Post by Sam Bisbee
duplicates and will re-read the whole log file).
Cheers,
--
Sam Bisbee

--
Robert Recchia

[Daniel Truemper]

In September or so I have implemented a little logfile analysis tool as CouchApp:

http://github.com/truemped/apache_loghouse

It contains a Python script for loading and extracting Http Agent information from the logs...

cheers,
Daniel

[Brian Candler]

Post by Nicholas Orr
I've created the log format I want, I just haven't sorted out the best way
to read each line into couchdb, when I do it'll be simple as pie :)

Remember that CustomLog can pipe into an external program. This could be a
small script which turns each line into a separate POST request in real
time, or batch a few seconds' worth of logs and then do a POST to
_bulk_docs.

The log program is persistent, and it can hold open a persistent HTTP/1.1
connection to CouchDB, so it should be pretty smooth.