Monday, 23 September 2019

Slicing giant files

Use 'dd' to copy (in this case 3) blocks of data from the giant file to the output:

dd  if=giant.json count=3 of=output.json

trucate could also work.

For JSON jq is a great tool:

Here we are catting json and selecting entries with messages length > 0
cat ~/giant.json  | jq -cn --stream 'fromstream(1|truncate_stream(inputs))' | jq 'select((.messages | length) > 0)

No comments:

Post a Comment