Showing posts with label file. Show all posts
Showing posts with label file. Show all posts

Monday, 23 September 2019

Slicing giant files



Use 'dd' to copy (in this case 3) blocks of data from the giant file to the output:

dd  if=giant.json count=3 of=output.json

trucate could also work.

For JSON jq is a great tool:

Here we are catting json and selecting entries with messages length > 0
cat ~/giant.json  | jq -cn --stream 'fromstream(1|truncate_stream(inputs))' | jq 'select((.messages | length) > 0)