[Swlug] help with text manipulation

[Mark Einon]

Hi there,

This is simple to do in python (using the Pandas lib), I do it on a
daily basis. A jupyter notebook in html and ipynb is attached to show
how (you can use google's colabatory to edit and run the notebook - 
https://colab.research.google.com/)

Mark


On Wed, 2018-06-20 at 03:26 +0100, bascule via Swlug wrote:
> this is a request for help, it's not actually linux related except
> inasmuch as 
> the solution may utilise a linux distro.
> 
> i have some json files i wish to edit, their being json files is
> irrelevant to 
> me, i'm not doing json stuff, these files come to me as they are and i
> need/
> wish to futz with them.
> 
> they have objects in them such as:
> 
> {
> "ConnectedObjects" : "",
> "ExtraData" : "",
> "FormID" : "004366",
> "FormID_dec" : 184566630,
> "PlotPath01FormID_dec" : 0,
> "PlotPath01Plugin" : "",
> "PlotPath02FormID_dec" : 0,
> "PlotPath02Plugin" : "",
> "PlotPath03FormID_dec" : 0,
> "PlotPath03Plugin" : "",
> "PlotPlanFormID" : "002151",
> "PlotPlanFormID_dec" : 385884497,
> "PlotPlanPlugin" : "SimSettlements_AddOnPack_Defences_Brae.esp",
> "PlotStage" : 0,
> "PlotVIPStoryFormID_dec" : 0,
> "PlotVIPStoryPlugin" : "",
> "RemoveHavok" : 0,
> "Scale" : 1.0,
> "idx" : 1471,
> "isActor" : 0,
> "isAssignable" : 1,
> "isCreated" : 1,
> "isFarmAnimal" : 0,
> "isFood" : 0,
> "isGenerator" : 0,
> "isPowered" : 1,
> "isTamedCreature" : 0,
> "isTurret" : 0,
> "isWater" : 0,
> "name" : "Martial Plot - 2 x 2",
> "plugin_name" : "SimSettlements.esm",
> "posX" : -35624.8515625,
> "posY" : 59097.19140625,
> "posZ" : 3894.25146484375,
> "rotX" : 0.0,
> "rotY" : 0.0,
> "rotZ" : -188.41806030273437
> },
> 
> whitespace may have been lost a bit in the copy/paste, 
> my files have thousands of such objects, i need to find all objects
> where all 
> of the the keys FormID,posX,posY,posZ,rotX,rotY,rotZ are the same as
> another 
> object's and delete all but one of the 'duplicated' objects regardless
> of 
> whether any other key is the same.  i need to do this for objects in
> the same 
> file, and if possible for objects in two different files.
> suggestions so far include learning python, and using jq, which man
> page i 
> have read and yes it seems to be a tool of sufficient power maybe, but
> i fear 
> i lack the chops to work out how to do all the necessary streaming
> and 
> filtering, 
> any other suggestions would be useful, 
> 
> bascule
> 
> 
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dropdups_python.ipynb
Type: application/x-ipynb+json
Size: 4812 bytes
Desc: not available
URL: <http://mailman.lug.org.uk/pipermail/swlug/attachments/20180620/cd436770/attachment-0001.bin>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/swlug/attachments/20180620/cd436770/attachment-0001.html>