77 Commits

Author SHA1 Message Date
95eff24418
Remove null rules from Categorizer
and allow for multiple null rules on the Nullifier.

Also create an empty list on the Nullifier constructor to simplify the
logic afterwards, in the case no "null" rule exist.
2024-01-22 22:02:09 +00:00
1cfc8dbe38
Create EUA and request user access permission
Move the token creation/renewal code into the NordigenClient and remove
external access. The NordigenClient now also takes the DB client in the
constructor.

While creating the unit test, noticed that the nordigen mocks for
downloaded transactions could match the simple transactions mock, thus
helping in the online download command.
Also created the nordigen.NordigenClient mock, with the used methods
mocked and raising a NotImplement when a new endpoint is requested.
2024-01-22 22:01:38 +00:00
420a6cdfaa
Nordigen token generation/refresh logic
Adds a new table, nordigen, with the access and refresh token, along
with their validity.
The Export/Import test would raise an integrety with the use of a real
DB and the export of the transaction IDs, so add a try-except block to
the database session to catch the error and re-raise an ImportError.
2024-01-22 21:59:24 +00:00
e6622d1e19
Full backup creation and import commands
Using the same logic as the single Export/Import commands, implement the
entire backup command by exporting all the serializable classes into a
single json file.
To select the correct class upon import, save a new property on the
backup json, the class_, which contains the name of the class to be
imported.

Fix the note serialization.
2024-01-22 21:56:17 +00:00
2cf0ba4374
[Export] Remove pickle format
Pickling directly from a DB object will also save DB related attributes,
namely the state of the object. When exporting values from a DB select,
they will be in detached state, which means on the import, the session
will not insert the object back into the DB, it already assumes they are
there.
2024-01-22 21:53:49 +00:00
729e15d4e8
Adds ImportFailedError for non-serializable types 2024-01-22 21:51:02 +00:00
638b833c74
ImportCommand and Serializable types
The new command ImportCommand takes a Serializable type, from which it
can call the deserialize method to generate a DB ORM type. The
Serializable interface also declares the serialize method.

(De)serialization moved to the ORM types, due to the inability to
properly use overloading.
Possible improvement for the future is to merge serialization
information on JSONDecoder/Encoder classes.

Adds a MockClient with the in-memory SQLite DB which can be used by
tests.
Most types export/import functionally tested using two DBs and comparing
entries.
2024-01-22 21:49:56 +00:00
271130b107
[DB] Rules collection using list instead of set
There pains of using a set as the aggregation of rules overweight the
small advantage. While it would make detecting similar rules faster, the
use of a collection type not inherently supporting by JSON brings some
issues in the serialization.
2024-01-22 21:47:47 +00:00
ec22b5e5bd
[DB][Refactor] Compact the category selector
The `CategorySelector` was possibly added to be incremented with other
attributes. However, since none other that the selector enum is used at
the moment, it is only adding unnecessary cluter.
The category selector value is moved to the parent
`TransactionCategory`.
2024-01-22 21:47:47 +00:00
60c7028f0b
Expand serializer to more types and test 2024-01-22 21:47:47 +00:00
6057ec59b4
[Deprecate] CSV export/import removed
CSV is not a good format, since it does not have a good support for
complex types, e.g. lists.
It served when the exported format was only a linear transaction, but it
is not worth the extra work around it anymore.
2024-01-22 21:47:47 +00:00
a527fa5e1d
[Fix] Export error on DB access
Export operations were using an old Client select methods signature to
pass in the opened session.
Session is no longer needed to a simple select.
2024-01-22 21:47:47 +00:00
a355ec3642
Initial work on a full backup option
Creates the Backup command and a general serializer.
2024-01-22 21:47:47 +00:00
bdd7cac4be
Download operation moves to Loader interface
to load extracted transactions to database.
2023-04-30 00:48:03 +01:00
13c783ca0e
[Refactor] Clean out old DB client class
Swap almost all remaining calls to the old postgresql only DB class with
the new DB client.

Warning! Some operations are currently not implement, such as setting
category schedules and dismantling links.

`update` and `delete` methods added to DB `Client`.
2023-04-30 00:38:15 +01:00
94322ae542
Generalize insert in new Database interface
Move over all inserts on manager.py to new interface.
2023-04-29 20:20:20 +01:00
9f39836083
[Refactor] Export ops move to Client interface
Introduce new `Manager` parameter database to use new DB client API.
Fix TransactionCategory format.
Fix Client kwargs type.
2023-04-29 20:20:20 +01:00
e7abae0d17
[Refactor] Database client interface changed
`add` method replaced with `insert`.
`insert` and `select` implemented for new database base class.
Database unit test added.

Due to SQLite implementation of the primary key autoinc, the type of the
IDs on the database for SQLite changed to Integer.
https://www.sqlite.org/autoinc.html
2023-04-29 20:20:20 +01:00
14131f50f9
[Refactor] Rename the PostgreSQL file 2023-04-23 00:00:45 +01:00
b1de4d519a
[Refactor] Categorizer now implements Transform
Test adapted for new interface.
Exchanged manual input functionality for throwing an exception.
Removed timer at transformer level.
2023-04-22 22:21:00 +01:00
2c7c527ea9
[Refactor] Creates Tagger transformer
Move the tag rules based transformer to its own class.
Verified with unit tests.
2023-04-22 22:21:00 +01:00
2843c66453
[Refactor] Creates nullifier transformer
Removes nullifier code from Categorizer.
Fixes bug in nullifier rules assertion.
2023-04-22 22:21:00 +01:00
d3c36e30d5
[Refactor] Creates Transform interface
Moves categorizer into transform module.
Puts the categorizer under unit tests.
2023-04-22 22:21:00 +01:00
d6cbb10c47
[Refactor] Move NordigenCredentials to nordigen
Create manager that for now only has the default values from the .env
file.
2023-04-22 22:20:56 +01:00
8710722eb2
[Refactor] Split extractor from Nordigen Client 2023-04-22 22:20:53 +01:00
eb5c1781f0
Get the PSD2 client under testing
Having a comprehension unit testing allows for peace of mind
refactoring.
Changed the PSD2Client constructor and extract method to accept
credentials and banks, respectively.

Create Exception tree for the Extract module.
Adds mocks to the NordigenClient requests' methods.
2023-04-15 13:57:46 +01:00
184628dcfc
Rename Nordigen to PSD2 2023-04-13 18:55:33 +01:00
26cf921a6f
Fix flake8 warnings 2023-04-13 18:38:07 +01:00
d7bdafa62a
Rename input -> extract
Trying to follow the common ETL pattern naming.
2023-04-12 19:51:10 +01:00
2cbf00a939
CLI now allow --no-nulls flag on categorize
Null categorization, which is a major time slog, may now be disabled.
2023-04-12 18:20:57 +01:00
ed2dda63e9
Allows using rules for the nullying step 2023-02-23 23:24:01 +00:00
1a774e3769
Adds get all transactions operation 2023-02-23 23:23:19 +00:00
dd724b6c28
Export in .csv
Importing is not supported, since there's no way to represent a Null
field in .csv
2023-02-23 23:21:54 +00:00
6f68d971ee
Clear up forge/dismantle logic 2023-02-11 22:48:04 +00:00
da348c4ffb
[Fix] Splitted categories were not being commited 2023-01-29 23:48:27 +00:00
abff76ad4e
[Interactive] Finish split command 2023-01-29 21:26:22 +00:00
13709b7a04
[Interactive] Adds new tag 2023-01-29 21:14:49 +00:00
fd24ac3318
Remove unnecessary __init__.pys
and import using submodules in __main__.py
2023-01-23 23:52:20 +00:00
36e7f84bd9
Interactive categorization moved to __main__.py
It didn't make sense to have it inside the manager, it should only be
used to process commands and its paramaters.
2023-01-23 23:31:38 +00:00
5235fcdfc3
Adds business logic for split transaction
Moves split member to be part of parent class and removes it from the
init.
2023-01-23 23:03:42 +00:00
ae0a195069
Split tables per different schemas 2023-01-23 19:36:07 +00:00
8760f5a0a4
Export/Import categories and groups 2023-01-23 00:06:36 +00:00
dd0aaa01b8
Export/import for banks 2023-01-22 23:42:32 +00:00
fd6793b4f4
Turned on type checking
and as a result, had to fix a LOT of minor potential future issue.
It also reorders and clears unused imports.

When exporting transactions, it will sort by date.
2023-01-22 20:44:05 +00:00
d4b5f1f11a
Rule inheritance
Both rules, categorries and tags, now derive from the rule base type.
This clears up some type definitions.
2023-01-22 20:22:46 +00:00
6110858d48
Extend export/import to rules
Removes additional bank/all options from the transactions export command
line.
Deletes the brief lived CSV class.
This patch start using pickle for simple export/import, other options
can be added later. An issue with the .csv is the lack of a Null field.
Moves logic to Manager, it is simple enough.
2023-01-15 23:06:20 +00:00
c42a399d3d
Adds the import operation and a timer
to the categorization. We can now import transactions from a csv file,
and later automatically categorize them all.
2023-01-10 23:45:09 +00:00
c37e7eb37c
Readds manual categorization
Also fixes a categorization bug in the Manager, in the DB client method.
2023-01-10 21:32:08 +00:00
86afa99217
Finish the remaining Nordigen operations
from the Manager POV and the update on the argparses.
Also clears unnecessary methods from the DB client interface.
Better assert information on the __main__.py
2023-01-08 19:41:07 +00:00
9b45ee4817
Update the export operation
to work with the Manager.
Also removes the run method from the runnable.py, since everything is
done in the __main__.py file of the pfbudget module.
2023-01-08 19:41:07 +00:00