230 Commits

Author SHA1 Message Date
0d4da4132d
Remove broken graph and report functions
They needed the old categorizer. They'll have to be reinstated at a
later date with the new workflow.
2024-03-17 20:13:49 +00:00
cd9f6ba536
Remove unused categorizer and yaml config 2024-01-22 23:02:30 +00:00
29c5206638
Remove JSON dump 2024-01-22 22:02:50 +00:00
f44c04d678
Ignore SQLite DB file 2024-01-22 22:02:43 +00:00
2267bb29b3
Update dependencies
Now using the PostgreSQL client library from the distribution package.
2024-01-22 22:02:38 +00:00
abf82176e5
Improve error logging on incorrect parsing 2024-01-22 22:02:31 +00:00
c6a13cc83a
Allow multi-category transactions
through the use of tags. Instead of failing when categorizing a
transaction which already has a category, add the new category as a tag
for that transaction.

Issue #2
2024-01-22 22:02:25 +00:00
a7b74237aa
Fix rule's values limits
They are supposed to catch also the explicit values, so the limits are
inclusive.
2024-01-22 22:02:19 +00:00
95eff24418
Remove null rules from Categorizer
and allow for multiple null rules on the Nullifier.

Also create an empty list on the Nullifier constructor to simplify the
logic afterwards, in the case no "null" rule exist.
2024-01-22 22:02:09 +00:00
f966868736
Update Interactive to new DB interface
Take the opportunity to improve the loop structure, so fix current
out-of-bound error.
2024-01-22 22:02:00 +00:00
2a68ddd152
[Refactor] Streamline the output of nordigen
Clean the output of the nordigen to return a list of transactions,
instead of its internal structure.
2024-01-22 22:01:51 +00:00
1cfc8dbe38
Create EUA and request user access permission
Move the token creation/renewal code into the NordigenClient and remove
external access. The NordigenClient now also takes the DB client in the
constructor.

While creating the unit test, noticed that the nordigen mocks for
downloaded transactions could match the simple transactions mock, thus
helping in the online download command.
Also created the nordigen.NordigenClient mock, with the used methods
mocked and raising a NotImplement when a new endpoint is requested.
2024-01-22 22:01:38 +00:00
420a6cdfaa
Nordigen token generation/refresh logic
Adds a new table, nordigen, with the access and refresh token, along
with their validity.
The Export/Import test would raise an integrety with the use of a real
DB and the export of the transaction IDs, so add a try-except block to
the database session to catch the error and re-raise an ImportError.
2024-01-22 21:59:24 +00:00
ea546fc2df
Fix typing warning on parsers.py 2024-01-22 21:57:16 +00:00
d11f753aa0
[Fix] Eager loading for subclasses
and CategoryGroup import/export.
2024-01-22 21:56:34 +00:00
e6622d1e19
Full backup creation and import commands
Using the same logic as the single Export/Import commands, implement the
entire backup command by exporting all the serializable classes into a
single json file.
To select the correct class upon import, save a new property on the
backup json, the class_, which contains the name of the class to be
imported.

Fix the note serialization.
2024-01-22 21:56:17 +00:00
2cf0ba4374
[Export] Remove pickle format
Pickling directly from a DB object will also save DB related attributes,
namely the state of the object. When exporting values from a DB select,
they will be in detached state, which means on the import, the session
will not insert the object back into the DB, it already assumes they are
there.
2024-01-22 21:53:49 +00:00
6574165f8f
Complete export->import cycle
Add the transaction ID to the export of transactions and enable the
import of SplitTransactions. The ID should function as a stable
transaction identifier.
Fix tag import on transactions.
Fix rules import of date for DBs that need datetime.date format.
2024-01-22 21:52:00 +00:00
42d84b02f4
Removes unnecessary FK from Rules __init__
The FK are correctly initialized when the rules are created as part of
the base category/tag.
Also removes the name from the CategorySchedule, same logic applies.
2024-01-22 21:51:49 +00:00
729e15d4e8
Adds ImportFailedError for non-serializable types 2024-01-22 21:51:02 +00:00
638b833c74
ImportCommand and Serializable types
The new command ImportCommand takes a Serializable type, from which it
can call the deserialize method to generate a DB ORM type. The
Serializable interface also declares the serialize method.

(De)serialization moved to the ORM types, due to the inability to
properly use overloading.
Possible improvement for the future is to merge serialization
information on JSONDecoder/Encoder classes.

Adds a MockClient with the in-memory SQLite DB which can be used by
tests.
Most types export/import functionally tested using two DBs and comparing
entries.
2024-01-22 21:49:56 +00:00
21099909c9
Allow Nordigen parameter in Bank constructor
and remove Nordigen FK to Bank from Nordigen constructor. It is auto
generated upon DB insertion.
2024-01-22 21:47:47 +00:00
3698654715
[DB missing] Remove SQLAlchemy Enum 2024-01-22 21:47:47 +00:00
271130b107
[DB] Rules collection using list instead of set
There pains of using a set as the aggregation of rules overweight the
small advantage. While it would make detecting similar rules faster, the
use of a collection type not inherently supporting by JSON brings some
issues in the serialization.
2024-01-22 21:47:47 +00:00
ec22b5e5bd
[DB][Refactor] Compact the category selector
The `CategorySelector` was possibly added to be incremented with other
attributes. However, since none other that the selector enum is used at
the moment, it is only adding unnecessary cluter.
The category selector value is moved to the parent
`TransactionCategory`.
2024-01-22 21:47:47 +00:00
48fae5483d
[Fix] Select statement dangling session
The `Client` was never closing the `Session` in `DatabaseSession`, which
meant that tests that compared transactions and banks would work since
attributes loaded only on select would still work.
However, this was not the intended behavior. Calling select from the
client with using sessions should detach objects completely from a DB
session. Therefore, attributes on Transaction, TransactionCategory and
Bank are now loaded on a join (lazy parameter).
2024-01-22 21:47:47 +00:00
1628f6c1ea
Update to python 3.11
Add pythonpath option to pytest so that VSCode testing tool successfully
imports the mocks dir.
2024-01-22 21:47:47 +00:00
60c7028f0b
Expand serializer to more types and test 2024-01-22 21:47:47 +00:00
6057ec59b4
[Deprecate] CSV export/import removed
CSV is not a good format, since it does not have a good support for
complex types, e.g. lists.
It served when the exported format was only a linear transaction, but it
is not worth the extra work around it anymore.
2024-01-22 21:47:47 +00:00
a527fa5e1d
[Fix] Export error on DB access
Export operations were using an old Client select methods signature to
pass in the opened session.
Session is no longer needed to a simple select.
2024-01-22 21:47:47 +00:00
a355ec3642
Initial work on a full backup option
Creates the Backup command and a general serializer.
2024-01-22 21:47:47 +00:00
0a0bc1baa0
Fix possible null access
Detected through the typing warning.
2024-01-22 21:47:47 +00:00
ffc3d477e2
Fix typing for model.py and hash functions
Set unsafe_hash parameter of dataclasses to true on types that need to
be hashed.
The __hash__ method won't be automatically generated since the types
can't be declared as frozen.
2024-01-22 21:45:49 +00:00
01df97ed46
back_populates option on category relationship
Due to the use of the dataclasses mixin on the SQLAlchemy types, a
back_populates creates a RecursiveError when comparing two types. This
occurs because the dataclass will overwrite the __eq__ operator, and it
doesn't know when to stop comparing relationships.

Removing the dataclasses isn't the best approach, since then __init__,
__eq__ and __repr__ methods would have to be added to all types. Thus
the solution was to remove the relationship on the child (on a
one-to-one relationship) from the __eq__ operation, with the use of the
compare parameter.

Took the opportunity to define more logical __init__ methods on the
`Rule` and child classes.
Also revised the parameter options on some DB types.
2024-01-22 21:45:49 +00:00
bdd7cac4be
Download operation moves to Loader interface
to load extracted transactions to database.
2023-04-30 00:48:03 +01:00
13c783ca0e
[Refactor] Clean out old DB client class
Swap almost all remaining calls to the old postgresql only DB class with
the new DB client.

Warning! Some operations are currently not implement, such as setting
category schedules and dismantling links.

`update` and `delete` methods added to DB `Client`.
2023-04-30 00:38:15 +01:00
da44ba5306
[DB] Change Bank PK 2023-04-29 20:28:39 +01:00
94322ae542
Generalize insert in new Database interface
Move over all inserts on manager.py to new interface.
2023-04-29 20:20:20 +01:00
78ff6faa12
DatabaseSession wrapper for orm.Session
For DB connections that want to keep a session alive, there's a new
`DatabaseSession` class that holds a SQLAlchemy session inside and
offers methods similar to the `Database` class.

The `Database` moves to use the `DatabaseSession` to remove duplicated
code.
2023-04-29 20:20:20 +01:00
9f39836083
[Refactor] Export ops move to Client interface
Introduce new `Manager` parameter database to use new DB client API.
Fix TransactionCategory format.
Fix Client kwargs type.
2023-04-29 20:20:20 +01:00
ad62317e56
[Fix] Fix sessionless database client
The database `Client` wasn't working correcly when no session was
passed, as the inserted transactions would still be bound to the newly
created session inside.
Creates a copy of the input transactions to insert on the DB.
2023-04-29 20:20:20 +01:00
e7abae0d17
[Refactor] Database client interface changed
`add` method replaced with `insert`.
`insert` and `select` implemented for new database base class.
Database unit test added.

Due to SQLite implementation of the primary key autoinc, the type of the
IDs on the database for SQLite changed to Integer.
https://www.sqlite.org/autoinc.html
2023-04-29 20:20:20 +01:00
9c7c06c181
[DB] Schema unified on pfbudget 2023-04-29 20:20:16 +01:00
d6779b40ea
Alembic added to poetry 2023-04-23 16:37:11 +01:00
761720b712
[Fix] Add typing information to tests
And fix caught errors.
2023-04-23 00:51:22 +01:00
541295ef05
Adds Loader, DatabaseLoader and (Db)Client
Adds unit test to test interfaces.
2023-04-23 00:21:44 +01:00
14131f50f9
[Refactor] Rename the PostgreSQL file 2023-04-23 00:00:45 +01:00
c4d2d5a378
Install the psycopg binary along with the package 2023-04-22 23:55:07 +01:00
ef9ca1d1ea
Adds the mypy linter 2023-04-22 22:21:00 +01:00
b1de4d519a
[Refactor] Categorizer now implements Transform
Test adapted for new interface.
Exchanged manual input functionality for throwing an exception.
Removed timer at transformer level.
2023-04-22 22:21:00 +01:00