Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
B
bcom-tp-etl-transformation-pipelines
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
general
bcom-tp-etl-transformation-pipelines
Commits
94bc9443
Commit
94bc9443
authored
Oct 07, 2023
by
Erly Villaroel
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Call al SP
parent
05e1b5a3
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
7 additions
and
5 deletions
+7
-5
Generation.py
dags/components/Generation.py
+0
-2
Transformation.py
dags/components/Transformation.py
+7
-3
No files found.
dags/components/Generation.py
View file @
94bc9443
...
@@ -144,8 +144,6 @@ def generate_and_deploy(command: str, intern_conn, params: Dict[str, Any], timez
...
@@ -144,8 +144,6 @@ def generate_and_deploy(command: str, intern_conn, params: Dict[str, Any], timez
dataframe
.
to_csv
(
tmp_file
,
sep
=
delimiter
,
index
=
False
,
mode
=
'a'
,
header
=
header
)
dataframe
.
to_csv
(
tmp_file
,
sep
=
delimiter
,
index
=
False
,
mode
=
'a'
,
header
=
header
)
except
StopIteration
:
except
StopIteration
:
break
break
list_outputs
=
params
[
"s3_params"
]
list_outputs
=
params
[
"s3_params"
]
size
=
os
.
path
.
getsize
(
tmp_file
)
size
=
os
.
path
.
getsize
(
tmp_file
)
for
output
in
list_outputs
:
for
output
in
list_outputs
:
...
...
dags/components/Transformation.py
View file @
94bc9443
...
@@ -98,9 +98,13 @@ def transformations(xcom_commands: str, intern_conn, timezone: str, **kwargs):
...
@@ -98,9 +98,13 @@ def transformations(xcom_commands: str, intern_conn, timezone: str, **kwargs):
with
engine
.
connect
()
as
connection
:
with
engine
.
connect
()
as
connection
:
for
command
in
commands
:
for
command
in
commands
:
if
any
(
command
.
startswith
(
palabra
)
or
command
.
startswith
(
palabra
.
lower
())
for
palabra
in
not_procedure
):
if
any
(
command
.
startswith
(
palabra
)
or
command
.
startswith
(
palabra
.
lower
())
for
palabra
in
not_procedure
):
print
(
2
)
logger
.
info
(
f
"Ejecutando comando de transformación: {command}"
)
logger
.
info
(
f
"Ejecutando comando de transformación: {command}"
)
_
=
connection
.
execute
(
command
)
_
=
connection
.
execute
(
command
)
else
:
logger
.
info
(
f
"Generando llamada al procedure según bd para: {command}"
)
command
=
intern_conn
.
generate_sql_procedure
(
command
)
logger
.
debug
(
f
"EJECUTANDO FINAL PROCEDURE: {command}"
)
_
=
connection
.
execute
(
command
)
end_process_datetime
=
datetime_by_tzone
(
timezone
)
.
strftime
(
'
%
d/
%
m/
%
Y
%
H:
%
M:
%
S'
)
end_process_datetime
=
datetime_by_tzone
(
timezone
)
.
strftime
(
'
%
d/
%
m/
%
Y
%
H:
%
M:
%
S'
)
task
.
xcom_push
(
key
=
"END_PROCESS_DATETIME_"
+
str
(
task
.
map_index
),
value
=
end_process_datetime
)
task
.
xcom_push
(
key
=
"END_PROCESS_DATETIME_"
+
str
(
task
.
map_index
),
value
=
end_process_datetime
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment