Jul 17 16:23:59.666459 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 17 16:23:59.666472 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 17 16:23:59.666481 ip-10-0-137-172 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 17 16:23:59.666931 ip-10-0-137-172 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 17 16:24:09.879370 ip-10-0-137-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 17 16:24:09.879384 ip-10-0-137-172 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b2bff4b3acc0413294c6fb0811306ff4 --
Jul 17 16:26:05.907300 ip-10-0-137-172 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 16:26:06.249992 ip-10-0-137-172 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 16:26:06.249992 ip-10-0-137-172 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 16:26:06.249992 ip-10-0-137-172 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 16:26:06.249992 ip-10-0-137-172 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 16:26:06.249992 ip-10-0-137-172 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 16:26:06.250930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.250855    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 16:26:06.255936 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255921    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.255936 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255936    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255940    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255943    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255946    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255949    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255952    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255955    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255958    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255960    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255963    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255965    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255969    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255971    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255974    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255976    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255979    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255982    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255984    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255987    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255989    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.256001 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.255998    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256000    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256003    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256006    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256009    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256012    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256014    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256017    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256020    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256022    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256025    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256027    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256030    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256032    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256034    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256037    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256039    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256055    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256058    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256062    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.256670 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256064    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256067    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256069    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256072    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256074    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256077    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256080    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256083    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256085    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256088    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256090    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256093    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256096    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256129    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256330    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256339    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256343    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256350    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256356    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.257288 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256361    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256367    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256374    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256381    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256385    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256390    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256395    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256406    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256411    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256416    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256421    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256437    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256441    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256446    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256450    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256454    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256459    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256463    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256467    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256472    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.257790 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256481    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256486    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256490    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256498    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256502    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256507    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256512    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256518    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256522    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256526    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256530    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256540    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256544    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256550    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256556    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256561    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256565    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256570    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.256574    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.258271 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257791    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257801    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257804    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257807    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257810    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257813    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257816    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257818    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257821    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257826    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257830    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257833    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257835    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257838    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257841    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257843    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257846    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257848    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257851    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.258735 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257853    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257856    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257858    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257865    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257868    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257870    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257873    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257876    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257878    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257881    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257883    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257886    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257888    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257891    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257894    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257898    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257900    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257903    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257905    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.259225 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257908    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257910    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257913    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257915    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257918    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257920    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257923    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257926    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257928    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257931    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257934    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257936    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257940    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257943    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257946    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257948    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257951    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257954    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257958    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.259684 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257961    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257965    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257968    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257970    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257974    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257976    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257979    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257981    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257985    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257987    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257990    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257992    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257995    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.257998    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258002    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258005    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258007    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258010    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258012    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258015    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.260165 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258018    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258021    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258024    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258026    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258029    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258031    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258034    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258036    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258039    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258058    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258061    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258064    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258067    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258069    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258072    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258074    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258077    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258080    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258083    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258086    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.260657 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258088    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258091    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258464    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258469    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258472    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258474    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258478    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258480    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258483    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258486    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258488    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258491    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258493    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258497    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258500    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258503    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258505    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258507    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258510    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258513    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.261148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258515    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258518    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258520    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258523    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258525    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258528    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258531    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258533    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258536    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258538    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258541    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258543    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258546    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258548    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258550    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258555    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258559    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258562    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258564    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.261637 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258568    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258571    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258574    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258576    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258579    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258582    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258585    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258588    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258591    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258593    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258596    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258599    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258601    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258604    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258607    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258609    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258611    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258614    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258617    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258620    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.262104 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258622    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258627    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258630    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258633    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258635    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258640    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258643    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258645    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258648    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258650    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258653    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258655    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258658    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258661    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258663    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258665    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258668    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258670    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258673    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.262591 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258676    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258678    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258681    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258683    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258685    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258688    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258691    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258694    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258696    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258699    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258702    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258704    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258708    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258711    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258714    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258717    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258720    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258722    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258724    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258727    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.263108 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258729    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.258732    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258798    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258805    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258811    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258815    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258821    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258824    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258829    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258833    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258836    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258839    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258843    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258846    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258849    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258852    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258855    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258857    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258861    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258864    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258879    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258882    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258884    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258888    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 16:26:06.263580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258891    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258895    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258898    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258901    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258904    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258907    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258910    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258913    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258916    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258919    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258923    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258926    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258928    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258931    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258934    2561 flags.go:64] FLAG: --enable-server="true"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258937    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258941    2561 flags.go:64] FLAG: --event-burst="100"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258944    2561 flags.go:64] FLAG: --event-qps="50"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258947    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258950    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258953    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258956    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258959    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258962    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258965    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 17 16:26:06.264204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258967    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258970    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258974    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258976    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258979    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258982    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258985    2561 flags.go:64] FLAG: --feature-gates=""
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258988    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258991    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258994    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.258997    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259000    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259003    2561 flags.go:64] FLAG: --help="false"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259006    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259009    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259012    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259014    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259018    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259021    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259024    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259027    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259029    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259032    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 16:26:06.264791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259035    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259039    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259052    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259056    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259058    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259061    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259064    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259067    2561 flags.go:64] FLAG: --lock-file=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259070    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259073    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259078    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259083    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259086    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259089    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259092    2561 flags.go:64] FLAG: --logging-format="text"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259095    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259098    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259101    2561 flags.go:64] FLAG: --manifest-url=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259104    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259108    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259111    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259115    2561 flags.go:64] FLAG: --max-pods="110"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259118    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259120    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259123    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 16:26:06.265382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259126    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259129    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259132    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259135    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259142    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259144    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259148    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259151    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259155    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259161    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259164    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259167    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259170    2561 flags.go:64] FLAG: --port="10250"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259173    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259175    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-078871a78228f02b7"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259178    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259181    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259184    2561 flags.go:64] FLAG: --register-node="true"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259188    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259192    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259194    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259197    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259200    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259203    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259207    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 16:26:06.265985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259210    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259213    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259216    2561 flags.go:64] FLAG: --runonce="false"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259219    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259222    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259225    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259228    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259231    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259235    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259238    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259241    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259243    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259246    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259249    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259252    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259256    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259259    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259262    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259267    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259270    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259272    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259276    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259279    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259282    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259285    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 16:26:06.266621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259289    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259292    2561 flags.go:64] FLAG: --v="2"
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259296    2561 flags.go:64] FLAG: --version="false"
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259300    2561 flags.go:64] FLAG: --vmodule=""
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259303    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259306    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259415    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259418    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259421    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259424    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259427    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259430    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259433    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259435    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259438    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259440    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259443    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259445    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259448    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259450    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259453    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.267223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259455    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259458    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259461    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259464    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259467    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259469    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259472    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259474    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259477    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259479    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259481    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259485    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259489    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259493    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259496    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259499    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259502    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259506    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259509    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.267758 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259512    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259515    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259517    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259519    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259522    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259524    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259527    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259530    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259532    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259535    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259537    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259540    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259542    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259547    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259550    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259552    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259556    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259558    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259560    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.268238 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259563    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259565    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259568    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259572    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259576    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259579    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259582    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259585    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259587    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259590    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259594    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259596    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259599    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259601    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259604    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259606    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259609    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259611    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259614    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.268702 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259616    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259620    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259623    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259625    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259628    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259630    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259633    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259635    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259638    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259641    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259643    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259647    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259649    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259651    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259654    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259656    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259659    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259662    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259665    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.269245 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259667    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259670    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259672    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259675    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259678    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259681    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.259684    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.259690    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266106    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266120    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266155    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266162    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266226    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266230    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266233    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.269786 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266236    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266240    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266243    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266246    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266248    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266251    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266254    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266256    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266259    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266262    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266264    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266267    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266269    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266271    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266274    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266276    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266279    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266281    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266284    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266287    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.270166 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266289    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266291    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266303    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266306    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266308    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266311    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266315    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266318    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266321    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266324    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266326    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266328    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266331    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266334    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266336    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266339    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266342    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266344    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266347    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266349    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.270662 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266352    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266354    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266357    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266359    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266362    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266364    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266367    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266370    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266372    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266375    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266379    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266383    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266386    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266388    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266391    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266393    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266396    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266399    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266401    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266405    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.271149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266407    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266410    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266412    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266415    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266417    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266420    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266423    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266425    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266429    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266432    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266435    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266438    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266440    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266443    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266445    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266448    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266450    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266452    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266455    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.271631 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266457    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266460    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266463    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266465    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266468    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266470    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266473    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266476    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266478    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266481    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266485    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266489    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266491    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266495    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266498    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266501    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.272107 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266503    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266509    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266660    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266666    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266669    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266672    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266675    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266677    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266680    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266682    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266685    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266688    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266690    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266693    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266695    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 16:26:06.272502 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266698    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266700    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266703    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266705    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266707    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266710    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266712    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266715    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266717    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266720    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266722    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266725    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266727    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266729    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266732    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266734    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266737    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266741    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266744    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 16:26:06.272924 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266747    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266751    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266755    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266758    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266761    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266764    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266767    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266770    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266773    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266775    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266778    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266781    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266784    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266787    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266790    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266793    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266796    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266798    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 16:26:06.273452 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266801    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266804    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266806    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266809    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266811    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266814    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266816    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266818    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266821    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266823    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266826    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266829    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266831    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266834    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266836    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266839    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266841    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266844    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266846    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266848    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 16:26:06.273867 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266851    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266853    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266856    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266859    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266861    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266863    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266866    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266869    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266871    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266874    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266876    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266879    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266881    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266883    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266886    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266888    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266891    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266893    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266895    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266898    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 16:26:06.274364 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266900    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266903    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266905    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266908    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266918    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266921    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266923    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266926    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:06.266928    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.266932    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.267028    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.268955    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.270257    2561 server.go:1019] "Starting client certificate rotation"
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.270382    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 16:26:06.274852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.270414    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 16:26:06.275215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.272739    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 16:26:06.276651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.276628    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 16:26:06.287097 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.287079    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 17 16:26:06.289119 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.289103    2561 log.go:25] "Validated CRI v1 image API"
Jul 17 16:26:06.289923 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.289906    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 16:26:06.293831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.293813    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e1f78887-0a14-46f2-85fa-0e4cc33b3f54:/dev/nvme0n1p4 e6a996b7-6885-4528-af19-09599d00c510:/dev/nvme0n1p3]
Jul 17 16:26:06.293894 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.293831    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 16:26:06.297928 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.297911    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 16:26:06.298665 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.298565    2561 manager.go:216] Machine: {Timestamp:2026-07-17 16:26:06.297437238 +0000 UTC m=+0.282524787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102800 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec251fec5f9e8e8fc6e6f62240436064 SystemUUID:ec251fec-5f9e-8e8f-c6e6-f62240436064 BootID:b2bff4b3-acc0-4132-94c6-fb0811306ff4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:cc:17:6f:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:cc:17:6f:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:af:b6:78:07:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 16:26:06.298665 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.298663    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 16:26:06.298766 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.298736    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 16:26:06.301319 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.301289    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 16:26:06.301446 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.301322    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 16:26:06.302077 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.302067    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 16:26:06.302118 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.302079    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 16:26:06.302118 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.302086    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 16:26:06.303040 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.303029    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 16:26:06.304372 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.304362    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 16:26:06.308127 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308116    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 16:26:06.308237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308228    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 16:26:06.308271 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308259    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 16:26:06.308313 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308281    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 16:26:06.308313 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308295    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 16:26:06.308399 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308314    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 17 16:26:06.308399 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308335    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 16:26:06.308747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308735    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 16:26:06.308778 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.308753    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 16:26:06.310571 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.310552    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 16:26:06.312405 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.312391    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 16:26:06.312474 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.312420    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 16:26:06.313495 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313483    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313500    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313506    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313512    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313521    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313526    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313534    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313539    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313545    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 16:26:06.313546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313550    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 16:26:06.313767 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313559    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 16:26:06.313767 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.313564    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 16:26:06.314217 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.314207    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 16:26:06.314217 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.314215    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 16:26:06.316556 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.316531    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 16:26:06.317186 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.317155    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 16:26:06.317533 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.317456    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 16:26:06.321152 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.321132    2561 server.go:1268] "Started kubelet"
Jul 17 16:26:06.321242 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.321172    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 16:26:06.321392 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.321349    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 16:26:06.321461 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.321427    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 16:26:06.321899 ip-10-0-137-172 systemd[1]: Started Kubernetes Kubelet.
Jul 17 16:26:06.323059 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.323028    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 16:26:06.323352 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.323338    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 16:26:06.325879 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.325068    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18c320757b2064b6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-07-17 16:26:06.31783135 +0000 UTC m=+0.302918880,LastTimestamp:2026-07-17 16:26:06.31783135 +0000 UTC m=+0.302918880,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}"
Jul 17 16:26:06.326369 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.326352    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 16:26:06.327177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.327164    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gcg64"
Jul 17 16:26:06.329199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329179    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 16:26:06.329199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329188    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 16:26:06.329336 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.329269    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.329745 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329728    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 16:26:06.329820 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329747    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 16:26:06.329820 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329726    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 16:26:06.329937 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329881    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 16:26:06.329937 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.329891    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.330839    2561 factory.go:55] Registering systemd factory
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.330855    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.331199    2561 factory.go:153] Registering CRI-O factory
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.331229    2561 factory.go:223] Registration of the crio container factory successfully
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.331283    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.331325    2561 factory.go:103] Registering Raw factory
Jul 17 16:26:06.331878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.331341    2561 manager.go:1196] Started watching for new ooms in manager
Jul 17 16:26:06.332558 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.332253    2561 manager.go:349] Starting recovery of all containers
Jul 17 16:26:06.337265 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.337246    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gcg64"
Jul 17 16:26:06.339502 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.339328    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 16:26:06.339502 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.339445    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 16:26:06.344093 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.344080    2561 manager.go:354] Recovery completed
Jul 17 16:26:06.348344 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.348321    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.350364 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350348    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.350430 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350374    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.350430 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350384    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.350823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350808    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 16:26:06.350823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350820    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 16:26:06.350823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350823    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.350943 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.350833    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 16:26:06.351582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.351567    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.351643 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.351595    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.351643 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.351607    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.352357 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.352280    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-172.ec2.internal.18c320757d10c733  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-172.ec2.internal,UID:ip-10-0-137-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-172.ec2.internal,},FirstTimestamp:2026-07-17 16:26:06.350362419 +0000 UTC m=+0.335449951,LastTimestamp:2026-07-17 16:26:06.350362419 +0000 UTC m=+0.335449951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-172.ec2.internal,}"
Jul 17 16:26:06.352769 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.352758    2561 policy_none.go:49] "None policy: Start"
Jul 17 16:26:06.352813 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.352774    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 16:26:06.352813 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.352784    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 16:26:06.354691 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.354681    2561 policy_none.go:47] "Start"
Jul 17 16:26:06.389597 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.389580    2561 manager.go:337] "Starting Device Plugin manager"
Jul 17 16:26:06.389674 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.389607    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 16:26:06.389674 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.389616    2561 server.go:87] "Starting device plugin registration server"
Jul 17 16:26:06.389863 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.389852    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 16:26:06.389918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.389864    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 16:26:06.389985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.389973    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 16:26:06.390079 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.390067    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 16:26:06.390130 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.390079    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 16:26:06.390621 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.390537    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 16:26:06.390621 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.390582    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.490718 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.490695    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.492841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.491562    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.492841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.491591    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.492841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.491601    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.492841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.491630    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.499625 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.499613    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.499677 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.499632    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-172.ec2.internal\": node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.506433 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.506381    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 16:26:06.507595 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.507576    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 16:26:06.507676 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.507620    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 16:26:06.507676 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.507643    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 16:26:06.507824 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.507680    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 16:26:06.507990 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.507970    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.509257 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.509241    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 16:26:06.608352 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.608323    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.608352 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.608302    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"]
Jul 17 16:26:06.608535 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.608401    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.609851 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.609833    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.609935 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.609865    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.609935 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.609878    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.612180 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612168    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.612294 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.612340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612312    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.612884 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612867    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.612884 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612880    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.612974 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612896    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.612974 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612899    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.612974 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612914    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.612974 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.612914    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.615174 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.615160    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.615216 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.615184    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 16:26:06.615811 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.615789    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 16:26:06.615811 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.615810    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 16:26:06.615917 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.615819    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeHasSufficientPID"
Jul 17 16:26:06.631520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.631364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.631588 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.631533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.631588 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.631550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.632745 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.632729    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.636409 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.636390    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-172.ec2.internal\" not found" node="ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.708905 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.708884    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.732560 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.732637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.732637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.732637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.732637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deef81c0d8c2f4458539863e90bc2e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal\" (UID: \"3deef81c0d8c2f4458539863e90bc2e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.732769 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.732658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbb9b55189c2ca1c136d2bbd25cb800a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-172.ec2.internal\" (UID: \"bbb9b55189c2ca1c136d2bbd25cb800a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.809805 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.809752    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.915889 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:06.915845    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:06.935720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.935695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:06.939301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:06.939280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:07.018507 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:07.018467    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:07.119236 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:07.119167    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:07.225338 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:07.225287    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:07.270346 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.270310    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 16:26:07.270817 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.270472    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 16:26:07.317767 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.317736    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 16:26:07.325453 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:07.325424    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-172.ec2.internal\" not found"
Jul 17 16:26:07.335285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.335262    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 16:26:07.342651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.342615    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 16:21:06 +0000 UTC" deadline="2028-01-30 19:35:57.56650408 +0000 UTC"
Jul 17 16:26:07.342713 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.342653    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13491h9m50.223856128s"
Jul 17 16:26:07.395828 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.395762    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 16:26:07.429492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.429469    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:07.443405 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.443378    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 16:26:07.444295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.444280    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal"
Jul 17 16:26:07.454148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.454126    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 16:26:07.677310 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:07.677276    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb9b55189c2ca1c136d2bbd25cb800a.slice/crio-8647f930482249549af1451eb2e3b56d3641d26faffb42a6f667974a3af3a92d WatchSource:0}: Error finding container 8647f930482249549af1451eb2e3b56d3641d26faffb42a6f667974a3af3a92d: Status 404 returned error can't find the container with id 8647f930482249549af1451eb2e3b56d3641d26faffb42a6f667974a3af3a92d
Jul 17 16:26:07.682549 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:07.682511    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 16:26:07.690160 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:07.690137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deef81c0d8c2f4458539863e90bc2e7.slice/crio-784ac536f3c4c48ea0fa4b1c0a540f276da21c016231aadac236a70c4ae4eeb0 WatchSource:0}: Error finding container 784ac536f3c4c48ea0fa4b1c0a540f276da21c016231aadac236a70c4ae4eeb0: Status 404 returned error can't find the container with id 784ac536f3c4c48ea0fa4b1c0a540f276da21c016231aadac236a70c4ae4eeb0
Jul 17 16:26:08.215285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.215252    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 16:26:08.309438 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.309395    2561 apiserver.go:52] "Watching apiserver"
Jul 17 16:26:08.317531 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.317494    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 16:26:08.319406 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.319375    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-bg5rz","openshift-multus/network-metrics-daemon-4rl4t","openshift-ovn-kubernetes/ovnkube-node-mj6xf","kube-system/global-pull-secret-syncer-mtrtl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5","openshift-image-registry/node-ca-c2jl7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal","openshift-network-diagnostics/network-check-target-8gj7m","openshift-network-operator/iptables-alerter-cd47q","kube-system/konnectivity-agent-d48vc","kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mkf2m","openshift-dns/node-resolver-nxzbs","openshift-multus/multus-additional-cni-plugins-t7m4q"]
Jul 17 16:26:08.322188 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.322160    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.325200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.324560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.325200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.324798    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 16:26:08.325200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.324921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pfmv4\""
Jul 17 16:26:08.325200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.324962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 16:26:08.326892 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.326858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.327042 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.326901    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.327205 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.327183    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mxr8t\""
Jul 17 16:26:08.327280 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.327238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.329743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329250    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.329743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gjqr8\""
Jul 17 16:26:08.329743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 16:26:08.329743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329567    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 16:26:08.329743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329633    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 16:26:08.330094 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329869    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 16:26:08.330094 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329880    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.330094 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.329964    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:08.330094 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.330009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.330294 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.329885    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 16:26:08.333042 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.332815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.335186 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.335158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.335291 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.335247    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 16:26:08.335378 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.335295    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.335378 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.335315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.335506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.335323    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4qfp\""
Jul 17 16:26:08.337328 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.337306    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.337663 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.337644    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.338397 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.337897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 16:26:08.338397 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.338260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.338397 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.338301    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mn58z\""
Jul 17 16:26:08.340674 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.340220    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.340674 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.340516    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 16:26:08.341141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.340712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxfgw\""
Jul 17 16:26:08.341141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.340741    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.341363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.341280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.342380 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-systemd-units\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342380 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-bin\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342380 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-script-lib\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-kubelet-config\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-tmp-dir\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-var-lib-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-etc-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-env-overrides\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbaaadc-1b66-4386-859a-b4320f0352fc-agent-certs\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbaaadc-1b66-4386-859a-b4320f0352fc-konnectivity-ca\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-hosts-file\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-kubelet\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-systemd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-log-socket\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.342887 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xnx\" (UniqueName: \"kubernetes.io/projected/4f6799b7-2071-4df3-9fb4-33d2ef840c41-kube-api-access-j7xnx\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.342936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-device-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.343025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8klj\" (UniqueName: \"kubernetes.io/projected/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-kube-api-access-v8klj\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.343054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.343170    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 16:26:08.343593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.343126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zf9r\" (UniqueName: \"kubernetes.io/projected/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kube-api-access-5zf9r\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.344152 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.344031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.344296 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.344259    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:08.344810 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.344785    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 16:26:08.345007 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.344983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-netns\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345076 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-node-log\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345243 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-netd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345299 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-slash\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345352 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-ovn\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-config\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovn-node-metrics-cert\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-socket-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-registration-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345677    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-sys-fs\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-dbus\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.345930 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.345834    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.346726 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.346531    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nrrzb\""
Jul 17 16:26:08.351094 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.351076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.352993 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.352963    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 16:26:08.353135 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.353112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tg4hr\""
Jul 17 16:26:08.353402 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.353380    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:08.353505 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.353460    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:08.356465 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.356442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.360349 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.360274    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 16:26:08.360349 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.360306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4l6l\""
Jul 17 16:26:08.360523 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.360376    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 16:26:08.365744 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.365718    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb25s"
Jul 17 16:26:08.374042 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.374012    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb25s"
Jul 17 16:26:08.431389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.431350    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 16:26:08.446567 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-netns\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446567 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-netd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-multus-daemon-config\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-slash\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-ovn\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-config\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovn-node-metrics-cert\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.446800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-registration-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-host\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-os-release\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-system-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-dbus\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-systemd-units\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-script-lib\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.446985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-cni-binary-copy\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-socket-dir-parent\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-var-lib-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-etc-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447112 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-env-overrides\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbaaadc-1b66-4386-859a-b4320f0352fc-konnectivity-ca\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-hosts-file\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-slash\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-ovn\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/fa33cc83-47ff-45b3-9179-5752917f8b0c-kube-api-access-jqddf\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-netns\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.447698 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-dbus\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447747    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-config\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.447985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbaaadc-1b66-4386-859a-b4320f0352fc-konnectivity-ca\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-etc-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-kubernetes\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-os-release\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-hosts-file\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.448161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovnkube-script-lib\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-netns\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-systemd-units\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-var-lib-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-netd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-hostroot\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-registration-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-sys\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f6799b7-2071-4df3-9fb4-33d2ef840c41-env-overrides\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-var-lib-kubelet\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gglvk\" (UniqueName: \"kubernetes.io/projected/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-kube-api-access-gglvk\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-log-socket\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-log-socket\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8klj\" (UniqueName: \"kubernetes.io/projected/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-kube-api-access-v8klj\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.448584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zf9r\" (UniqueName: \"kubernetes.io/projected/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kube-api-access-5zf9r\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hhs\" (UniqueName: \"kubernetes.io/projected/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-kube-api-access-v9hhs\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-conf\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-node-log\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-node-log\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-openvswitch\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448916    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-serviceca\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-lib-modules\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-tmp\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/839e96e8-afc8-472e-923c-8eef62db3a83-iptables-alerter-script\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-socket-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.449347 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-socket-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-sys-fs\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.448705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-etc-selinux\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-tmp-dir\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-tuned\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-sys-fs\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxh7x\" (UniqueName: \"kubernetes.io/projected/3b436c25-3bb8-4029-be28-e05e46f89a92-kube-api-access-mxh7x\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-bin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-bin\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-kubelet-config\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/839e96e8-afc8-472e-923c-8eef62db3a83-host-slash\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-multus\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-conf-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-multus-certs\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-etc-kubernetes\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-tmp-dir\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.450192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbaaadc-1b66-4386-859a-b4320f0352fc-agent-certs\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-cni-bin\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83fb69b8-a939-4d03-a0f8-995040d1230b-kubelet-config\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-run\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-host\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-modprobe-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysconfig\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-cnibin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.449982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-kubelet\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-kubelet\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.450021    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-system-cni-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.451024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-k8s-cni-cncf-io\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7vf\" (UniqueName: \"kubernetes.io/projected/6521c29d-f99c-465b-980d-15eb1c948b26-kube-api-access-qs7vf\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.450117    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:08.950094696 +0000 UTC m=+2.935182235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-kubelet\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-systemd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kubelet-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xnx\" (UniqueName: \"kubernetes.io/projected/4f6799b7-2071-4df3-9fb4-33d2ef840c41-kube-api-access-j7xnx\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-device-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-systemd\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-cnibin\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec56f3a1-eaf2-413e-ab24-1cc938a27137-device-dir\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfkx\" (UniqueName: \"kubernetes.io/projected/839e96e8-afc8-472e-923c-8eef62db3a83-kube-api-access-5kfkx\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-host-run-ovn-kubernetes\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.451880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.450279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f6799b7-2071-4df3-9fb4-33d2ef840c41-run-systemd\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.453714 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.453687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbaaadc-1b66-4386-859a-b4320f0352fc-agent-certs\") pod \"konnectivity-agent-d48vc\" (UID: \"8fbaaadc-1b66-4386-859a-b4320f0352fc\") " pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.453991 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.453970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f6799b7-2071-4df3-9fb4-33d2ef840c41-ovn-node-metrics-cert\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.457361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.457338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8klj\" (UniqueName: \"kubernetes.io/projected/bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6-kube-api-access-v8klj\") pod \"node-resolver-nxzbs\" (UID: \"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6\") " pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.459060 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.459034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zf9r\" (UniqueName: \"kubernetes.io/projected/ec56f3a1-eaf2-413e-ab24-1cc938a27137-kube-api-access-5zf9r\") pod \"aws-ebs-csi-driver-node-42jg5\" (UID: \"ec56f3a1-eaf2-413e-ab24-1cc938a27137\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.461148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.461116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xnx\" (UniqueName: \"kubernetes.io/projected/4f6799b7-2071-4df3-9fb4-33d2ef840c41-kube-api-access-j7xnx\") pod \"ovnkube-node-mj6xf\" (UID: \"4f6799b7-2071-4df3-9fb4-33d2ef840c41\") " pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.516497 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.516381    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerStarted","Data":"784ac536f3c4c48ea0fa4b1c0a540f276da21c016231aadac236a70c4ae4eeb0"}
Jul 17 16:26:08.517856 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.517826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"bbb9b55189c2ca1c136d2bbd25cb800a","Type":"ContainerStarted","Data":"8647f930482249549af1451eb2e3b56d3641d26faffb42a6f667974a3af3a92d"}
Jul 17 16:26:08.551702 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-kubernetes\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-os-release\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-netns\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-netns\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-kubernetes\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-hostroot\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.551891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-sys\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-os-release\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-var-lib-kubelet\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-var-lib-kubelet\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-sys\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.551985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglvk\" (UniqueName: \"kubernetes.io/projected/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-kube-api-access-gglvk\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-hostroot\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hhs\" (UniqueName: \"kubernetes.io/projected/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-kube-api-access-v9hhs\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-conf\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-serviceca\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.552234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-lib-modules\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-tmp\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/839e96e8-afc8-472e-923c-8eef62db3a83-iptables-alerter-script\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-tuned\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxh7x\" (UniqueName: \"kubernetes.io/projected/3b436c25-3bb8-4029-be28-e05e46f89a92-kube-api-access-mxh7x\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-bin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/839e96e8-afc8-472e-923c-8eef62db3a83-host-slash\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-multus\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-conf-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-multus-certs\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-etc-kubernetes\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-conf\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-run\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.552825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-host\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-conf-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-multus-certs\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-etc-kubernetes\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.552954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-serviceca\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-modprobe-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysconfig\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-bin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-cnibin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-kubelet\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-system-cni-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-k8s-cni-cncf-io\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7vf\" (UniqueName: \"kubernetes.io/projected/6521c29d-f99c-465b-980d-15eb1c948b26-kube-api-access-qs7vf\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.553722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-systemd\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-cnibin\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/839e96e8-afc8-472e-923c-8eef62db3a83-host-slash\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfkx\" (UniqueName: \"kubernetes.io/projected/839e96e8-afc8-472e-923c-8eef62db3a83-kube-api-access-5kfkx\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysctl-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-multus-daemon-config\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-cni-multus\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-host\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-os-release\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-run\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-system-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-lib-modules\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-cni-binary-copy\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-var-lib-kubelet\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-socket-dir-parent\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/fa33cc83-47ff-45b3-9179-5752917f8b0c-kube-api-access-jqddf\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-host-run-k8s-cni-cncf-io\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.554596 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.553803    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.553870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:09.053851911 +0000 UTC m=+3.038939442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-systemd\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa33cc83-47ff-45b3-9179-5752917f8b0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-system-cni-dir\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-cnibin\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-host\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-host\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-cnibin\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-modprobe-d\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6521c29d-f99c-465b-980d-15eb1c948b26-multus-socket-dir-parent\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-multus-daemon-config\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.553718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-system-cni-dir\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa33cc83-47ff-45b3-9179-5752917f8b0c-os-release\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-sysconfig\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/839e96e8-afc8-472e-923c-8eef62db3a83-iptables-alerter-script\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.555565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.554797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6521c29d-f99c-465b-980d-15eb1c948b26-cni-binary-copy\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.557378 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.557354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-tmp\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.557751 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.557703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-etc-tuned\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.558965 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.558945    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:08.558965 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.558968    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:08.559098 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.558979    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:08.559098 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.559038    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:09.059023965 +0000 UTC m=+3.044111484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:08.562308 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.562283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gglvk\" (UniqueName: \"kubernetes.io/projected/d57f7b11-d8b8-40f7-9294-ce581b8b5d47-kube-api-access-gglvk\") pod \"tuned-mkf2m\" (UID: \"d57f7b11-d8b8-40f7-9294-ce581b8b5d47\") " pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.563079 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.563054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hhs\" (UniqueName: \"kubernetes.io/projected/4b5f27d7-42b3-49c3-b3b0-069f1f2ec263-kube-api-access-v9hhs\") pod \"node-ca-c2jl7\" (UID: \"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263\") " pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.565048 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.565023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxh7x\" (UniqueName: \"kubernetes.io/projected/3b436c25-3bb8-4029-be28-e05e46f89a92-kube-api-access-mxh7x\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:08.566104 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.566047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/fa33cc83-47ff-45b3-9179-5752917f8b0c-kube-api-access-jqddf\") pod \"multus-additional-cni-plugins-t7m4q\" (UID: \"fa33cc83-47ff-45b3-9179-5752917f8b0c\") " pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.567554 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.567531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7vf\" (UniqueName: \"kubernetes.io/projected/6521c29d-f99c-465b-980d-15eb1c948b26-kube-api-access-qs7vf\") pod \"multus-bg5rz\" (UID: \"6521c29d-f99c-465b-980d-15eb1c948b26\") " pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.570174 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.570151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfkx\" (UniqueName: \"kubernetes.io/projected/839e96e8-afc8-472e-923c-8eef62db3a83-kube-api-access-5kfkx\") pod \"iptables-alerter-cd47q\" (UID: \"839e96e8-afc8-472e-923c-8eef62db3a83\") " pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.637353 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.637309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:08.645661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.645631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxzbs"
Jul 17 16:26:08.656051 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.655333    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbaaadc_1b66_4386_859a_b4320f0352fc.slice/crio-320ccc414e8b67d7a29a638cc393702f443e0ac6a58985947cfcffe1f8d1aa98 WatchSource:0}: Error finding container 320ccc414e8b67d7a29a638cc393702f443e0ac6a58985947cfcffe1f8d1aa98: Status 404 returned error can't find the container with id 320ccc414e8b67d7a29a638cc393702f443e0ac6a58985947cfcffe1f8d1aa98
Jul 17 16:26:08.656051 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.655765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:08.665332 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.665069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5"
Jul 17 16:26:08.669425 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.669369    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6799b7_2071_4df3_9fb4_33d2ef840c41.slice/crio-b89f860635aab63b86c624851177d5db456689841078a9da699040762a50a74c WatchSource:0}: Error finding container b89f860635aab63b86c624851177d5db456689841078a9da699040762a50a74c: Status 404 returned error can't find the container with id b89f860635aab63b86c624851177d5db456689841078a9da699040762a50a74c
Jul 17 16:26:08.675133 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.675095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c2jl7"
Jul 17 16:26:08.676223 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.676196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec56f3a1_eaf2_413e_ab24_1cc938a27137.slice/crio-94c454db9eb0df6beefd7ee522b624d0ecce9903a707b20784fc82a7542585cc WatchSource:0}: Error finding container 94c454db9eb0df6beefd7ee522b624d0ecce9903a707b20784fc82a7542585cc: Status 404 returned error can't find the container with id 94c454db9eb0df6beefd7ee522b624d0ecce9903a707b20784fc82a7542585cc
Jul 17 16:26:08.684534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.684510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cd47q"
Jul 17 16:26:08.687088 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.687059    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5f27d7_42b3_49c3_b3b0_069f1f2ec263.slice/crio-45746ea1c0032e25049fb0c403d2310d289e4ceeb76e6ab56cce7f8875acfd7a WatchSource:0}: Error finding container 45746ea1c0032e25049fb0c403d2310d289e4ceeb76e6ab56cce7f8875acfd7a: Status 404 returned error can't find the container with id 45746ea1c0032e25049fb0c403d2310d289e4ceeb76e6ab56cce7f8875acfd7a
Jul 17 16:26:08.694281 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.694258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7m4q"
Jul 17 16:26:08.699314 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.699283    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839e96e8_afc8_472e_923c_8eef62db3a83.slice/crio-69ea8b09fbd56de8b50e5ba5b431024d206694c8f59d51edbf81eddec95172ff WatchSource:0}: Error finding container 69ea8b09fbd56de8b50e5ba5b431024d206694c8f59d51edbf81eddec95172ff: Status 404 returned error can't find the container with id 69ea8b09fbd56de8b50e5ba5b431024d206694c8f59d51edbf81eddec95172ff
Jul 17 16:26:08.703707 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.703648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bg5rz"
Jul 17 16:26:08.705614 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.705586    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa33cc83_47ff_45b3_9179_5752917f8b0c.slice/crio-d8aeb2948590aac6f7d609a6c04fa2714ec456d16623b30f50077cfaace5bc37 WatchSource:0}: Error finding container d8aeb2948590aac6f7d609a6c04fa2714ec456d16623b30f50077cfaace5bc37: Status 404 returned error can't find the container with id d8aeb2948590aac6f7d609a6c04fa2714ec456d16623b30f50077cfaace5bc37
Jul 17 16:26:08.712476 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.712366    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkf2m"
Jul 17 16:26:08.715362 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.715339    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6521c29d_f99c_465b_980d_15eb1c948b26.slice/crio-3b759d2077bfc358b7cfaeebaabff2326332807f54d9f5c0a2402db3acc9958a WatchSource:0}: Error finding container 3b759d2077bfc358b7cfaeebaabff2326332807f54d9f5c0a2402db3acc9958a: Status 404 returned error can't find the container with id 3b759d2077bfc358b7cfaeebaabff2326332807f54d9f5c0a2402db3acc9958a
Jul 17 16:26:08.725149 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:08.725114    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57f7b11_d8b8_40f7_9294_ce581b8b5d47.slice/crio-e4b43fd04da3ecc4f0e201d902d1f14a51667a90a37ac5617862f014733f3047 WatchSource:0}: Error finding container e4b43fd04da3ecc4f0e201d902d1f14a51667a90a37ac5617862f014733f3047: Status 404 returned error can't find the container with id e4b43fd04da3ecc4f0e201d902d1f14a51667a90a37ac5617862f014733f3047
Jul 17 16:26:08.957265 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:08.957227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:08.957460 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.957358    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:08.957460 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:08.957423    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:09.957409028 +0000 UTC m=+3.942496545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:09.058523 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.058484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:09.058804 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.058680    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:09.058804 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.058760    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:10.058737804 +0000 UTC m=+4.043825355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:09.159321 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.159280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:09.159502 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.159457    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:09.159502 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.159479    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:09.159502 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.159490    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:09.159625 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.159552    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:10.159534518 +0000 UTC m=+4.144622036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:09.377702 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.377597    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 16:21:08 +0000 UTC" deadline="2028-03-14 15:09:50.996027442 +0000 UTC"
Jul 17 16:26:09.377702 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.377666    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14542h43m41.618367274s"
Jul 17 16:26:09.508484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.508441    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:09.508654 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.508599    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:09.522064 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.521976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkf2m" event={"ID":"d57f7b11-d8b8-40f7-9294-ce581b8b5d47","Type":"ContainerStarted","Data":"e4b43fd04da3ecc4f0e201d902d1f14a51667a90a37ac5617862f014733f3047"}
Jul 17 16:26:09.523528 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.523492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bg5rz" event={"ID":"6521c29d-f99c-465b-980d-15eb1c948b26","Type":"ContainerStarted","Data":"3b759d2077bfc358b7cfaeebaabff2326332807f54d9f5c0a2402db3acc9958a"}
Jul 17 16:26:09.524992 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.524961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerStarted","Data":"d8aeb2948590aac6f7d609a6c04fa2714ec456d16623b30f50077cfaace5bc37"}
Jul 17 16:26:09.526460 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.526410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cd47q" event={"ID":"839e96e8-afc8-472e-923c-8eef62db3a83","Type":"ContainerStarted","Data":"69ea8b09fbd56de8b50e5ba5b431024d206694c8f59d51edbf81eddec95172ff"}
Jul 17 16:26:09.528099 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.528073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c2jl7" event={"ID":"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263","Type":"ContainerStarted","Data":"45746ea1c0032e25049fb0c403d2310d289e4ceeb76e6ab56cce7f8875acfd7a"}
Jul 17 16:26:09.529477 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.529443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d48vc" event={"ID":"8fbaaadc-1b66-4386-859a-b4320f0352fc","Type":"ContainerStarted","Data":"320ccc414e8b67d7a29a638cc393702f443e0ac6a58985947cfcffe1f8d1aa98"}
Jul 17 16:26:09.530951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.530925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5" event={"ID":"ec56f3a1-eaf2-413e-ab24-1cc938a27137","Type":"ContainerStarted","Data":"94c454db9eb0df6beefd7ee522b624d0ecce9903a707b20784fc82a7542585cc"}
Jul 17 16:26:09.532361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.532335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"b89f860635aab63b86c624851177d5db456689841078a9da699040762a50a74c"}
Jul 17 16:26:09.533600 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.533575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxzbs" event={"ID":"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6","Type":"ContainerStarted","Data":"e383cd8b7721987573560f44e4cf1efb02e705c77c62c832027dcdf5e7af4437"}
Jul 17 16:26:09.965918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:09.965874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:09.966203 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.966054    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:09.966203 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:09.966136    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:11.966117341 +0000 UTC m=+5.951204873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:10.066587 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.066544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:10.066794 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.066724    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:10.066879 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.066811    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:12.066776932 +0000 UTC m=+6.051864456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:10.167130 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.167061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:10.167302 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.167273    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:10.167302 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.167294    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:10.167407 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.167307    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:10.167407 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.167366    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:12.167347761 +0000 UTC m=+6.152435281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:10.378359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.378290    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 16:21:08 +0000 UTC" deadline="2028-01-06 02:51:00.607141615 +0000 UTC"
Jul 17 16:26:10.378359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.378332    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12898h24m50.228814455s"
Jul 17 16:26:10.510500 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.510464    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:10.510672 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.510613    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:10.510764 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:10.510664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:10.510822 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:10.510801    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:11.508941 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:11.508601    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:11.509529 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:11.508981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:11.549088 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:11.549035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerStarted","Data":"51aaba2a76e3d723cabde4ab7b89f7770c24e4779ce1e13fe9968b2c60c93a4a"}
Jul 17 16:26:11.559150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:11.559102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" event={"ID":"bbb9b55189c2ca1c136d2bbd25cb800a","Type":"ContainerStarted","Data":"58438be2220e8bdcb661180548202e7145a944e6d969eb03e5938d83942569e7"}
Jul 17 16:26:11.982624 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:11.982593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:11.982791 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:11.982747    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:11.982880 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:11.982826    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:15.982807484 +0000 UTC m=+9.967895052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:12.083708 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:12.083677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:12.083872 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.083808    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:12.083872 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.083867    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:16.083849379 +0000 UTC m=+10.068936901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:12.184563 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:12.184529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:12.184748 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.184728    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:12.184748 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.184746    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:12.184861 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.184759    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:12.184861 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.184812    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:16.184794028 +0000 UTC m=+10.169881549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:12.508073 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:12.508022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:12.508237 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.508200    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:12.508325 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:12.508252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:12.508469 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:12.508443    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:13.508223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:13.508194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:13.508658 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:13.508310    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:13.565191 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:13.565097    2561 generic.go:356] "Generic (PLEG): container finished" podID="3deef81c0d8c2f4458539863e90bc2e7" containerID="51aaba2a76e3d723cabde4ab7b89f7770c24e4779ce1e13fe9968b2c60c93a4a" exitCode=0
Jul 17 16:26:13.565191 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:13.565150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerDied","Data":"51aaba2a76e3d723cabde4ab7b89f7770c24e4779ce1e13fe9968b2c60c93a4a"}
Jul 17 16:26:13.582257 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:13.582032    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-172.ec2.internal" podStartSLOduration=6.582012946 podStartE2EDuration="6.582012946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:11.575724575 +0000 UTC m=+5.560812114" watchObservedRunningTime="2026-07-17 16:26:13.582012946 +0000 UTC m=+7.567100485"
Jul 17 16:26:14.508312 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:14.508272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:14.508755 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:14.508411    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:14.508817 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:14.508784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:14.508934 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:14.508892    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:15.508004 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:15.507970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:15.508180 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:15.508122    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:16.020191 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:16.020156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:16.020627 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.020288    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:16.020627 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.020365    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:24.020347049 +0000 UTC m=+18.005434566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:16.120542 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:16.120507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:16.120711 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.120665    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:16.120782 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.120741    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:24.120719775 +0000 UTC m=+18.105807312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:16.221886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:16.221561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:16.221886 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.221728    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:16.221886 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.221750    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:16.221886 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.221763    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:16.221886 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.221816    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:24.221800136 +0000 UTC m=+18.206887655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:16.511862 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:16.511359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:16.511862 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.511504    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:16.511862 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:16.511739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:16.512862 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:16.512756    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:17.509258 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:17.508779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:17.509258 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:17.508906    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:18.508389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:18.508314    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:18.508553 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:18.508436    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:18.508823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:18.508678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:18.508823 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:18.508801    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:19.508345 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:19.508274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:19.508759 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:19.508378    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:20.508706 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:20.508667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:20.509173 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:20.508799    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:20.509173 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:20.508858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:20.509173 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:20.508961    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:21.508031 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:21.508000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:21.508202 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:21.508128    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:22.508176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:22.508140    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:22.508601 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:22.508279    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:22.508601 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:22.508332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:22.508601 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:22.508462    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:23.508420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:23.508384    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:23.508836 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:23.508509    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:24.079209 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:24.079174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:24.079490 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.079320    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:24.079490 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.079383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:26:40.079364132 +0000 UTC m=+34.064451659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:24.180277 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:24.180244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:24.180439 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.180358    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:24.180439 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.180425    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:40.180407505 +0000 UTC m=+34.165495020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:24.280544 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:24.280515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:24.280700 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.280669    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:24.280700 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.280694    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:24.280700 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.280703    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:24.280840 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.280752    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:26:40.280738845 +0000 UTC m=+34.265826361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:24.508286 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:24.508206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:24.508448 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.508339    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:24.508448 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:24.508394    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:24.508845 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:24.508484    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:25.508902 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:25.508865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:25.509369 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:25.508997    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:26.509895 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:26.509809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:26.510431 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:26.509903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:26.510431 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:26.509924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:26.510431 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:26.510025    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:27.508329 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:27.508308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:27.508441 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:27.508423    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:27.593272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:27.593211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" event={"ID":"3deef81c0d8c2f4458539863e90bc2e7","Type":"ContainerStarted","Data":"c4bacee13dba0a52f611ede204baed32dae96d795499d2acec0465e9b01ffc2a"}
Jul 17 16:26:27.597054 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:27.597004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxzbs" event={"ID":"bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6","Type":"ContainerStarted","Data":"1239ffd0c4b2aad3976bbb8c8e603b186293f8013d4f310927a908a2d23e208e"}
Jul 17 16:26:28.508458 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.508013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:28.508655 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.508099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:28.508655 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:28.508576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:28.508655 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:28.508636    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:28.600082 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.600026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkf2m" event={"ID":"d57f7b11-d8b8-40f7-9294-ce581b8b5d47","Type":"ContainerStarted","Data":"2d7591e21af8ac5e13f70cb0fa5e5b14dd6014438c333ff4b0ed4d62614c1dc6"}
Jul 17 16:26:28.601397 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.601366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bg5rz" event={"ID":"6521c29d-f99c-465b-980d-15eb1c948b26","Type":"ContainerStarted","Data":"40579988c11c97dc7d2bf6e47f6341a5c00185aedd67af16c4077fb66215facc"}
Jul 17 16:26:28.602735 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.602710    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="87b3b16a91ef8352efd246b17f3fa4603157555335b3a91a14fb8356278d84fa" exitCode=0
Jul 17 16:26:28.602830 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.602742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"87b3b16a91ef8352efd246b17f3fa4603157555335b3a91a14fb8356278d84fa"}
Jul 17 16:26:28.604209 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.604188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c2jl7" event={"ID":"4b5f27d7-42b3-49c3-b3b0-069f1f2ec263","Type":"ContainerStarted","Data":"3f4f7e701e630a98d79cc2e62e95dcbf1ab375060a295e11625a7a2c12bfa66f"}
Jul 17 16:26:28.605565 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.605538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d48vc" event={"ID":"8fbaaadc-1b66-4386-859a-b4320f0352fc","Type":"ContainerStarted","Data":"afc9fecca39668ebeb129886ed77b85933455ccf78c45935c6f623077f3d0c12"}
Jul 17 16:26:28.607014 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.606995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5" event={"ID":"ec56f3a1-eaf2-413e-ab24-1cc938a27137","Type":"ContainerStarted","Data":"9ffac917bee5b76e5c089a345807780a7a84cae93c4dc9fb8fd832dbbef16f14"}
Jul 17 16:26:28.609875 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.609858    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:26:28.610177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610159    2561 generic.go:356] "Generic (PLEG): container finished" podID="4f6799b7-2071-4df3-9fb4-33d2ef840c41" containerID="00931f335ce1dde25d3a38a3f0bc33d88feea14611e3badc9386ee8294e25335" exitCode=1
Jul 17 16:26:28.610263 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"75b8e9f616fdd325542c20584f1ec0fe0ef3a8c537a1afb88be39c8da9aa8c45"}
Jul 17 16:26:28.610313 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"06499d33e2a635b0e78f4b45260444ae1ec141be9547fad693b7d5bf79a69b5e"}
Jul 17 16:26:28.610358 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"e4e7cc406fc8812b7ab9b6594b29630a29e57a3ddaa6b4f808c6dda4c70f0736"}
Jul 17 16:26:28.610400 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"5238d4e4e9296df9e13280b8bce28fd8f383be52baf266c5199fb493b4abf85e"}
Jul 17 16:26:28.610433 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerDied","Data":"00931f335ce1dde25d3a38a3f0bc33d88feea14611e3badc9386ee8294e25335"}
Jul 17 16:26:28.610462 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.610454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"20e6a455a82c43495a9e4af25a2e800528b8cf52b71016afe568863f461f9dd6"}
Jul 17 16:26:28.621765 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.621725    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mkf2m" podStartSLOduration=3.891821853 podStartE2EDuration="22.621712063s" totalImagesPullingTime="18.72989021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.621277743 +0000 UTC m=+22.606365282" watchObservedRunningTime="2026-07-17 16:26:28.621712063 +0000 UTC m=+22.606799601"
Jul 17 16:26:28.635492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.635452    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c2jl7" podStartSLOduration=4.088030138 podStartE2EDuration="22.63543673s" totalImagesPullingTime="18.547406592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.634835768 +0000 UTC m=+22.619923305" watchObservedRunningTime="2026-07-17 16:26:28.63543673 +0000 UTC m=+22.620524268"
Jul 17 16:26:28.647780 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.647746    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-172.ec2.internal" podStartSLOduration=21.647732968 podStartE2EDuration="21.647732968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.647703254 +0000 UTC m=+22.632790791" watchObservedRunningTime="2026-07-17 16:26:28.647732968 +0000 UTC m=+22.632820488"
Jul 17 16:26:28.663972 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.663931    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nxzbs" podStartSLOduration=4.090254028 podStartE2EDuration="22.663920096s" totalImagesPullingTime="18.573666068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.663732999 +0000 UTC m=+22.648820531" watchObservedRunningTime="2026-07-17 16:26:28.663920096 +0000 UTC m=+22.649007633"
Jul 17 16:26:28.710617 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.710562    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bg5rz" podStartSLOduration=3.528783653 podStartE2EDuration="22.710548423s" totalImagesPullingTime="19.18176477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.710299833 +0000 UTC m=+22.695387382" watchObservedRunningTime="2026-07-17 16:26:28.710548423 +0000 UTC m=+22.695635961"
Jul 17 16:26:28.725783 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.725741    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d48vc" podStartSLOduration=3.931405559 podStartE2EDuration="22.725725149s" totalImagesPullingTime="18.79431959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:28.725642442 +0000 UTC m=+22.710729979" watchObservedRunningTime="2026-07-17 16:26:28.725725149 +0000 UTC m=+22.710812687"
Jul 17 16:26:28.802549 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.802513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:28.876783 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.876754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:28.977125 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:28.977094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:29.028326 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.028302    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 16:26:29.081823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.081763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:29.404652 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.404496    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T16:26:29.028322938Z","UUID":"9aa51191-3b78-451c-9eac-608a9259e451","Handler":null,"Name":"","Endpoint":""}
Jul 17 16:26:29.406582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.406561    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 16:26:29.406707 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.406606    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 16:26:29.508400 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.508357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:29.508635 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:29.508489    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:29.615106 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.615033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cd47q" event={"ID":"839e96e8-afc8-472e-923c-8eef62db3a83","Type":"ContainerStarted","Data":"a8bdb0bf0358bd68f2ade23ab83bf1dcd395b86af9b8eadf4476ad147a921592"}
Jul 17 16:26:29.617503 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.617459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5" event={"ID":"ec56f3a1-eaf2-413e-ab24-1cc938a27137","Type":"ContainerStarted","Data":"f39e9afc230a741d011fb4dfefe18f3e4a3858e3b76508576184bd40698cecc5"}
Jul 17 16:26:29.644586 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.644526    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cd47q" podStartSLOduration=5.107860824 podStartE2EDuration="23.644506389s" totalImagesPullingTime="18.536645565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:29.642965262 +0000 UTC m=+23.628052801" watchObservedRunningTime="2026-07-17 16:26:29.644506389 +0000 UTC m=+23.629593929"
Jul 17 16:26:29.889594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:29.889533    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:30.125492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.125460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:30.187817 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.187729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:30.508716 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.508687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:30.508890 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:30.508834    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:30.508890 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.508872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:30.509004 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:30.508966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:30.621368 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.621333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5" event={"ID":"ec56f3a1-eaf2-413e-ab24-1cc938a27137","Type":"ContainerStarted","Data":"8a78f315767acfaa5cb7f8a80fe4fffd18a51b9c3ae72bd32fd6734304fff4ec"}
Jul 17 16:26:30.624158 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.624138    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:26:30.624544 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.624523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"1082f569a3200cc505b4b9508f1cbcc1abf6009a34eadb6cf08cf988fcadfc10"}
Jul 17 16:26:30.641272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.641210    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-42jg5" podStartSLOduration=3.474998043 podStartE2EDuration="24.641188557s" totalImagesPullingTime="21.166190514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:30.640463992 +0000 UTC m=+24.625551530" watchObservedRunningTime="2026-07-17 16:26:30.641188557 +0000 UTC m=+24.626276097"
Jul 17 16:26:30.926097 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:30.926023    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:31.350014 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:31.349781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:31.467734 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:31.467683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:31.508605 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:31.508583    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:31.508773 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:31.508697    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:31.567238 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:31.567205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:31.954032 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:31.953999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:32.054462 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.054415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:32.405744 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.405713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:32.406293 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.406274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:32.508773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.508741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:32.508949 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:32.508894    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:32.509015 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.508951    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:32.509112 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:32.509087    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:32.629377 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.629344    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:32.629904 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.629887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d48vc"
Jul 17 16:26:32.697926 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:32.697857    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:33.508820 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:33.508792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:33.509390 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:33.508906    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:33.634454 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:33.634311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:26:33.634891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:33.634864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"d22599c72972b17cf8530e0f49dcd1ddeb046c1b6a4e44db4668d0bdb9a9d681"}
Jul 17 16:26:33.635302 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:33.635284    2561 scope.go:117] "RemoveContainer" containerID="00931f335ce1dde25d3a38a3f0bc33d88feea14611e3badc9386ee8294e25335"
Jul 17 16:26:34.154638 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.154591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:34.508142 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.508083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:34.508298 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.508083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:34.508298 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:34.508181    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:34.508298 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:34.508253    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:34.641213 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.641186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:26:34.641647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.641610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" event={"ID":"4f6799b7-2071-4df3-9fb4-33d2ef840c41","Type":"ContainerStarted","Data":"f58008d6c8870746011d3f95a3464ddf85f60f7fcb4a8ecd8c1306c287fb5048"}
Jul 17 16:26:34.641899 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.641867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:34.642070 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.642034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:34.642166 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.642131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:34.643920 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.643891    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="102f94a2c0667e1f0d488f90c2094ac2db207cfbd3bddfb098512ddf5cd98497" exitCode=0
Jul 17 16:26:34.644063 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.643982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"102f94a2c0667e1f0d488f90c2094ac2db207cfbd3bddfb098512ddf5cd98497"}
Jul 17 16:26:34.658578 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.658555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:34.659844 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.659821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:26:34.706436 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:34.706381    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf" podStartSLOduration=9.874161862 podStartE2EDuration="28.706363908s" totalImagesPullingTime="18.832202046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:34.671694341 +0000 UTC m=+28.656781880" watchObservedRunningTime="2026-07-17 16:26:34.706363908 +0000 UTC m=+28.691451424"
Jul 17 16:26:35.508851 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.508829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:35.508959 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:35.508922    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:35.616802 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.616772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtrtl"]
Jul 17 16:26:35.616958 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.616930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:35.617074 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:35.617036    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:35.620958 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.620923    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gj7m"]
Jul 17 16:26:35.621691 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.621658    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rl4t"]
Jul 17 16:26:35.621796 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.621771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:35.621884 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:35.621863    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:35.646406 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:35.646387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:35.646750 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:35.646487    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:36.650320 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:36.650293    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="cc21f8f62e71c9e6c6ce376d9f200c6ef80cc91d1fcda19b92254cd2cc637dfe" exitCode=0
Jul 17 16:26:36.650697 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:36.650371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"cc21f8f62e71c9e6c6ce376d9f200c6ef80cc91d1fcda19b92254cd2cc637dfe"}
Jul 17 16:26:37.508793 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:37.508764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:37.508918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:37.508800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:37.508918 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:37.508887    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:37.508983 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:37.508924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:37.508983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:37.508961    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:37.509089 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:37.509072    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:38.657618 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:38.657585    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="7a429bbc6c9277ee4b7e33312ce37221402c9ca6794abbdc115f25c738d8ee6a" exitCode=0
Jul 17 16:26:38.658193 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:38.657650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"7a429bbc6c9277ee4b7e33312ce37221402c9ca6794abbdc115f25c738d8ee6a"}
Jul 17 16:26:39.199361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.199330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:39.268366 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.268326    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:39.508840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.508748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:39.508840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.508788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:39.509076 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.508874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:39.509076 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:39.508873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtrtl" podUID="83fb69b8-a939-4d03-a0f8-995040d1230b"
Jul 17 16:26:39.509076 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:39.508966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gj7m" podUID="60734c62-8f46-4187-9170-3b13aca22a6a"
Jul 17 16:26:39.509076 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:39.509058    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rl4t" podUID="3b436c25-3bb8-4029-be28-e05e46f89a92"
Jul 17 16:26:39.666352 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.666301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:39.734457 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.734426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:39.984070 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:39.984028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:40.084033 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.084001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:40.097870 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.097841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:40.097995 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.097975    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:40.098067 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.098035    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret podName:83fb69b8-a939-4d03-a0f8-995040d1230b nodeName:}" failed. No retries permitted until 2026-07-17 16:27:12.098017453 +0000 UTC m=+66.083104969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret") pod "global-pull-secret-syncer-mtrtl" (UID: "83fb69b8-a939-4d03-a0f8-995040d1230b") : object "kube-system"/"original-pull-secret" not registered
Jul 17 16:26:40.198458 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.198429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:40.198616 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.198596    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:40.198682 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.198671    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs podName:3b436c25-3bb8-4029-be28-e05e46f89a92 nodeName:}" failed. No retries permitted until 2026-07-17 16:27:12.198650569 +0000 UTC m=+66.183738089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs") pod "network-metrics-daemon-4rl4t" (UID: "3b436c25-3bb8-4029-be28-e05e46f89a92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 16:26:40.299118 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.299090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:40.299288 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.299274    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 16:26:40.299352 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.299295    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 16:26:40.299352 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.299309    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kv4k8 for pod openshift-network-diagnostics/network-check-target-8gj7m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:40.299452 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:40.299366    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8 podName:60734c62-8f46-4187-9170-3b13aca22a6a nodeName:}" failed. No retries permitted until 2026-07-17 16:27:12.299350479 +0000 UTC m=+66.284437999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kv4k8" (UniqueName: "kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8") pod "network-check-target-8gj7m" (UID: "60734c62-8f46-4187-9170-3b13aca22a6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 16:26:40.691253 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.691157    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:40.792641 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.792609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:40.809339 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.809312    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-172.ec2.internal" event="NodeReady"
Jul 17 16:26:40.809463 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.809452    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 16:26:40.851651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.851627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:26:40.854177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.854157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:40.860814 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.857889    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 16:26:40.860814 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.858652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 16:26:40.860814 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.859239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 16:26:40.860814 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.860744    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ktnr2\""
Jul 17 16:26:40.865449 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.865429    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-flgqs"]
Jul 17 16:26:40.866823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.866801    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 16:26:40.868456 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.868438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:40.871080 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.871025    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4x2wh\""
Jul 17 16:26:40.871235 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.871180    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 16:26:40.871374 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.871358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ldwx6"]
Jul 17 16:26:40.872325 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.872305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 16:26:40.874059 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.874025    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:26:40.874189 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.874170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:40.876459 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.876440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 16:26:40.876556 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.876471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 16:26:40.876615 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.876575    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j87cv\""
Jul 17 16:26:40.876843 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.876811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 16:26:40.881697 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.881674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flgqs"]
Jul 17 16:26:40.884932 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.884911    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ldwx6"]
Jul 17 16:26:40.895605 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.895586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-85m6q"]
Jul 17 16:26:40.898555 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.898537    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:40.900813 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.900794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 16:26:40.900915 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.900804    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 16:26:40.901098 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.901078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 16:26:40.901344 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.901317    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 16:26:40.901425 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.901391    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bxh6h\""
Jul 17 16:26:40.912570 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:40.912545    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85m6q"]
Jul 17 16:26:41.003768 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.003768 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps422\" (UniqueName: \"kubernetes.io/projected/c4615327-7d8b-4645-9bc2-dea866d9673d-kube-api-access-ps422\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.003768 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afdf257-121e-4a42-9ea8-73fd40d0e70c-data-volume\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmd9\" (UniqueName: \"kubernetes.io/projected/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-api-access-bgmd9\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfrs\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.003947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfff028-810a-45b5-bf27-cf1f65f7e108-config-volume\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfff028-810a-45b5-bf27-cf1f65f7e108-tmp-dir\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.003988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mq8\" (UniqueName: \"kubernetes.io/projected/9bfff028-810a-45b5-bf27-cf1f65f7e108-kube-api-access-h5mq8\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4615327-7d8b-4645-9bc2-dea866d9673d-cert\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afdf257-121e-4a42-9ea8-73fd40d0e70c-crio-socket\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.004199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.004176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.014812 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.014789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.079419 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.079400    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxzbs_bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6/dns-node-resolver/0.log"
Jul 17 16:26:41.104728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mq8\" (UniqueName: \"kubernetes.io/projected/9bfff028-810a-45b5-bf27-cf1f65f7e108-kube-api-access-h5mq8\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.104848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4615327-7d8b-4645-9bc2-dea866d9673d-cert\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.104848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afdf257-121e-4a42-9ea8-73fd40d0e70c-crio-socket\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.104848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.104848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.104848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps422\" (UniqueName: \"kubernetes.io/projected/c4615327-7d8b-4645-9bc2-dea866d9673d-kube-api-access-ps422\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afdf257-121e-4a42-9ea8-73fd40d0e70c-data-volume\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.104975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmd9\" (UniqueName: \"kubernetes.io/projected/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-api-access-bgmd9\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.105114 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105113    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105128    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56ff7cbb5b-m6cxl: secret "image-registry-tls" not found
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105181    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls podName:88c8cc46-a339-434a-bb34-fc8fa77d7a26 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:41.605161092 +0000 UTC m=+35.590248631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls") pod "image-registry-56ff7cbb5b-m6cxl" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26") : secret "image-registry-tls" not found
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8afdf257-121e-4a42-9ea8-73fd40d0e70c-crio-socket\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afdf257-121e-4a42-9ea8-73fd40d0e70c-data-volume\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.105444 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105442    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105486    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105497    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:41.605480109 +0000 UTC m=+35.590567686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : configmap "kube-rbac-proxy" not found
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105650    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls podName:9bfff028-810a-45b5-bf27-cf1f65f7e108 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:41.605634968 +0000 UTC m=+35.590722491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls") pod "dns-default-flgqs" (UID: "9bfff028-810a-45b5-bf27-cf1f65f7e108") : secret "dns-default-metrics-tls" not found
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfrs\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfff028-810a-45b5-bf27-cf1f65f7e108-config-volume\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfff028-810a-45b5-bf27-cf1f65f7e108-tmp-dir\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.105781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.105771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.106241 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105864    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:41.106241 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.105902    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:41.605890875 +0000 UTC m=+35.590978393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:41.106241 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.106085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.106241 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.106169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.106441 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.106343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfff028-810a-45b5-bf27-cf1f65f7e108-tmp-dir\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.106554 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.106532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfff028-810a-45b5-bf27-cf1f65f7e108-config-volume\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.109780 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.109761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.109886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.109759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.109886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.109801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4615327-7d8b-4645-9bc2-dea866d9673d-cert\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.117369 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.117343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmd9\" (UniqueName: \"kubernetes.io/projected/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-api-access-bgmd9\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.117510 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.117461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mq8\" (UniqueName: \"kubernetes.io/projected/9bfff028-810a-45b5-bf27-cf1f65f7e108-kube-api-access-h5mq8\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.118188 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.118147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.118480 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.118463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfrs\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.119437 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.119420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps422\" (UniqueName: \"kubernetes.io/projected/c4615327-7d8b-4645-9bc2-dea866d9673d-kube-api-access-ps422\") pod \"ingress-canary-ldwx6\" (UID: \"c4615327-7d8b-4645-9bc2-dea866d9673d\") " pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.121001 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.120969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.188225 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.188199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ldwx6"
Jul 17 16:26:41.501619 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.501579    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.508087 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.508063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:26:41.508250 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.508165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:26:41.508354 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.508326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:26:41.509375 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.509352    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cxldp"]
Jul 17 16:26:41.510299 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.510275    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 16:26:41.510852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.510655    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 16:26:41.510852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.510674    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54g7g\""
Jul 17 16:26:41.510852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.510675    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 16:26:41.511128 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.511114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 16:26:41.511213 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.511116    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h7wh7\""
Jul 17 16:26:41.513197 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.513180    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.515224 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.515156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 16:26:41.515429 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.515409    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 16:26:41.515429 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.515421    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 16:26:41.515594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.515438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wxzbq\""
Jul 17 16:26:41.515594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.515519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 16:26:41.520003 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.519983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cxldp"]
Jul 17 16:26:41.564282 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.564251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.610653 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.610747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqx8v\" (UniqueName: \"kubernetes.io/projected/120d486b-083c-4410-8cb5-b325aeb8fdfd-kube-api-access-jqx8v\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.610747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-cabundle\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.610747 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.610715    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 16:26:41.610886 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.610775    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:42.610755791 +0000 UTC m=+36.595843306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : configmap "kube-rbac-proxy" not found
Jul 17 16:26:41.610886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:41.610979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-key\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.610979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.610954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:41.611088 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611015    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:41.611145 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.611100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:41.611145 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:42.611101288 +0000 UTC m=+36.596188819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:41.611145 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611118    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 16:26:41.611145 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611133    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56ff7cbb5b-m6cxl: secret "image-registry-tls" not found
Jul 17 16:26:41.611309 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611161    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls podName:88c8cc46-a339-434a-bb34-fc8fa77d7a26 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:42.611153587 +0000 UTC m=+36.596241106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls") pod "image-registry-56ff7cbb5b-m6cxl" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26") : secret "image-registry-tls" not found
Jul 17 16:26:41.611309 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611209    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 16:26:41.611309 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:41.611266    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls podName:9bfff028-810a-45b5-bf27-cf1f65f7e108 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:42.611252431 +0000 UTC m=+36.596339951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls") pod "dns-default-flgqs" (UID: "9bfff028-810a-45b5-bf27-cf1f65f7e108") : secret "dns-default-metrics-tls" not found
Jul 17 16:26:41.666491 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.666454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.679528 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.679507    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c2jl7_4b5f27d7-42b3-49c3-b3b0-069f1f2ec263/node-ca/0.log"
Jul 17 16:26:41.711883 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.711861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-key\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.712327 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.711929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx8v\" (UniqueName: \"kubernetes.io/projected/120d486b-083c-4410-8cb5-b325aeb8fdfd-kube-api-access-jqx8v\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.712327 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.711957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-cabundle\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.712760 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.712730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-cabundle\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.714515 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.714493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/120d486b-083c-4410-8cb5-b325aeb8fdfd-signing-key\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.720380 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.720359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx8v\" (UniqueName: \"kubernetes.io/projected/120d486b-083c-4410-8cb5-b325aeb8fdfd-kube-api-access-jqx8v\") pod \"service-ca-58598695f7-cxldp\" (UID: \"120d486b-083c-4410-8cb5-b325aeb8fdfd\") " pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.752244 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.752189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:41.841278 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.841255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-cxldp"
Jul 17 16:26:41.905436 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:41.905407    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:42.008208 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.008132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:42.370543 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.370510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:42.619393 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.619180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:42.619490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.619428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:42.619569 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.619550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:42.619630 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.619598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619735    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619762    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619813    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:44.619788494 +0000 UTC m=+38.604876013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : configmap "kube-rbac-proxy" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619836    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:44.619823163 +0000 UTC m=+38.604910682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619859    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619899    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls podName:9bfff028-810a-45b5-bf27-cf1f65f7e108 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:44.619880646 +0000 UTC m=+38.604968179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls") pod "dns-default-flgqs" (UID: "9bfff028-810a-45b5-bf27-cf1f65f7e108") : secret "dns-default-metrics-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619920    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619932    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56ff7cbb5b-m6cxl: secret "image-registry-tls" not found
Jul 17 16:26:42.620002 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:42.619968    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls podName:88c8cc46-a339-434a-bb34-fc8fa77d7a26 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:44.619956639 +0000 UTC m=+38.605044176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls") pod "image-registry-56ff7cbb5b-m6cxl" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26") : secret "image-registry-tls" not found
Jul 17 16:26:42.665663 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.665629    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cxldp"]
Jul 17 16:26:42.668973 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:42.668867    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ldwx6"]
Jul 17 16:26:42.671426 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:42.671402    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120d486b_083c_4410_8cb5_b325aeb8fdfd.slice/crio-59ad100dbe005b611b19965d8dde06e5722cf5927aaaef05423c361375cfdf93 WatchSource:0}: Error finding container 59ad100dbe005b611b19965d8dde06e5722cf5927aaaef05423c361375cfdf93: Status 404 returned error can't find the container with id 59ad100dbe005b611b19965d8dde06e5722cf5927aaaef05423c361375cfdf93
Jul 17 16:26:42.672344 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:42.672319    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4615327_7d8b_4645_9bc2_dea866d9673d.slice/crio-acfa3278a46a4a1248c8825b5e72b317e085315af8daaceff38865fffcb81260 WatchSource:0}: Error finding container acfa3278a46a4a1248c8825b5e72b317e085315af8daaceff38865fffcb81260: Status 404 returned error can't find the container with id acfa3278a46a4a1248c8825b5e72b317e085315af8daaceff38865fffcb81260
Jul 17 16:26:43.678809 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:43.678762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-cxldp" event={"ID":"120d486b-083c-4410-8cb5-b325aeb8fdfd","Type":"ContainerStarted","Data":"59ad100dbe005b611b19965d8dde06e5722cf5927aaaef05423c361375cfdf93"}
Jul 17 16:26:43.680231 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:43.680202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ldwx6" event={"ID":"c4615327-7d8b-4645-9bc2-dea866d9673d","Type":"ContainerStarted","Data":"acfa3278a46a4a1248c8825b5e72b317e085315af8daaceff38865fffcb81260"}
Jul 17 16:26:43.683620 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:43.683591    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="c61e9600c9914cce4dad36795009851fcd5940c220ccbfa61622fdfe065d9a87" exitCode=0
Jul 17 16:26:43.683752 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:43.683636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"c61e9600c9914cce4dad36795009851fcd5940c220ccbfa61622fdfe065d9a87"}
Jul 17 16:26:44.640154 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.639890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.640185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640077    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.640239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640270    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:48.640245985 +0000 UTC m=+42.625333504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.640309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:44.640336 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640329    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640361    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640378    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls podName:9bfff028-810a-45b5-bf27-cf1f65f7e108 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:48.64036263 +0000 UTC m=+42.625450147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls") pod "dns-default-flgqs" (UID: "9bfff028-810a-45b5-bf27-cf1f65f7e108") : secret "dns-default-metrics-tls" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640378    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56ff7cbb5b-m6cxl: secret "image-registry-tls" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640393    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640413    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls podName:88c8cc46-a339-434a-bb34-fc8fa77d7a26 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:48.640403787 +0000 UTC m=+42.625491305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls") pod "image-registry-56ff7cbb5b-m6cxl" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26") : secret "image-registry-tls" not found
Jul 17 16:26:44.640619 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:44.640429    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:48.64041838 +0000 UTC m=+42.625505895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : configmap "kube-rbac-proxy" not found
Jul 17 16:26:44.690582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.690551    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa33cc83-47ff-45b3-9179-5752917f8b0c" containerID="7fafd51124efcbc84bc7d9bcf80eefc548fb00de87d33c68177669a7b1ce4d2a" exitCode=0
Jul 17 16:26:44.690582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.690591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerDied","Data":"7fafd51124efcbc84bc7d9bcf80eefc548fb00de87d33c68177669a7b1ce4d2a"}
Jul 17 16:26:44.964393 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:44.964309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:45.067086 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:45.067025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:45.468800 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:45.468769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:45.572581 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:45.572543    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:45.943456 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:45.943427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:46.696489 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.696443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ldwx6" event={"ID":"c4615327-7d8b-4645-9bc2-dea866d9673d","Type":"ContainerStarted","Data":"7c1fa750c48a5ce3ad47f2012f6a50c55f4470071da2333100f8b5ce0fa213ee"}
Jul 17 16:26:46.699880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.699841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" event={"ID":"fa33cc83-47ff-45b3-9179-5752917f8b0c","Type":"ContainerStarted","Data":"69d159f9c4e9cc253626ae39d66ea557a317906891c7e3cb14f7f41b5380470c"}
Jul 17 16:26:46.701210 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.701183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-cxldp" event={"ID":"120d486b-083c-4410-8cb5-b325aeb8fdfd","Type":"ContainerStarted","Data":"2c4cdf3a5c1092528be2038751d7286e93f41824683c29adc6525c53bc08c8d9"}
Jul 17 16:26:46.724166 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.724124    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ldwx6" podStartSLOduration=3.616757797 podStartE2EDuration="6.724111437s" totalImagesPullingTime="3.10735364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:46.723432051 +0000 UTC m=+40.708519602" watchObservedRunningTime="2026-07-17 16:26:46.724111437 +0000 UTC m=+40.709198975"
Jul 17 16:26:46.744082 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.744025    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-cxldp" podStartSLOduration=2.630366386 podStartE2EDuration="5.744016606s" totalImagesPullingTime="3.11365022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:46.743276861 +0000 UTC m=+40.728364401" watchObservedRunningTime="2026-07-17 16:26:46.744016606 +0000 UTC m=+40.729104163"
Jul 17 16:26:46.772239 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:46.772200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t7m4q" podStartSLOduration=11.079828275 podStartE2EDuration="40.77218946s" totalImagesPullingTime="29.692361185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:46.771927269 +0000 UTC m=+40.757014807" watchObservedRunningTime="2026-07-17 16:26:46.77218946 +0000 UTC m=+40.757277026"
Jul 17 16:26:47.321951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:47.321907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:47.423449 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:47.423375    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:47.523128 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:47.523091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:47.624256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:47.624175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:47.701859 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:47.701819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:48.673528 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.673489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:48.673528 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.673530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.673586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.673612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673639    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls podName:9bfff028-810a-45b5-bf27-cf1f65f7e108 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:56.673687475 +0000 UTC m=+50.658774996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls") pod "dns-default-flgqs" (UID: "9bfff028-810a-45b5-bf27-cf1f65f7e108") : secret "dns-default-metrics-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673740    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673770    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673781    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56ff7cbb5b-m6cxl: secret "image-registry-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673806    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls podName:8afdf257-121e-4a42-9ea8-73fd40d0e70c nodeName:}" failed. No retries permitted until 2026-07-17 16:26:56.673789225 +0000 UTC m=+50.658876748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-85m6q" (UID: "8afdf257-121e-4a42-9ea8-73fd40d0e70c") : secret "insights-runtime-extractor-tls" not found
Jul 17 16:26:48.673954 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:26:48.673825    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls podName:88c8cc46-a339-434a-bb34-fc8fa77d7a26 nodeName:}" failed. No retries permitted until 2026-07-17 16:26:56.673816164 +0000 UTC m=+50.658903680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls") pod "image-registry-56ff7cbb5b-m6cxl" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26") : secret "image-registry-tls" not found
Jul 17 16:26:48.674254 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.674190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8afdf257-121e-4a42-9ea8-73fd40d0e70c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:48.829621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:48.829585    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:49.014428 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:49.014335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:49.217873 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:49.217840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:49.424316 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:49.424273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:49.619464 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:49.619429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:50.614180 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:50.614146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:50.815090 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:50.815036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.017319 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.017241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.218749 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.218717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.319777 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.319741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.423987 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.423956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.820081 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.820034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:51.878813 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:51.878782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:52.620849 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:52.620809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:52.821972 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:52.821932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.026974 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.026934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.251261 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.251229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.435955 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.435869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.531949 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.531914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.630335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.630295    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:53.792177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:53.792120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:54.049213 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:54.049102    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:54.427774 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:54.427683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:54.838290 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:54.838255    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:55.031919 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:55.031884    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:55.433613 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:55.433579    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.219166 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.219129    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.426365 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.426331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.672862 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.672831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.728442 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.728413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:56.728625 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.728606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:56.728695 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.728656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:56.732237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.732207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8afdf257-121e-4a42-9ea8-73fd40d0e70c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-85m6q\" (UID: \"8afdf257-121e-4a42-9ea8-73fd40d0e70c\") " pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:56.732330 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.732220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bfff028-810a-45b5-bf27-cf1f65f7e108-metrics-tls\") pod \"dns-default-flgqs\" (UID: \"9bfff028-810a-45b5-bf27-cf1f65f7e108\") " pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:56.732330 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.732272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"image-registry-56ff7cbb5b-m6cxl\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") " pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:56.767123 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.767099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:56.774176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.774155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.781716 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.781689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flgqs"
Jul 17 16:26:56.808171 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.808151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-85m6q"
Jul 17 16:26:56.886171 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.885984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:56.933734 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.933530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:26:56.935546 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:56.935522    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c8cc46_a339_434a_bb34_fc8fa77d7a26.slice/crio-8f7a93cce18137360727fa712c8eedf0b4384ec80d1a27d51d8a29789bc2da57 WatchSource:0}: Error finding container 8f7a93cce18137360727fa712c8eedf0b4384ec80d1a27d51d8a29789bc2da57: Status 404 returned error can't find the container with id 8f7a93cce18137360727fa712c8eedf0b4384ec80d1a27d51d8a29789bc2da57
Jul 17 16:26:56.952952 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:56.952925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flgqs"]
Jul 17 16:26:56.956274 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:26:56.956244    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfff028_810a_45b5_bf27_cf1f65f7e108.slice/crio-c3e1f002b2f07cd40f6a0bd8609a7a785d1a914125da2eb0dab4ad66c5f779d2 WatchSource:0}: Error finding container c3e1f002b2f07cd40f6a0bd8609a7a785d1a914125da2eb0dab4ad66c5f779d2: Status 404 returned error can't find the container with id c3e1f002b2f07cd40f6a0bd8609a7a785d1a914125da2eb0dab4ad66c5f779d2
Jul 17 16:26:57.013609 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.013581    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-85m6q"]
Jul 17 16:26:57.050754 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.050718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:57.630483 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.630417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:57.729818 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.729782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85m6q" event={"ID":"8afdf257-121e-4a42-9ea8-73fd40d0e70c","Type":"ContainerStarted","Data":"70663a48afc2b99e5ce1709dfad5f9d682c0ac96b49e1f6fd48cf72908eb6166"}
Jul 17 16:26:57.733227 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.732744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" event={"ID":"88c8cc46-a339-434a-bb34-fc8fa77d7a26","Type":"ContainerStarted","Data":"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"}
Jul 17 16:26:57.733227 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.732790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" event={"ID":"88c8cc46-a339-434a-bb34-fc8fa77d7a26","Type":"ContainerStarted","Data":"8f7a93cce18137360727fa712c8eedf0b4384ec80d1a27d51d8a29789bc2da57"}
Jul 17 16:26:57.733227 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.732860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:26:57.734580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.734509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flgqs" event={"ID":"9bfff028-810a-45b5-bf27-cf1f65f7e108","Type":"ContainerStarted","Data":"c3e1f002b2f07cd40f6a0bd8609a7a785d1a914125da2eb0dab4ad66c5f779d2"}
Jul 17 16:26:57.761508 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.761425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" podStartSLOduration=32.761408275 podStartE2EDuration="32.761408275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:26:57.760230783 +0000 UTC m=+51.745318360" watchObservedRunningTime="2026-07-17 16:26:57.761408275 +0000 UTC m=+51.746495794"
Jul 17 16:26:57.825806 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:57.825632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:58.022774 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:58.022656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:26:59.741802 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:26:59.741728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flgqs" event={"ID":"9bfff028-810a-45b5-bf27-cf1f65f7e108","Type":"ContainerStarted","Data":"81b430f7837ee7c2a581dc61761439447e504886b7bdc48f90bb66c35e806b07"}
Jul 17 16:27:02.753488 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:02.753433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85m6q" event={"ID":"8afdf257-121e-4a42-9ea8-73fd40d0e70c","Type":"ContainerStarted","Data":"6c2396d6c763b138ac870a808c8076aa3aef359790d96dec9411e58ee7e9484e"}
Jul 17 16:27:02.756001 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:02.755971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flgqs" event={"ID":"9bfff028-810a-45b5-bf27-cf1f65f7e108","Type":"ContainerStarted","Data":"da6a3d8837eb49a8461a27adceb6836f257324e3684a94cf51b5e340968d1d5a"}
Jul 17 16:27:02.756238 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:02.756209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-flgqs"
Jul 17 16:27:02.777453 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:02.777407    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-flgqs" podStartSLOduration=20.284252976 podStartE2EDuration="22.777388547s" totalImagesPullingTime="2.493135571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:02.776652409 +0000 UTC m=+56.761739979" watchObservedRunningTime="2026-07-17 16:27:02.777388547 +0000 UTC m=+56.762476082"
Jul 17 16:27:04.929489 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.929461    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"]
Jul 17 16:27:04.932911 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.932893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:04.939916 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.939888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 16:27:04.940754 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.940732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 16:27:04.940754 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.940748    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 16:27:04.941943 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.941926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 16:27:04.957973 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.957937    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"]
Jul 17 16:27:04.967495 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.967472    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:27:04.992557 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.992527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2f8\" (UniqueName: \"kubernetes.io/projected/01f9cf0f-899e-4952-b0c7-f523498bb52b-kube-api-access-gk2f8\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:04.992693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.992576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f9cf0f-899e-4952-b0c7-f523498bb52b-tmp\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:04.992693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:04.992639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f9cf0f-899e-4952-b0c7-f523498bb52b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.044370 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.044344    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"]
Jul 17 16:27:05.051630 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.051612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.065949 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.065707    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"]
Jul 17 16:27:05.081541 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.081513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:05.093642 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f9cf0f-899e-4952-b0c7-f523498bb52b-tmp\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.093758 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537e195f-3c1f-45a5-8349-79e09e38b6d6-ca-trust-extracted\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.093758 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-installation-pull-secrets\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.093758 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-certificates\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.093904 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-trusted-ca\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.093904 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-image-registry-private-configuration\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.093904 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f9cf0f-899e-4952-b0c7-f523498bb52b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.094067 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-tls\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.094067 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-bound-sa-token\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.094067 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g88hd\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-kube-api-access-g88hd\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.094067 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.093996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2f8\" (UniqueName: \"kubernetes.io/projected/01f9cf0f-899e-4952-b0c7-f523498bb52b-kube-api-access-gk2f8\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.094247 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.094079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01f9cf0f-899e-4952-b0c7-f523498bb52b-tmp\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.097111 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.097089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01f9cf0f-899e-4952-b0c7-f523498bb52b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.103471 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.103445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2f8\" (UniqueName: \"kubernetes.io/projected/01f9cf0f-899e-4952-b0c7-f523498bb52b-kube-api-access-gk2f8\") pod \"klusterlet-addon-workmgr-6597575fb-wv95s\" (UID: \"01f9cf0f-899e-4952-b0c7-f523498bb52b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.195297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-tls\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-bound-sa-token\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g88hd\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-kube-api-access-g88hd\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537e195f-3c1f-45a5-8349-79e09e38b6d6-ca-trust-extracted\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-installation-pull-secrets\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-certificates\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-trusted-ca\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-image-registry-private-configuration\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.195966 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.195942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537e195f-3c1f-45a5-8349-79e09e38b6d6-ca-trust-extracted\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.196765 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.196737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-certificates\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.196895 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.196832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537e195f-3c1f-45a5-8349-79e09e38b6d6-trusted-ca\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.198024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.198000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-image-registry-private-configuration\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.198200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.198181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-registry-tls\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.198273 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.198241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537e195f-3c1f-45a5-8349-79e09e38b6d6-installation-pull-secrets\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.205058 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.205018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-bound-sa-token\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.205524 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.205499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g88hd\" (UniqueName: \"kubernetes.io/projected/537e195f-3c1f-45a5-8349-79e09e38b6d6-kube-api-access-g88hd\") pod \"image-registry-5d54d7c4d9-rxc4v\" (UID: \"537e195f-3c1f-45a5-8349-79e09e38b6d6\") " pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.242444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.242422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:05.272272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.272245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:05.362365 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.362340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:05.368350 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.368324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:05.863918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.863892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"]
Jul 17 16:27:05.866783 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:05.866758    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f9cf0f_899e_4952_b0c7_f523498bb52b.slice/crio-88c5d3d9650de6d92acb5758eec07ef338fa10374c5595f9a96065259d110ac4 WatchSource:0}: Error finding container 88c5d3d9650de6d92acb5758eec07ef338fa10374c5595f9a96065259d110ac4: Status 404 returned error can't find the container with id 88c5d3d9650de6d92acb5758eec07ef338fa10374c5595f9a96065259d110ac4
Jul 17 16:27:05.885606 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:05.885582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"]
Jul 17 16:27:05.888849 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:05.888824    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537e195f_3c1f_45a5_8349_79e09e38b6d6.slice/crio-2b0032151476180f0c3acb331ea77e129feed04a2ccd71e79a0c1b516b8db30b WatchSource:0}: Error finding container 2b0032151476180f0c3acb331ea77e129feed04a2ccd71e79a0c1b516b8db30b: Status 404 returned error can't find the container with id 2b0032151476180f0c3acb331ea77e129feed04a2ccd71e79a0c1b516b8db30b
Jul 17 16:27:06.668134 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.668109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mj6xf"
Jul 17 16:27:06.772634 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.772593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85m6q" event={"ID":"8afdf257-121e-4a42-9ea8-73fd40d0e70c","Type":"ContainerStarted","Data":"0571fdfafee6c02480106cebe06b8162c4513e257cd024f48a75f0b40fd12052"}
Jul 17 16:27:06.774748 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.774604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v" event={"ID":"537e195f-3c1f-45a5-8349-79e09e38b6d6","Type":"ContainerStarted","Data":"90583cf37f1e93d392739ee53521de07bc89d3f39ab1156759661fa7ca4b085c"}
Jul 17 16:27:06.774748 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.774647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v" event={"ID":"537e195f-3c1f-45a5-8349-79e09e38b6d6","Type":"ContainerStarted","Data":"2b0032151476180f0c3acb331ea77e129feed04a2ccd71e79a0c1b516b8db30b"}
Jul 17 16:27:06.774886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.774820    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:06.776432 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.776379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s" event={"ID":"01f9cf0f-899e-4952-b0c7-f523498bb52b","Type":"ContainerStarted","Data":"88c5d3d9650de6d92acb5758eec07ef338fa10374c5595f9a96065259d110ac4"}
Jul 17 16:27:06.801016 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.800960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v" podStartSLOduration=1.800941746 podStartE2EDuration="1.800941746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:06.799593888 +0000 UTC m=+60.784681423" watchObservedRunningTime="2026-07-17 16:27:06.800941746 +0000 UTC m=+60.786029285"
Jul 17 16:27:06.852764 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.852684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:06.954636 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:06.954549    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:07.373095 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:07.373062    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:07.433095 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:07.433064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:07.772144 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:07.772034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:07.826419 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:07.826386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.091725 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.091638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.260381 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.260347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.263085 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.263057    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"]
Jul 17 16:27:09.276122 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.276103    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"]
Jul 17 16:27:09.276235 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.276197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:09.278316 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.278296    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 17 16:27:09.278408 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.278374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dbnpl\""
Jul 17 16:27:09.332399 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.332371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ac7fdd0-767a-45b9-a6b0-4a165c337dd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thmth\" (UID: \"1ac7fdd0-767a-45b9-a6b0-4a165c337dd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:09.380239 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.380159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.433181 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.433157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ac7fdd0-767a-45b9-a6b0-4a165c337dd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thmth\" (UID: \"1ac7fdd0-767a-45b9-a6b0-4a165c337dd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:09.435878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.435852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ac7fdd0-767a-45b9-a6b0-4a165c337dd0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thmth\" (UID: \"1ac7fdd0-767a-45b9-a6b0-4a165c337dd0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:09.587101 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.587066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:09.649034 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.648967    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.702210 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.702181    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:09.761621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:09.761592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-flgqs"
Jul 17 16:27:10.002032 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.001959    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-nrbnv"]
Jul 17 16:27:10.014008 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.013980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:10.015325 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.015295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-nrbnv"]
Jul 17 16:27:10.016658 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.016216    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 16:27:10.016658 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.016325    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x4plk\""
Jul 17 16:27:10.016658 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.016475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 16:27:10.140236 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.140206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt566\" (UniqueName: \"kubernetes.io/projected/2f948260-9c95-4077-8ed2-3a9e53529335-kube-api-access-vt566\") pod \"downloads-5bc67dcd46-nrbnv\" (UID: \"2f948260-9c95-4077-8ed2-3a9e53529335\") " pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:10.241117 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.241083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt566\" (UniqueName: \"kubernetes.io/projected/2f948260-9c95-4077-8ed2-3a9e53529335-kube-api-access-vt566\") pod \"downloads-5bc67dcd46-nrbnv\" (UID: \"2f948260-9c95-4077-8ed2-3a9e53529335\") " pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:10.253137 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.253065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt566\" (UniqueName: \"kubernetes.io/projected/2f948260-9c95-4077-8ed2-3a9e53529335-kube-api-access-vt566\") pod \"downloads-5bc67dcd46-nrbnv\" (UID: \"2f948260-9c95-4077-8ed2-3a9e53529335\") " pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:10.325828 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:10.325797    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:11.436130 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.436087    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-nrbnv"]
Jul 17 16:27:11.439692 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:11.439667    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f948260_9c95_4077_8ed2_3a9e53529335.slice/crio-60a77d53f3bf07d30d6cf9d01163bf10e2a14f2ac3fba792a04108a949627f3a WatchSource:0}: Error finding container 60a77d53f3bf07d30d6cf9d01163bf10e2a14f2ac3fba792a04108a949627f3a: Status 404 returned error can't find the container with id 60a77d53f3bf07d30d6cf9d01163bf10e2a14f2ac3fba792a04108a949627f3a
Jul 17 16:27:11.448609 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.448590    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"]
Jul 17 16:27:11.451010 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:11.450989    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac7fdd0_767a_45b9_a6b0_4a165c337dd0.slice/crio-1f973cc8a78e49d24791e290fd50ee65cd49bc91a7420b032188c99e8a8def2f WatchSource:0}: Error finding container 1f973cc8a78e49d24791e290fd50ee65cd49bc91a7420b032188c99e8a8def2f: Status 404 returned error can't find the container with id 1f973cc8a78e49d24791e290fd50ee65cd49bc91a7420b032188c99e8a8def2f
Jul 17 16:27:11.796773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.796743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-85m6q" event={"ID":"8afdf257-121e-4a42-9ea8-73fd40d0e70c","Type":"ContainerStarted","Data":"2bd59014b778e04298c087c93da28978942c5d1e4cad3ecaa61ce158b775e1be"}
Jul 17 16:27:11.798343 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.798315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s" event={"ID":"01f9cf0f-899e-4952-b0c7-f523498bb52b","Type":"ContainerStarted","Data":"6eb09a6b940678bbb3f9f2f142fae7ecbeef9bfe904ded2345110d05da79a7b9"}
Jul 17 16:27:11.798491 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.798467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:11.799502 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.799474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth" event={"ID":"1ac7fdd0-767a-45b9-a6b0-4a165c337dd0","Type":"ContainerStarted","Data":"1f973cc8a78e49d24791e290fd50ee65cd49bc91a7420b032188c99e8a8def2f"}
Jul 17 16:27:11.800134 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.800115    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s"
Jul 17 16:27:11.800651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.800634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-nrbnv" event={"ID":"2f948260-9c95-4077-8ed2-3a9e53529335","Type":"ContainerStarted","Data":"60a77d53f3bf07d30d6cf9d01163bf10e2a14f2ac3fba792a04108a949627f3a"}
Jul 17 16:27:11.813135 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.812982    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-85m6q" podStartSLOduration=17.688663049 podStartE2EDuration="31.812967254s" totalImagesPullingTime="14.124304205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:11.812697078 +0000 UTC m=+65.797784616" watchObservedRunningTime="2026-07-17 16:27:11.812967254 +0000 UTC m=+65.798054792"
Jul 17 16:27:11.827272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:11.827237    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6597575fb-wv95s" podStartSLOduration=2.378480584 podStartE2EDuration="7.827224646s" totalImagesPullingTime="5.448744062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:11.826252625 +0000 UTC m=+65.811340228" watchObservedRunningTime="2026-07-17 16:27:11.827224646 +0000 UTC m=+65.812312184"
Jul 17 16:27:12.158126 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.158020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:27:12.160275 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.160249    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 16:27:12.170835 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.170806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83fb69b8-a939-4d03-a0f8-995040d1230b-original-pull-secret\") pod \"global-pull-secret-syncer-mtrtl\" (UID: \"83fb69b8-a939-4d03-a0f8-995040d1230b\") " pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:27:12.259131 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.259100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:27:12.261154 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.261135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 16:27:12.271997 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.271975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b436c25-3bb8-4029-be28-e05e46f89a92-metrics-certs\") pod \"network-metrics-daemon-4rl4t\" (UID: \"3b436c25-3bb8-4029-be28-e05e46f89a92\") " pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:27:12.360506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.360465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:27:12.362761 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.362735    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 16:27:12.373710 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.373685    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 16:27:12.385414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.385362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4k8\" (UniqueName: \"kubernetes.io/projected/60734c62-8f46-4187-9170-3b13aca22a6a-kube-api-access-kv4k8\") pod \"network-check-target-8gj7m\" (UID: \"60734c62-8f46-4187-9170-3b13aca22a6a\") " pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:27:12.423658 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.423600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54g7g\""
Jul 17 16:27:12.428647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.428624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtrtl"
Jul 17 16:27:12.431709 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.431684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:27:12.438023 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.438001    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h7wh7\""
Jul 17 16:27:12.445922 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.445900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rl4t"
Jul 17 16:27:12.754840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.754769    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtrtl"]
Jul 17 16:27:12.758357 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:12.758309    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fb69b8_a939_4d03_a0f8_995040d1230b.slice/crio-a11bf106f565b795f779a5bcee321d6908ab71b982a0c8571ef359fc7c39e654 WatchSource:0}: Error finding container a11bf106f565b795f779a5bcee321d6908ab71b982a0c8571ef359fc7c39e654: Status 404 returned error can't find the container with id a11bf106f565b795f779a5bcee321d6908ab71b982a0c8571ef359fc7c39e654
Jul 17 16:27:12.788480 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.788454    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rl4t"]
Jul 17 16:27:12.791261 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:12.791233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b436c25_3bb8_4029_be28_e05e46f89a92.slice/crio-bb312043c9325ae805c68dac570ffae7238d7931b38a0839d4a4676d73cf4295 WatchSource:0}: Error finding container bb312043c9325ae805c68dac570ffae7238d7931b38a0839d4a4676d73cf4295: Status 404 returned error can't find the container with id bb312043c9325ae805c68dac570ffae7238d7931b38a0839d4a4676d73cf4295
Jul 17 16:27:12.805024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.804995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtrtl" event={"ID":"83fb69b8-a939-4d03-a0f8-995040d1230b","Type":"ContainerStarted","Data":"a11bf106f565b795f779a5bcee321d6908ab71b982a0c8571ef359fc7c39e654"}
Jul 17 16:27:12.806429 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.806398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rl4t" event={"ID":"3b436c25-3bb8-4029-be28-e05e46f89a92","Type":"ContainerStarted","Data":"bb312043c9325ae805c68dac570ffae7238d7931b38a0839d4a4676d73cf4295"}
Jul 17 16:27:12.994867 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:12.994685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gj7m"]
Jul 17 16:27:12.997436 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:12.997408    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60734c62_8f46_4187_9170_3b13aca22a6a.slice/crio-377f2178447c0efcc1529fd972611d2b045226e80bad2a483ef41925a34a980d WatchSource:0}: Error finding container 377f2178447c0efcc1529fd972611d2b045226e80bad2a483ef41925a34a980d: Status 404 returned error can't find the container with id 377f2178447c0efcc1529fd972611d2b045226e80bad2a483ef41925a34a980d
Jul 17 16:27:13.812788 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:13.812741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gj7m" event={"ID":"60734c62-8f46-4187-9170-3b13aca22a6a","Type":"ContainerStarted","Data":"377f2178447c0efcc1529fd972611d2b045226e80bad2a483ef41925a34a980d"}
Jul 17 16:27:13.816340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:13.816310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth" event={"ID":"1ac7fdd0-767a-45b9-a6b0-4a165c337dd0","Type":"ContainerStarted","Data":"aa85f7d1ef01629bff362ae0d3289070df5217fb24e7f0c108be3683c577186e"}
Jul 17 16:27:13.817527 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:13.817503    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:13.823338 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:13.823290    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth"
Jul 17 16:27:13.831740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:13.830766    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thmth" podStartSLOduration=3.274460924 podStartE2EDuration="4.83074899s" totalImagesPullingTime="1.556288066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:13.829749447 +0000 UTC m=+67.814837000" watchObservedRunningTime="2026-07-17 16:27:13.83074899 +0000 UTC m=+67.815836530"
Jul 17 16:27:14.327991 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.327419    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"]
Jul 17 16:27:14.332126 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.330854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.334755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.333672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 16:27:14.334755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.333934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 16:27:14.334755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.334206    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kqkkv\""
Jul 17 16:27:14.334755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.334389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 16:27:14.334755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.334609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 16:27:14.339730 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.338951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"]
Jul 17 16:27:14.340175 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.339975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 16:27:14.376799 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.376701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zdw\" (UniqueName: \"kubernetes.io/projected/28f361ff-5641-43dd-9353-028c51f1de7a-kube-api-access-92zdw\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.376799 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.376751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.376995 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.376836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.376995 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.376875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f361ff-5641-43dd-9353-028c51f1de7a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.478836 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.478802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zdw\" (UniqueName: \"kubernetes.io/projected/28f361ff-5641-43dd-9353-028c51f1de7a-kube-api-access-92zdw\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.479028 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.478875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.479028 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.478934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.479028 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.478965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f361ff-5641-43dd-9353-028c51f1de7a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.479910 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.479856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28f361ff-5641-43dd-9353-028c51f1de7a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.482804 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.482755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.483341 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.483302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28f361ff-5641-43dd-9353-028c51f1de7a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.487770 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.487744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zdw\" (UniqueName: \"kubernetes.io/projected/28f361ff-5641-43dd-9353-028c51f1de7a-kube-api-access-92zdw\") pod \"prometheus-operator-5945b5cfd-mgt8h\" (UID: \"28f361ff-5641-43dd-9353-028c51f1de7a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.647308 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.647236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"
Jul 17 16:27:14.804170 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.804137    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h"]
Jul 17 16:27:14.808262 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:14.808229    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f361ff_5641_43dd_9353_028c51f1de7a.slice/crio-657e6e602e80eeeae35b1f6ba102f4736c25b08cd74e8334f047f85470548e20 WatchSource:0}: Error finding container 657e6e602e80eeeae35b1f6ba102f4736c25b08cd74e8334f047f85470548e20: Status 404 returned error can't find the container with id 657e6e602e80eeeae35b1f6ba102f4736c25b08cd74e8334f047f85470548e20
Jul 17 16:27:14.822574 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.822474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rl4t" event={"ID":"3b436c25-3bb8-4029-be28-e05e46f89a92","Type":"ContainerStarted","Data":"93156c6f378b7b392ed47a8109bd09d6154d055101565147ccf988e6a63962dc"}
Jul 17 16:27:14.822574 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.822534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rl4t" event={"ID":"3b436c25-3bb8-4029-be28-e05e46f89a92","Type":"ContainerStarted","Data":"30895e5abc52e6204722f70d8bf94ed4e89903b139fec50491cf5011a1c02d3f"}
Jul 17 16:27:14.824714 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.824677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" event={"ID":"28f361ff-5641-43dd-9353-028c51f1de7a","Type":"ContainerStarted","Data":"657e6e602e80eeeae35b1f6ba102f4736c25b08cd74e8334f047f85470548e20"}
Jul 17 16:27:14.839264 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.839075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4rl4t" podStartSLOduration=67.680679748 podStartE2EDuration="1m8.83905586s" totalImagesPullingTime="1.158376112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:14.838224444 +0000 UTC m=+68.823312006" watchObservedRunningTime="2026-07-17 16:27:14.83905586 +0000 UTC m=+68.824143597"
Jul 17 16:27:14.975796 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.975463    2561 patch_prober.go:28] interesting pod/image-registry-56ff7cbb5b-m6cxl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 16:27:14.975796 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:14.975521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 16:27:18.844434 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:18.844374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gj7m" event={"ID":"60734c62-8f46-4187-9170-3b13aca22a6a","Type":"ContainerStarted","Data":"dcee23c3d4d164ff76fe86da7a489274051715979d1496695af3b24125cdfdae"}
Jul 17 16:27:18.844851 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:18.844508    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:27:18.845908 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:18.845885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtrtl" event={"ID":"83fb69b8-a939-4d03-a0f8-995040d1230b","Type":"ContainerStarted","Data":"609befd2b2de0eccf67c5793a5025b99c439c7c80b293e51cae5560e067b9cd4"}
Jul 17 16:27:18.860607 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:18.860565    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gj7m" podStartSLOduration=67.580017061 podStartE2EDuration="1m12.860552371s" totalImagesPullingTime="5.28053531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:18.858306883 +0000 UTC m=+72.843394420" watchObservedRunningTime="2026-07-17 16:27:18.860552371 +0000 UTC m=+72.845639928"
Jul 17 16:27:18.872536 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:18.872491    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mtrtl" podStartSLOduration=67.343223602 podStartE2EDuration="1m12.872475441s" totalImagesPullingTime="5.529251839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:18.872055036 +0000 UTC m=+72.857142586" watchObservedRunningTime="2026-07-17 16:27:18.872475441 +0000 UTC m=+72.857562979"
Jul 17 16:27:19.322804 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.322771    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:27:19.326278 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.326261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.328461 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.328435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 16:27:19.328563 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.328498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 16:27:19.329285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.329265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 16:27:19.329285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.329281    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 16:27:19.329450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.329314    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 16:27:19.329450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.329314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zvldz\""
Jul 17 16:27:19.335394 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.335373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:27:19.417360 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.417515 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5s5p\" (UniqueName: \"kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.417515 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.417515 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.417680 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.417680 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.417583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s5p\" (UniqueName: \"kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518458 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.518898 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.519019 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.518949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.519449 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.519432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.521077 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.521034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.521225 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.521203    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.539293 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.539269    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5s5p\" (UniqueName: \"kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p\") pod \"console-5b56b8c749-54r5b\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") " pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.635264 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.635193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:19.798840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.798762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:19.829330 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.829298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:27:19.832490 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:19.832462    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c1f19f_caaa_4351_bde1_ce134d517809.slice/crio-ce9263739013ab3a7253a0aaa8dc3514d96b4bd7738f35bc43e8f2bd17cbbe91 WatchSource:0}: Error finding container ce9263739013ab3a7253a0aaa8dc3514d96b4bd7738f35bc43e8f2bd17cbbe91: Status 404 returned error can't find the container with id ce9263739013ab3a7253a0aaa8dc3514d96b4bd7738f35bc43e8f2bd17cbbe91
Jul 17 16:27:19.854102 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.854062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56b8c749-54r5b" event={"ID":"b7c1f19f-caaa-4351-bde1-ce134d517809","Type":"ContainerStarted","Data":"ce9263739013ab3a7253a0aaa8dc3514d96b4bd7738f35bc43e8f2bd17cbbe91"}
Jul 17 16:27:19.899670 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:19.899595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:22.645968 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:22.645928    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:22.746099 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:22.746061    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:23.116722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:23.116683    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:23.224083 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:23.224035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:24.107847 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:24.107809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:24.165693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:24.165658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:24.972840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:24.972811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:27:25.203921 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:25.203882    2561 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ee63fef2ed7a9138d8861c98047a04a620909e35084ab481ec14674f84c2c4d"
Jul 17 16:27:25.204398 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:25.204009    2561 kuberuntime_manager.go:1459] "Unhandled Error" err="container prometheus-operator start failed in pod prometheus-operator-5945b5cfd-mgt8h_openshift-monitoring(28f361ff-5641-43dd-9353-028c51f1de7a): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 17 16:27:26.674308 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:26.674266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:26.774298 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:26.774268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:27.713509 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.713471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:27:27.757089 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.757055    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:27:27.757258 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.757151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.766795 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.766762    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 16:27:27.785804 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.785783    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d54d7c4d9-rxc4v"
Jul 17 16:27:27.882000 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.881966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lmm\" (UniqueName: \"kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882177 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882354 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882354 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882354 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.882693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.882366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.911896 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.911862    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:27.983137 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983137 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lmm\" (UniqueName: \"kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.983309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.983298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.984275 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.984244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.984387 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.984361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.984483 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.984451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.984596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.984565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.985879 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.985862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.986090 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.986072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:27.990576 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:27.990555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lmm\" (UniqueName: \"kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm\") pod \"console-8998f8dd5-h2rp4\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") " pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:28.012295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:28.012258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:28.069441 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:28.069417    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:28.071200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:28.071149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:29.992536 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:29.992424    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerName="registry" containerID="cri-o://0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803" gracePeriod=30
Jul 17 16:27:30.388781 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:30.388710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"prometheus-operator\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" podUID="28f361ff-5641-43dd-9353-028c51f1de7a"
Jul 17 16:27:30.430137 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.430111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:27:30.458259 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.458199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:27:30.461976 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:30.461953    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351617e9_712b_40c3_b57a_71c2890b2fff.slice/crio-aa88ff4aa98fd90d84efd7f23c83890b293cc868b9e70386cf607a923954d327 WatchSource:0}: Error finding container aa88ff4aa98fd90d84efd7f23c83890b293cc868b9e70386cf607a923954d327: Status 404 returned error can't find the container with id aa88ff4aa98fd90d84efd7f23c83890b293cc868b9e70386cf607a923954d327
Jul 17 16:27:30.502765 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.502869 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.502869 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.502869 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502830    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.503024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502872    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.503024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.503024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502954    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.503024 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.502996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvfrs\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs\") pod \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\" (UID: \"88c8cc46-a339-434a-bb34-fc8fa77d7a26\") "
Jul 17 16:27:30.503487 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.503439    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:27:30.503951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.503925    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:27:30.505659 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.505628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:27:30.505763 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.505645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:27:30.505763 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.505751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:27:30.505955 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.505934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs" (OuterVolumeSpecName: "kube-api-access-dvfrs") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "kube-api-access-dvfrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:27:30.506450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.506368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:27:30.515654 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.515241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "88c8cc46-a339-434a-bb34-fc8fa77d7a26" (UID: "88c8cc46-a339-434a-bb34-fc8fa77d7a26"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:27:30.604462 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604432    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-installation-pull-secrets\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604462 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604461    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-trusted-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604471    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604479    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-bound-sa-token\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604489    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88c8cc46-a339-434a-bb34-fc8fa77d7a26-image-registry-private-configuration\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604502    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88c8cc46-a339-434a-bb34-fc8fa77d7a26-ca-trust-extracted\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604511    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88c8cc46-a339-434a-bb34-fc8fa77d7a26-registry-certificates\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.604647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.604520    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfrs\" (UniqueName: \"kubernetes.io/projected/88c8cc46-a339-434a-bb34-fc8fa77d7a26-kube-api-access-dvfrs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:27:30.899160 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.899118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56b8c749-54r5b" event={"ID":"b7c1f19f-caaa-4351-bde1-ce134d517809","Type":"ContainerStarted","Data":"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"}
Jul 17 16:27:30.901215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.901183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8998f8dd5-h2rp4" event={"ID":"351617e9-712b-40c3-b57a-71c2890b2fff","Type":"ContainerStarted","Data":"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"}
Jul 17 16:27:30.901364 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.901233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8998f8dd5-h2rp4" event={"ID":"351617e9-712b-40c3-b57a-71c2890b2fff","Type":"ContainerStarted","Data":"aa88ff4aa98fd90d84efd7f23c83890b293cc868b9e70386cf607a923954d327"}
Jul 17 16:27:30.903003 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.902973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-nrbnv" event={"ID":"2f948260-9c95-4077-8ed2-3a9e53529335","Type":"ContainerStarted","Data":"b1844719fa02a05b54c6538ca6c7f3ed90cf2c5b795182f53ca2d712ae71639f"}
Jul 17 16:27:30.903185 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.903154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:30.905480 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.904921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" event={"ID":"28f361ff-5641-43dd-9353-028c51f1de7a","Type":"ContainerStarted","Data":"13111d57cf30c8c50587d278b17307ea4aaddbc61a13ccc99d27fe6c65575b33"}
Jul 17 16:27:30.906341 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:30.906308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"prometheus-operator\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ee63fef2ed7a9138d8861c98047a04a620909e35084ab481ec14674f84c2c4d\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" podUID="28f361ff-5641-43dd-9353-028c51f1de7a"
Jul 17 16:27:30.906481 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.906454    2561 generic.go:356] "Generic (PLEG): container finished" podID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerID="0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803" exitCode=0
Jul 17 16:27:30.906548 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.906491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" event={"ID":"88c8cc46-a339-434a-bb34-fc8fa77d7a26","Type":"ContainerDied","Data":"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"}
Jul 17 16:27:30.906548 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.906517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"
Jul 17 16:27:30.906548 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.906532    2561 scope.go:117] "RemoveContainer" containerID="0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"
Jul 17 16:27:30.906697 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.906518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56ff7cbb5b-m6cxl" event={"ID":"88c8cc46-a339-434a-bb34-fc8fa77d7a26","Type":"ContainerDied","Data":"8f7a93cce18137360727fa712c8eedf0b4384ec80d1a27d51d8a29789bc2da57"}
Jul 17 16:27:30.915473 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.915450    2561 scope.go:117] "RemoveContainer" containerID="0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"
Jul 17 16:27:30.915780 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:30.915758    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803\": container with ID starting with 0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803 not found: ID does not exist" containerID="0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"
Jul 17 16:27:30.915872 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.915785    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803"} err="failed to get container status \"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803\": rpc error: code = NotFound desc = could not find container \"0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803\": container with ID starting with 0758a01b52b4e00af8ef087c002ebdb5fc2eed2ebbd2493124c1977980124803 not found: ID does not exist"
Jul 17 16:27:30.917977 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.917960    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-nrbnv"
Jul 17 16:27:30.918096 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.918000    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b56b8c749-54r5b" podStartSLOduration=1.4518591650000001 podStartE2EDuration="11.917984193s" totalImagesPullingTime="10.466125028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:30.916439142 +0000 UTC m=+84.901526696" watchObservedRunningTime="2026-07-17 16:27:30.917984193 +0000 UTC m=+84.903071709"
Jul 17 16:27:30.954165 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.954123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8998f8dd5-h2rp4" podStartSLOduration=3.954109976 podStartE2EDuration="3.954109976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:30.95295582 +0000 UTC m=+84.938043384" watchObservedRunningTime="2026-07-17 16:27:30.954109976 +0000 UTC m=+84.939197513"
Jul 17 16:27:30.966093 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.966069    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:27:30.970127 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.970099    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56ff7cbb5b-m6cxl"]
Jul 17 16:27:30.984576 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:30.984529    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-nrbnv" podStartSLOduration=3.062592523 podStartE2EDuration="21.984511462s" totalImagesPullingTime="18.921918939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:30.984074983 +0000 UTC m=+84.969162547" watchObservedRunningTime="2026-07-17 16:27:30.984511462 +0000 UTC m=+84.969598995"
Jul 17 16:27:31.913466 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:31.913437    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"prometheus-operator\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ee63fef2ed7a9138d8861c98047a04a620909e35084ab481ec14674f84c2c4d\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" podUID="28f361ff-5641-43dd-9353-028c51f1de7a"
Jul 17 16:27:32.115147 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:32.115098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:32.216547 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:32.216466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:32.282033 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:32.281997    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:32.514639 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:32.514555    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" path="/var/lib/kubelet/pods/88c8cc46-a339-434a-bb34-fc8fa77d7a26/volumes"
Jul 17 16:27:35.321255 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:35.321208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:35.421181 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:35.421144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:36.911624 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:36.911585    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:37.011769 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:37.011737    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:37.560734 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:37.560699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:37.661326 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:37.661285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:38.070163 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:38.070133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:38.070608 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:38.070231    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:38.075775 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:38.075752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:38.946891 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:38.946862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:27:39.001320 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.001292    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:27:39.112343 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.112311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:39.212842 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.212758    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:39.313651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.313625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:39.365700 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.365674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:39.635402 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:39.635379    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:27:41.252420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:41.252382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:41.313621 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:41.313575    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:41.676519 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:41.676496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:41.781827 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:41.781799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:45.613897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:45.613807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:45.967668 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:45.967596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" event={"ID":"28f361ff-5641-43dd-9353-028c51f1de7a","Type":"ContainerStarted","Data":"82c74db7133e7c88f19dbb30da7a6ef9db906fccaa4626ee0a346f3799fef144"}
Jul 17 16:27:45.991797 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:45.991753    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mgt8h" podStartSLOduration=1.533878511 podStartE2EDuration="31.991739377s" totalImagesPullingTime="30.457860866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-17 16:27:45.991217395 +0000 UTC m=+99.976304933" watchObservedRunningTime="2026-07-17 16:27:45.991739377 +0000 UTC m=+99.976826915"
Jul 17 16:27:47.439531 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.439497    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:47.620358 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.620314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:47.750231 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.750162    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:47.799451 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.799424    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-s967w"]
Jul 17 16:27:47.799761 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.799748    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerName="registry"
Jul 17 16:27:47.799826 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.799764    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerName="registry"
Jul 17 16:27:47.799878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.799827    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c8cc46-a339-434a-bb34-fc8fa77d7a26" containerName="registry"
Jul 17 16:27:47.822636 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.822590    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"]
Jul 17 16:27:47.822861 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.822717    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.826096 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.825682    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 17 16:27:47.826883 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.826850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 17 16:27:47.827063 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.827031    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2fdz4\""
Jul 17 16:27:47.827326 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.827282    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 17 16:27:47.841130 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.841110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-s967w"]
Jul 17 16:27:47.841230 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.841217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:47.841288 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.841239    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"]
Jul 17 16:27:47.841288 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.841273    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q7xm2"]
Jul 17 16:27:47.848688 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.848669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 17 16:27:47.848982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.848898    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gpbj5\""
Jul 17 16:27:47.856945 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.856916    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 17 16:27:47.862996 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.862979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:47.866271 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.866254    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zdw49\""
Jul 17 16:27:47.868202 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.868185    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 16:27:47.869233 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.869203    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 16:27:47.880230 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.880214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 16:27:47.915632 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.915564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:47.940318 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:47.940421 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f6cef1-a1ce-4dec-8e35-96bedd751bde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:47.940421 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.940421 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.940581 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spzg\" (UniqueName: \"kubernetes.io/projected/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-api-access-5spzg\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.940581 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.940581 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:47.940729 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:47.940729 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5495\" (UniqueName: \"kubernetes.io/projected/77f6cef1-a1ce-4dec-8e35-96bedd751bde-kube-api-access-t5495\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:47.940729 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:47.940691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.041875 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.041848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5495\" (UniqueName: \"kubernetes.io/projected/77f6cef1-a1ce-4dec-8e35-96bedd751bde-kube-api-access-t5495\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.041979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.041887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-sys\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.041979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.041912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-metrics-client-ca\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.041979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.041961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-root\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.041993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-textfile\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.042297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f6cef1-a1ce-4dec-8e35-96bedd751bde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.042297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042297 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:48.042289    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:48.042346    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls podName:77f6cef1-a1ce-4dec-8e35-96bedd751bde nodeName:}" failed. No retries permitted until 2026-07-17 16:27:48.542328415 +0000 UTC m=+102.527415938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-8tc7p" (UID: "77f6cef1-a1ce-4dec-8e35-96bedd751bde") : secret "openshift-state-metrics-tls" not found
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-tls\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-wtmp\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spzg\" (UniqueName: \"kubernetes.io/projected/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-api-access-5spzg\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mcl\" (UniqueName: \"kubernetes.io/projected/f06ab5f3-e61e-415e-8ec9-329d90f88197-kube-api-access-r2mcl\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:48.042561    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:27:48.042620    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls podName:033ef5f1-3ba2-4148-adfc-4ff2175f51ad nodeName:}" failed. No retries permitted until 2026-07-17 16:27:48.542603243 +0000 UTC m=+102.527690779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-s967w" (UID: "033ef5f1-3ba2-4148-adfc-4ff2175f51ad") : secret "kube-state-metrics-tls" not found
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.042845 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.043176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.042976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77f6cef1-a1ce-4dec-8e35-96bedd751bde-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.043332 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.043312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.043398 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.043315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.044900 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.044877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.045274 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.045256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.060856 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.060830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5495\" (UniqueName: \"kubernetes.io/projected/77f6cef1-a1ce-4dec-8e35-96bedd751bde-kube-api-access-t5495\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.061091 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.061074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spzg\" (UniqueName: \"kubernetes.io/projected/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-api-access-5spzg\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.099909 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.099847    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:48.144098 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-sys\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144186 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-metrics-client-ca\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144186 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-root\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144186 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-sys\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144186 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144384 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-textfile\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144384 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144384 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-root\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144522 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-tls\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144522 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-wtmp\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144522 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mcl\" (UniqueName: \"kubernetes.io/projected/f06ab5f3-e61e-415e-8ec9-329d90f88197-kube-api-access-r2mcl\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144660 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-textfile\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144660 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-wtmp\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.144950 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.144919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f06ab5f3-e61e-415e-8ec9-329d90f88197-metrics-client-ca\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.146435 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.146415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.146616 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.146598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f06ab5f3-e61e-415e-8ec9-329d90f88197-node-exporter-tls\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.164389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.164365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mcl\" (UniqueName: \"kubernetes.io/projected/f06ab5f3-e61e-415e-8ec9-329d90f88197-kube-api-access-r2mcl\") pod \"node-exporter-q7xm2\" (UID: \"f06ab5f3-e61e-415e-8ec9-329d90f88197\") " pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.173135 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.173117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7xm2"
Jul 17 16:27:48.212783 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:48.212762    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06ab5f3_e61e_415e_8ec9_329d90f88197.slice/crio-c1a73be7d03756936fd4a19e968f5e956f1428e59125f8298fcd8080eadc68a6 WatchSource:0}: Error finding container c1a73be7d03756936fd4a19e968f5e956f1428e59125f8298fcd8080eadc68a6: Status 404 returned error can't find the container with id c1a73be7d03756936fd4a19e968f5e956f1428e59125f8298fcd8080eadc68a6
Jul 17 16:27:48.238406 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.238378    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:48.389484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.389427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:48.547875 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.547849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.548202 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.547890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.550192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.550174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/033ef5f1-3ba2-4148-adfc-4ff2175f51ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-s967w\" (UID: \"033ef5f1-3ba2-4148-adfc-4ff2175f51ad\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.550251 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.550216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/77f6cef1-a1ce-4dec-8e35-96bedd751bde-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-8tc7p\" (UID: \"77f6cef1-a1ce-4dec-8e35-96bedd751bde\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.737737 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.737681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w"
Jul 17 16:27:48.752072 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.752033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"
Jul 17 16:27:48.818892 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.818216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:48.933076 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.932922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-s967w"]
Jul 17 16:27:48.937761 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:48.937689    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033ef5f1_3ba2_4148_adfc_4ff2175f51ad.slice/crio-b3b846eab37925a0df9f6cd9a634843f175bf59fc9439636a47e9202f0f6046c WatchSource:0}: Error finding container b3b846eab37925a0df9f6cd9a634843f175bf59fc9439636a47e9202f0f6046c: Status 404 returned error can't find the container with id b3b846eab37925a0df9f6cd9a634843f175bf59fc9439636a47e9202f0f6046c
Jul 17 16:27:48.970909 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.970809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p"]
Jul 17 16:27:48.975860 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:48.975834    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f6cef1_a1ce_4dec_8e35_96bedd751bde.slice/crio-63ae3d966939c262bf2bb160872b30a29a97067073a6dbeded58c5633eb7d458 WatchSource:0}: Error finding container 63ae3d966939c262bf2bb160872b30a29a97067073a6dbeded58c5633eb7d458: Status 404 returned error can't find the container with id 63ae3d966939c262bf2bb160872b30a29a97067073a6dbeded58c5633eb7d458
Jul 17 16:27:48.980242 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.980212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7xm2" event={"ID":"f06ab5f3-e61e-415e-8ec9-329d90f88197","Type":"ContainerStarted","Data":"c1a73be7d03756936fd4a19e968f5e956f1428e59125f8298fcd8080eadc68a6"}
Jul 17 16:27:48.981437 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:48.981412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w" event={"ID":"033ef5f1-3ba2-4148-adfc-4ff2175f51ad","Type":"ContainerStarted","Data":"b3b846eab37925a0df9f6cd9a634843f175bf59fc9439636a47e9202f0f6046c"}
Jul 17 16:27:49.088787 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.088752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:49.459892 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.459861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:49.857293 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.857269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gj7m"
Jul 17 16:27:49.990295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.990256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7xm2" event={"ID":"f06ab5f3-e61e-415e-8ec9-329d90f88197","Type":"ContainerStarted","Data":"3c46228ccd9bee9e95a26d1084b692281865ad65ffd25502e6c168f51b59a216"}
Jul 17 16:27:49.993909 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.993873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p" event={"ID":"77f6cef1-a1ce-4dec-8e35-96bedd751bde","Type":"ContainerStarted","Data":"47a0bcb5c8cac6bd7dd8c6951ee2096598e2abc65ceca02bacb494ec8f54120f"}
Jul 17 16:27:49.994059 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.993933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p" event={"ID":"77f6cef1-a1ce-4dec-8e35-96bedd751bde","Type":"ContainerStarted","Data":"8be8fed3217764c7635138bd814e739cbb9a1d49096e1c78e00c83e63cdc3771"}
Jul 17 16:27:49.994059 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:49.993961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p" event={"ID":"77f6cef1-a1ce-4dec-8e35-96bedd751bde","Type":"ContainerStarted","Data":"63ae3d966939c262bf2bb160872b30a29a97067073a6dbeded58c5633eb7d458"}
Jul 17 16:27:50.750587 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:50.750382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:50.860013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:50.859982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:51.000141 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.000111    2561 generic.go:356] "Generic (PLEG): container finished" podID="f06ab5f3-e61e-415e-8ec9-329d90f88197" containerID="3c46228ccd9bee9e95a26d1084b692281865ad65ffd25502e6c168f51b59a216" exitCode=0
Jul 17 16:27:51.000269 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.000163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7xm2" event={"ID":"f06ab5f3-e61e-415e-8ec9-329d90f88197","Type":"ContainerDied","Data":"3c46228ccd9bee9e95a26d1084b692281865ad65ffd25502e6c168f51b59a216"}
Jul 17 16:27:51.105668 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.105636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:51.205352 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.205325    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b56b8c749-54r5b_b7c1f19f-caaa-4351-bde1-ce134d517809/console/0.log"
Jul 17 16:27:51.229988 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.229961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:51.404946 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.404919    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8998f8dd5-h2rp4_351617e9-712b-40c3-b57a-71c2890b2fff/console/0.log"
Jul 17 16:27:51.607732 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:51.607681    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-nrbnv_2f948260-9c95-4077-8ed2-3a9e53529335/download-server/0.log"
Jul 17 16:27:52.005512 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.005417    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ldwx6_c4615327-7d8b-4645-9bc2-dea866d9673d/serve-healthcheck-canary/0.log"
Jul 17 16:27:52.006413 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.006389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7xm2" event={"ID":"f06ab5f3-e61e-415e-8ec9-329d90f88197","Type":"ContainerStarted","Data":"565c04acb99cbbe9b16808cbe741af2cc224d8fd2b2f42ad183b8319467fdb99"}
Jul 17 16:27:52.006522 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.006483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7xm2" event={"ID":"f06ab5f3-e61e-415e-8ec9-329d90f88197","Type":"ContainerStarted","Data":"1090513360aa58563508a7d734ba2b74c408f355f3369cb871d1b7eeb38842b5"}
Jul 17 16:27:52.008426 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.008398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p" event={"ID":"77f6cef1-a1ce-4dec-8e35-96bedd751bde","Type":"ContainerStarted","Data":"2bd52064cc32404e09bc3d4fddbca0f9abdeeb54d1dc3590097df00c3938a2eb"}
Jul 17 16:27:52.010590 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.010566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w" event={"ID":"033ef5f1-3ba2-4148-adfc-4ff2175f51ad","Type":"ContainerStarted","Data":"8ee63e952f89f49443c5159b4475b4a423dba9ee97d75b5db7b9eac411569d59"}
Jul 17 16:27:52.010693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.010610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w" event={"ID":"033ef5f1-3ba2-4148-adfc-4ff2175f51ad","Type":"ContainerStarted","Data":"1233c10da67b2c10e919206045376bfc8761ff8799abaa13830a0612bf476765"}
Jul 17 16:27:52.010693 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.010629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w" event={"ID":"033ef5f1-3ba2-4148-adfc-4ff2175f51ad","Type":"ContainerStarted","Data":"7566c855eb190e0220fe7dbf5e0c36b05b3ad4c4b3b9b1275f6f2d9ebe7c258d"}
Jul 17 16:27:52.029196 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.029154    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q7xm2" podStartSLOduration=3.320383968 podStartE2EDuration="5.029141934s" totalImagesPullingTime="1.598854837s" totalInitContainerRuntime="109.903129ms" isStatefulPod=true podCreationTimestamp="2026-07-17 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:52.027421894 +0000 UTC m=+106.012509432" watchObservedRunningTime="2026-07-17 16:27:52.029141934 +0000 UTC m=+106.014229472"
Jul 17 16:27:52.047078 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.047022    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-s967w" podStartSLOduration=2.6802288880000003 podStartE2EDuration="5.047009297s" totalImagesPullingTime="2.366780409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:52.04593991 +0000 UTC m=+106.031027469" watchObservedRunningTime="2026-07-17 16:27:52.047009297 +0000 UTC m=+106.032096835"
Jul 17 16:27:52.071841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.071803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-8tc7p" podStartSLOduration=2.970254272 podStartE2EDuration="5.071791013s" totalImagesPullingTime="2.101536741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:52.070592367 +0000 UTC m=+106.055679904" watchObservedRunningTime="2026-07-17 16:27:52.071791013 +0000 UTC m=+106.056878551"
Jul 17 16:27:52.194859 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.194830    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54fb5d875-wssh8"]
Jul 17 16:27:52.214243 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.214217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54fb5d875-wssh8"]
Jul 17 16:27:52.214380 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.214303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xdxc6\""
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216745    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-55aerbq6c5m97\""
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 17 16:27:52.216848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.216715    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 17 16:27:52.281074 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-metrics-server-audit-profiles\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281229 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-tls\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281229 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkmb\" (UniqueName: \"kubernetes.io/projected/d385da69-b81b-49db-b821-137775c5122e-kube-api-access-6rkmb\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-client-certs\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d385da69-b81b-49db-b821-137775c5122e-audit-log\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.281348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.281321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-client-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382462 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d385da69-b81b-49db-b821-137775c5122e-audit-log\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382586 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-client-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382586 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-metrics-server-audit-profiles\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382586 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-tls\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382586 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkmb\" (UniqueName: \"kubernetes.io/projected/d385da69-b81b-49db-b821-137775c5122e-kube-api-access-6rkmb\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382785 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-client-certs\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382785 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.382901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.382872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d385da69-b81b-49db-b821-137775c5122e-audit-log\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.383317 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.383293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.383604 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.383587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d385da69-b81b-49db-b821-137775c5122e-metrics-server-audit-profiles\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.384945 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.384917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-client-certs\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.385159 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.385139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-client-ca-bundle\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.385206 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.385156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d385da69-b81b-49db-b821-137775c5122e-secret-metrics-server-tls\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.391226 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.391206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkmb\" (UniqueName: \"kubernetes.io/projected/d385da69-b81b-49db-b821-137775c5122e-kube-api-access-6rkmb\") pod \"metrics-server-54fb5d875-wssh8\" (UID: \"d385da69-b81b-49db-b821-137775c5122e\") " pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.460849 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.460823    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:52.495628 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.495604    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:27:52.522950 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.522927    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.524308 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.523848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:27:52.529923 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.529896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:27:52.561996 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.561965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:52.584690 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.584654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.585323 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.586107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.586107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.586107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.586107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8snh\" (UniqueName: \"kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.586107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.585647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.669317 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.669271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:52.670188 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.670029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54fb5d875-wssh8"]
Jul 17 16:27:52.672328 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:52.672302    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd385da69_b81b_49db_b821_137775c5122e.slice/crio-3d4158f8df35239c1a82747de161a90c5c6a1109dfc1be7abc0a721fe5ca4d43 WatchSource:0}: Error finding container 3d4158f8df35239c1a82747de161a90c5c6a1109dfc1be7abc0a721fe5ca4d43: Status 404 returned error can't find the container with id 3d4158f8df35239c1a82747de161a90c5c6a1109dfc1be7abc0a721fe5ca4d43
Jul 17 16:27:52.686952 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.686931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687066 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.686958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8snh\" (UniqueName: \"kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687066 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.686980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687066 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.687023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.687078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.687108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.687152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.687885 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.687861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.688058 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.688003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.688058 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.688014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.688423 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.688403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.689632 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.689608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.689711 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.689671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.695402 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.695383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8snh\" (UniqueName: \"kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh\") pod \"console-84dfdcc964-6tfhv\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") " pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.760987 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.760958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:52.834754 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.834686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:27:52.982530 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:52.982510    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:27:52.984912 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:52.984882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ec6481_b2b5_4cc7_a90f_c7b4280036c5.slice/crio-94133eac85ed0d15595e312c45a33a3e2311d993c7dac730cefc1a63dc1b0b06 WatchSource:0}: Error finding container 94133eac85ed0d15595e312c45a33a3e2311d993c7dac730cefc1a63dc1b0b06: Status 404 returned error can't find the container with id 94133eac85ed0d15595e312c45a33a3e2311d993c7dac730cefc1a63dc1b0b06
Jul 17 16:27:53.015323 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:53.015295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8" event={"ID":"d385da69-b81b-49db-b821-137775c5122e","Type":"ContainerStarted","Data":"3d4158f8df35239c1a82747de161a90c5c6a1109dfc1be7abc0a721fe5ca4d43"}
Jul 17 16:27:53.018869 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:53.018830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dfdcc964-6tfhv" event={"ID":"36ec6481-b2b5-4cc7-a90f-c7b4280036c5","Type":"ContainerStarted","Data":"94133eac85ed0d15595e312c45a33a3e2311d993c7dac730cefc1a63dc1b0b06"}
Jul 17 16:27:53.987529 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:53.987494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:54.001777 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:53.998368    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:27:54.027824 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.027465    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:27:54.027824 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.027534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dfdcc964-6tfhv" event={"ID":"36ec6481-b2b5-4cc7-a90f-c7b4280036c5","Type":"ContainerStarted","Data":"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"}
Jul 17 16:27:54.027824 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.027646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.032459 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kxfk8\""
Jul 17 16:27:54.032676 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032658    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 16:27:54.032773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032681    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 16:27:54.032838 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032448    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 16:27:54.032957 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032930    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 16:27:54.033030 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032971    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 16:27:54.033030 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.032995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 16:27:54.033172 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033093    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 16:27:54.033172 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 16:27:54.033172 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 16:27:54.033325 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 16:27:54.033390 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033367    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 16:27:54.033677 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 16:27:54.033949 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.033929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 16:27:54.034156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.034141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bidrh4ndp8l84\""
Jul 17 16:27:54.036257 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.036230    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 16:27:54.050331 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.050281    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84dfdcc964-6tfhv" podStartSLOduration=2.050262996 podStartE2EDuration="2.050262996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:54.049950112 +0000 UTC m=+108.035037688" watchObservedRunningTime="2026-07-17 16:27:54.050262996 +0000 UTC m=+108.035350527"
Jul 17 16:27:54.104014 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.103983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104201 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104201 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104201 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104201 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104415 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104415 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104505 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104505 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104719 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpphz\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.104951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.105133 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.105133 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.104984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.105133 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.105009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.105133 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.105066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206124 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.206720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpphz\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.207295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.207295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.206903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.207295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.207039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.207451 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.207298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.208992 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.208497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.210385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.210398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.212451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.212636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213598 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.213572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213718 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.213688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213816 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.213797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213892 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.213877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.213951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.213937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.216661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.214870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.216661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.215291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.216661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.215699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.217121 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.216928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.217937 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.217894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.217937 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.217920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpphz\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz\") pod \"prometheus-k8s-0\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.342779 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.342747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:27:54.503247 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:54.503199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:27:54.650188 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:54.650114    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f61e205_340d_4f54_bebe_106d079e4447.slice/crio-fe7b83aea94b1b0c95ad97f86f5973e8d957623de53dbd0fa00d39257cf50401 WatchSource:0}: Error finding container fe7b83aea94b1b0c95ad97f86f5973e8d957623de53dbd0fa00d39257cf50401: Status 404 returned error can't find the container with id fe7b83aea94b1b0c95ad97f86f5973e8d957623de53dbd0fa00d39257cf50401
Jul 17 16:27:55.032306 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:55.032268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"fe7b83aea94b1b0c95ad97f86f5973e8d957623de53dbd0fa00d39257cf50401"}
Jul 17 16:27:55.033744 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:55.033718    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8" event={"ID":"d385da69-b81b-49db-b821-137775c5122e","Type":"ContainerStarted","Data":"95b26f47f09a416b0b73aa1363d76743230b25b3a39d51f74d782efa906de545"}
Jul 17 16:27:55.064552 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:55.064512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8" podStartSLOduration=1.060150246 podStartE2EDuration="3.064498479s" totalImagesPullingTime="2.004348233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:55.064348394 +0000 UTC m=+109.049435948" watchObservedRunningTime="2026-07-17 16:27:55.064498479 +0000 UTC m=+109.049586017"
Jul 17 16:27:56.542662 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.542632    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:27:56.627032 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.626997    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:27:56.642901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.642870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.657084 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.657063    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:27:56.727789 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.727762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:56.730752 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.730859 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.730859 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.730936 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.730982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.731026 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.730978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh24n\" (UniqueName: \"kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.731026 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.731012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831512 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831512 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831512 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24n\" (UniqueName: \"kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.831882 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.831767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.832436 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.832413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.832553 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.832413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.832553 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.832528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.832553 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.832542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.834072 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.834034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.834191 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.834173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.839389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.839364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:27:56.842914 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.842893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh24n\" (UniqueName: \"kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n\") pod \"console-66644b6b6f-wbrrn\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") " pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:56.953156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:56.953135    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:27:57.046170 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:57.046082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"}
Jul 17 16:27:57.047057 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:57.047017    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa" exitCode=0
Jul 17 16:27:57.129983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:57.129648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:27:57.132226 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:27:57.132196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d939392_fd63_47ca_874d_65f3078c315b.slice/crio-07398d9e728f0b09992bf6dae4de990721e5be92e9a53d53ca33ff2f6590915a WatchSource:0}: Error finding container 07398d9e728f0b09992bf6dae4de990721e5be92e9a53d53ca33ff2f6590915a: Status 404 returned error can't find the container with id 07398d9e728f0b09992bf6dae4de990721e5be92e9a53d53ca33ff2f6590915a
Jul 17 16:27:58.053761 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:58.053724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66644b6b6f-wbrrn" event={"ID":"9d939392-fd63-47ca-874d-65f3078c315b","Type":"ContainerStarted","Data":"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"}
Jul 17 16:27:58.054327 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:58.053783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66644b6b6f-wbrrn" event={"ID":"9d939392-fd63-47ca-874d-65f3078c315b","Type":"ContainerStarted","Data":"07398d9e728f0b09992bf6dae4de990721e5be92e9a53d53ca33ff2f6590915a"}
Jul 17 16:27:58.074848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:27:58.074780    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66644b6b6f-wbrrn" podStartSLOduration=2.074763608 podStartE2EDuration="2.074763608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:27:58.073688864 +0000 UTC m=+112.058776402" watchObservedRunningTime="2026-07-17 16:27:58.074763608 +0000 UTC m=+112.059851146"
Jul 17 16:28:01.074983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:01.074805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"}
Jul 17 16:28:01.074983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:01.074890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"}
Jul 17 16:28:02.835812 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:02.835771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:28:04.032359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.032315    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b56b8c749-54r5b" podUID="b7c1f19f-caaa-4351-bde1-ce134d517809" containerName="console" containerID="cri-o://9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3" gracePeriod=15
Jul 17 16:28:04.097058 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.096837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"}
Jul 17 16:28:04.097058 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.096927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"}
Jul 17 16:28:04.379722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.379593    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:28:04.512302 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512260    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.512492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.512492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512371    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5s5p\" (UniqueName: \"kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.512492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.512492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512480    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.512722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.512547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca\") pod \"b7c1f19f-caaa-4351-bde1-ce134d517809\" (UID: \"b7c1f19f-caaa-4351-bde1-ce134d517809\") "
Jul 17 16:28:04.513552 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.513517    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:04.514540 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.513988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config" (OuterVolumeSpecName: "console-config") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:04.514540 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.514288    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:04.520175 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.519235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p" (OuterVolumeSpecName: "kube-api-access-z5s5p") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "kube-api-access-z5s5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:28:04.521603 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.521575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:04.521817 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.521790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7c1f19f-caaa-4351-bde1-ce134d517809" (UID: "b7c1f19f-caaa-4351-bde1-ce134d517809"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:04.613634 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613600    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:04.613634 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613635    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:04.613634 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613650    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:04.613854 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613662    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7c1f19f-caaa-4351-bde1-ce134d517809-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:04.613854 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613675    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5s5p\" (UniqueName: \"kubernetes.io/projected/b7c1f19f-caaa-4351-bde1-ce134d517809-kube-api-access-z5s5p\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:04.613854 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:04.613691    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7c1f19f-caaa-4351-bde1-ce134d517809-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:05.102716 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.102649    2561 generic.go:356] "Generic (PLEG): container finished" podID="b7c1f19f-caaa-4351-bde1-ce134d517809" containerID="9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3" exitCode=0
Jul 17 16:28:05.102716 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.102698    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b56b8c749-54r5b"
Jul 17 16:28:05.103220 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.102728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56b8c749-54r5b" event={"ID":"b7c1f19f-caaa-4351-bde1-ce134d517809","Type":"ContainerDied","Data":"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"}
Jul 17 16:28:05.103220 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.102783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b56b8c749-54r5b" event={"ID":"b7c1f19f-caaa-4351-bde1-ce134d517809","Type":"ContainerDied","Data":"ce9263739013ab3a7253a0aaa8dc3514d96b4bd7738f35bc43e8f2bd17cbbe91"}
Jul 17 16:28:05.103220 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.102811    2561 scope.go:117] "RemoveContainer" containerID="9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"
Jul 17 16:28:05.106596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.106573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"}
Jul 17 16:28:05.106747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.106644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerStarted","Data":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"}
Jul 17 16:28:05.111991 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.111933    2561 scope.go:117] "RemoveContainer" containerID="9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"
Jul 17 16:28:05.112224 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:28:05.112206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3\": container with ID starting with 9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3 not found: ID does not exist" containerID="9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"
Jul 17 16:28:05.112303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.112232    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3"} err="failed to get container status \"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3\": rpc error: code = NotFound desc = could not find container \"9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3\": container with ID starting with 9ba19fda70954f83768e6aafb6f513d3e735f0e60695c97d72d2b74d90bff5d3 not found: ID does not exist"
Jul 17 16:28:05.145070 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.144971    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.274932465 podStartE2EDuration="12.144943233s" totalImagesPullingTime="7.846094119s" totalInitContainerRuntime="23.916649ms" isStatefulPod=false podCreationTimestamp="2026-07-17 16:27:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:28:05.143032152 +0000 UTC m=+119.128119717" watchObservedRunningTime="2026-07-17 16:28:05.144943233 +0000 UTC m=+119.130030780"
Jul 17 16:28:05.164073 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.164033    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:28:05.166871 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:05.166848    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b56b8c749-54r5b"]
Jul 17 16:28:06.514273 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:06.514245    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c1f19f-caaa-4351-bde1-ce134d517809" path="/var/lib/kubelet/pods/b7c1f19f-caaa-4351-bde1-ce134d517809/volumes"
Jul 17 16:28:06.954183 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:06.954153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:28:06.954332 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:06.954210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:28:06.958572 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:06.958548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:28:07.119733 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:07.119710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:28:07.235613 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:07.235535    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:28:07.473063 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:07.473007    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:28:07.568280 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:07.568247    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:28:09.343144 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:09.343109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:28:12.524867 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:12.524836    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:28:12.525262 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:12.524884    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:28:21.565293 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.565229    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84dfdcc964-6tfhv" podUID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" containerName="console" containerID="cri-o://e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee" gracePeriod=15
Jul 17 16:28:21.808852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.808831    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:28:21.956340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956497 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956554 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956531    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956618 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956602    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956667 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956667 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8snh\" (UniqueName: \"kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh\") pod \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\" (UID: \"36ec6481-b2b5-4cc7-a90f-c7b4280036c5\") "
Jul 17 16:28:21.956772 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca" (OuterVolumeSpecName: "service-ca") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:21.956982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956931    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config" (OuterVolumeSpecName: "console-config") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:21.956982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956976    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:21.957121 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.956971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:21.957121 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.957076    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:21.958464 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.958438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:21.958579 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.958530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:21.958579 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:21.958556    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh" (OuterVolumeSpecName: "kube-api-access-z8snh") pod "36ec6481-b2b5-4cc7-a90f-c7b4280036c5" (UID: "36ec6481-b2b5-4cc7-a90f-c7b4280036c5"). InnerVolumeSpecName "kube-api-access-z8snh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:28:22.057738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057711    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.057738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057735    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.057738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057744    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8snh\" (UniqueName: \"kubernetes.io/projected/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-kube-api-access-z8snh\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.057921 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057754    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.057921 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057762    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.057921 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.057772    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36ec6481-b2b5-4cc7-a90f-c7b4280036c5-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:22.174804 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.174767    2561 generic.go:356] "Generic (PLEG): container finished" podID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" containerID="e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee" exitCode=0
Jul 17 16:28:22.174948 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.174811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dfdcc964-6tfhv"
Jul 17 16:28:22.174948 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.174853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dfdcc964-6tfhv" event={"ID":"36ec6481-b2b5-4cc7-a90f-c7b4280036c5","Type":"ContainerDied","Data":"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"}
Jul 17 16:28:22.174948 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.174923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dfdcc964-6tfhv" event={"ID":"36ec6481-b2b5-4cc7-a90f-c7b4280036c5","Type":"ContainerDied","Data":"94133eac85ed0d15595e312c45a33a3e2311d993c7dac730cefc1a63dc1b0b06"}
Jul 17 16:28:22.174948 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.174944    2561 scope.go:117] "RemoveContainer" containerID="e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"
Jul 17 16:28:22.183377 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.183362    2561 scope.go:117] "RemoveContainer" containerID="e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"
Jul 17 16:28:22.183600 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:28:22.183583    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee\": container with ID starting with e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee not found: ID does not exist" containerID="e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"
Jul 17 16:28:22.183645 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.183608    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee"} err="failed to get container status \"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee\": rpc error: code = NotFound desc = could not find container \"e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee\": container with ID starting with e11d7dba9114cef8b5ca910ffc7a3633f5b3fef5ee67e605fc3c50f53b4688ee not found: ID does not exist"
Jul 17 16:28:22.194945 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.194927    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:28:22.199366 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.199347    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84dfdcc964-6tfhv"]
Jul 17 16:28:22.512985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:22.512944    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" path="/var/lib/kubelet/pods/36ec6481-b2b5-4cc7-a90f-c7b4280036c5/volumes"
Jul 17 16:28:32.255353 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.255318    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8998f8dd5-h2rp4" podUID="351617e9-712b-40c3-b57a-71c2890b2fff" containerName="console" containerID="cri-o://a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80" gracePeriod=15
Jul 17 16:28:32.506457 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.506398    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:28:32.531226 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.531202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:28:32.535030 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.535012    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54fb5d875-wssh8"
Jul 17 16:28:32.544569 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.544547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.545090 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.544901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:32.645079 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9lmm\" (UniqueName: \"kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645245 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645245 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645245 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645410 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645410 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle\") pod \"351617e9-712b-40c3-b57a-71c2890b2fff\" (UID: \"351617e9-712b-40c3-b57a-71c2890b2fff\") "
Jul 17 16:28:32.645524 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca" (OuterVolumeSpecName: "service-ca") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:32.645576 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config" (OuterVolumeSpecName: "console-config") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:32.645720 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645703    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.645799 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645724    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.645799 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645737    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.645799 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.645759    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:28:32.647216 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.647192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:32.647329 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.647313    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:28:32.647401 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.647380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm" (OuterVolumeSpecName: "kube-api-access-h9lmm") pod "351617e9-712b-40c3-b57a-71c2890b2fff" (UID: "351617e9-712b-40c3-b57a-71c2890b2fff"). InnerVolumeSpecName "kube-api-access-h9lmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:28:32.746443 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.746418    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lmm\" (UniqueName: \"kubernetes.io/projected/351617e9-712b-40c3-b57a-71c2890b2fff-kube-api-access-h9lmm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.746443 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.746439    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.746443 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.746448    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/351617e9-712b-40c3-b57a-71c2890b2fff-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:32.746606 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:32.746457    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/351617e9-712b-40c3-b57a-71c2890b2fff-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:28:33.218826 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.218787    2561 generic.go:356] "Generic (PLEG): container finished" podID="351617e9-712b-40c3-b57a-71c2890b2fff" containerID="a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80" exitCode=0
Jul 17 16:28:33.219747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.219511    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8998f8dd5-h2rp4"
Jul 17 16:28:33.219747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.219664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8998f8dd5-h2rp4" event={"ID":"351617e9-712b-40c3-b57a-71c2890b2fff","Type":"ContainerDied","Data":"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"}
Jul 17 16:28:33.219747 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.219740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8998f8dd5-h2rp4" event={"ID":"351617e9-712b-40c3-b57a-71c2890b2fff","Type":"ContainerDied","Data":"aa88ff4aa98fd90d84efd7f23c83890b293cc868b9e70386cf607a923954d327"}
Jul 17 16:28:33.220008 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.219772    2561 scope.go:117] "RemoveContainer" containerID="a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"
Jul 17 16:28:33.230838 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.230815    2561 scope.go:117] "RemoveContainer" containerID="a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"
Jul 17 16:28:33.231179 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:28:33.231148    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80\": container with ID starting with a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80 not found: ID does not exist" containerID="a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"
Jul 17 16:28:33.231259 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.231191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80"} err="failed to get container status \"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80\": rpc error: code = NotFound desc = could not find container \"a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80\": container with ID starting with a41bb7c25e11d5ac388cf4881fed4b6112bcf66a35e54011896bb327aba68f80 not found: ID does not exist"
Jul 17 16:28:33.259087 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.259036    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:28:33.265582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:33.265557    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8998f8dd5-h2rp4"]
Jul 17 16:28:34.513576 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:34.513539    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351617e9-712b-40c3-b57a-71c2890b2fff" path="/var/lib/kubelet/pods/351617e9-712b-40c3-b57a-71c2890b2fff/volumes"
Jul 17 16:28:54.343136 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:54.343103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:28:54.361602 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:54.361576    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:28:55.318336 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:28:55.318311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:03.887385 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:03.887301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:03.986829 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:03.986803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:04.086965 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:04.086912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:04.142430 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:04.142369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.007321 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.007283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.085726 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.085680    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.218740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.218712    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:29:10.219338 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219318    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c1f19f-caaa-4351-bde1-ce134d517809" containerName="console"
Jul 17 16:29:10.219463 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219452    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c1f19f-caaa-4351-bde1-ce134d517809" containerName="console"
Jul 17 16:29:10.219541 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219532    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" containerName="console"
Jul 17 16:29:10.219609 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" containerName="console"
Jul 17 16:29:10.219689 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351617e9-712b-40c3-b57a-71c2890b2fff" containerName="console"
Jul 17 16:29:10.219755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219747    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="351617e9-712b-40c3-b57a-71c2890b2fff" containerName="console"
Jul 17 16:29:10.219888 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219879    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="351617e9-712b-40c3-b57a-71c2890b2fff" containerName="console"
Jul 17 16:29:10.219960 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.219952    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ec6481-b2b5-4cc7-a90f-c7b4280036c5" containerName="console"
Jul 17 16:29:10.220021 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.220013    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c1f19f-caaa-4351-bde1-ce134d517809" containerName="console"
Jul 17 16:29:10.225017 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.224994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.232368 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.232348    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:29:10.317122 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbps\" (UniqueName: \"kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317234 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317368 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.317420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.317395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.351932 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.351911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.418564 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418653 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418653 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbps\" (UniqueName: \"kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418653 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418765 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418804 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.418851 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.418820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.419366 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.419334    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.419499 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.419478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.419570 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.419478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.419623 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.419596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.420862 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.420839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.421232 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.421210    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.426911 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.426889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbps\" (UniqueName: \"kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps\") pod \"console-cfb5c7f97-x4jgm\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") " pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.452907 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.452886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.517295 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.517273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:10.534310 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.534297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:10.666558 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:10.662613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:29:10.668148 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:29:10.668116    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0388fc_ef73_47ad_9ff0_961f8c32e4af.slice/crio-a6c5c2b2a9bc07c0f24312a3b7a8978691c769f3dfea2d4f581cd5ae448c0f66 WatchSource:0}: Error finding container a6c5c2b2a9bc07c0f24312a3b7a8978691c769f3dfea2d4f581cd5ae448c0f66: Status 404 returned error can't find the container with id a6c5c2b2a9bc07c0f24312a3b7a8978691c769f3dfea2d4f581cd5ae448c0f66
Jul 17 16:29:11.212317 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:11.212284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:11.319242 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:11.319206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:11.366743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:11.366715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb5c7f97-x4jgm" event={"ID":"1d0388fc-ef73-47ad-9ff0-961f8c32e4af","Type":"ContainerStarted","Data":"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"}
Jul 17 16:29:11.366848 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:11.366767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb5c7f97-x4jgm" event={"ID":"1d0388fc-ef73-47ad-9ff0-961f8c32e4af","Type":"ContainerStarted","Data":"a6c5c2b2a9bc07c0f24312a3b7a8978691c769f3dfea2d4f581cd5ae448c0f66"}
Jul 17 16:29:11.384653 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:11.384610    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cfb5c7f97-x4jgm" podStartSLOduration=1.384596502 podStartE2EDuration="1.384596502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:29:11.383208034 +0000 UTC m=+185.368295595" watchObservedRunningTime="2026-07-17 16:29:11.384596502 +0000 UTC m=+185.369684038"
Jul 17 16:29:12.433675 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.433640    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434027    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="prometheus" containerID="cri-o://08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06" gracePeriod=600
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434068    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-thanos" containerID="cri-o://18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2" gracePeriod=600
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434156    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-web" containerID="cri-o://1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984" gracePeriod=600
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434234    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="thanos-sidecar" containerID="cri-o://df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6" gracePeriod=600
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434249    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="config-reloader" containerID="cri-o://22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c" gracePeriod=600
Jul 17 16:29:12.434363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.434269    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy" containerID="cri-o://22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e" gracePeriod=600
Jul 17 16:29:12.679504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.679482    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:12.737637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737637 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737718    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737757    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpphz\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.737897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737883    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.737988    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.738335 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738224    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets\") pod \"6f61e205-340d-4f54-bebe-106d079e4447\" (UID: \"6f61e205-340d-4f54-bebe-106d079e4447\") "
Jul 17 16:29:12.738843 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.738843 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738495    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.738843 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.738517    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.740013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.739001    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.740013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.739124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.740013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.739702    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.740013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.739742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:29:12.740013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.739809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:12.741980 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.741948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:29:12.742120 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.742038    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.742617 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.742550    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out" (OuterVolumeSpecName: "config-out") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:29:12.742943 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.742919    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.743319 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.743299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.743523 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.743483    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.743742 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.743696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config" (OuterVolumeSpecName: "config") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.746349 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.744133    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.746843 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.746805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.746960 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.746938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.746960 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.746948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz" (OuterVolumeSpecName: "kube-api-access-mpphz") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "kube-api-access-mpphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:29:12.757392 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.755820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config" (OuterVolumeSpecName: "web-config") pod "6f61e205-340d-4f54-bebe-106d079e4447" (UID: "6f61e205-340d-4f54-bebe-106d079e4447"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:12.839355 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839319    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-kube-rbac-proxy\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839355 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839352    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839355 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839362    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839371    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839382    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-metrics-client-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839391    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839399    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839407    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpphz\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-kube-api-access-mpphz\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839416    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-prometheus-k8s-db\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839424    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-web-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839432    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-metrics-client-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839441    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f61e205-340d-4f54-bebe-106d079e4447-config-out\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839448    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-grpc-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839455    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f61e205-340d-4f54-bebe-106d079e4447-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839463    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839470    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839479    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6f61e205-340d-4f54-bebe-106d079e4447-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:12.839546 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:12.839488    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f61e205-340d-4f54-bebe-106d079e4447-tls-assets\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:13.378148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378121    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2" exitCode=0
Jul 17 16:29:13.378148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378145    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e" exitCode=0
Jul 17 16:29:13.378148 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378152    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984" exitCode=0
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378158    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6" exitCode=0
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378163    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c" exitCode=0
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378169    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f61e205-340d-4f54-bebe-106d079e4447" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06" exitCode=0
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378212    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378332    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"}
Jul 17 16:29:13.378414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.378394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6f61e205-340d-4f54-bebe-106d079e4447","Type":"ContainerDied","Data":"fe7b83aea94b1b0c95ad97f86f5973e8d957623de53dbd0fa00d39257cf50401"}
Jul 17 16:29:13.386724 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.386702    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.393835 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.393819    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.399908 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.399891    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.406840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.406825    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.407131 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.407064    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:13.411414 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.411394    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:13.413692 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.413675    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.420487 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.420471    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.426402 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.426386    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.426640 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.426614    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.426711 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.426642    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.426711 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.426667    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.426919 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.426903    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.426957 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.426925    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.426957 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.426943    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.427209 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.427191    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.427294 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427212    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.427294 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427228    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.427464 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.427451    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.427504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427466    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.427504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427478    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.427670 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.427655    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.427705 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427672    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.427705 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427683    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.427866 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.427851    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.427914 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427869    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.427914 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.427880    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.428075 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:13.428058    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.428151 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.428151 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428086    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.428356 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428331    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.428356 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428356    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.428589 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428572    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.428624 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428591    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.428797 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428782    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.428831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.428799    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.429026 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429009    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.429087 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429027    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.429254 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429238    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.429293 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429254    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.429435 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429420    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.429475 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429435    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.429634 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429618    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.429689 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429634    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.429797 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429782    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.429833 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429797    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.429981 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.429981 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.429980    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.430200 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430183    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.430260 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430201    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.430388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430373    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.430434 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430388    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.430591 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430576    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.430591 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430591    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.430801 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430780    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.430801 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.430802    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.431021 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431006    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.431021 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431021    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.431251 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431234    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.431311 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431251    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.431463 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431447    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.431463 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431462    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.431660 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431641    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.431699 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431662    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.431874 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431856    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.431915 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.431875    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.432098 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432080    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.432156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432100    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.432307 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432290    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.432307 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432306    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.432503 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432485    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.432547 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432505    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.432707 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432692    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.432707 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432706    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.432917 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432899    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.432990 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.432931    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.433178 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433159    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.433222 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433179    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.433364 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433349    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.433399 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433366    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.433556 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433542    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.433599 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433557    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.433728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433711    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.433952 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433729    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.433952 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433917    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.433952 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.433930    2561 scope.go:117] "RemoveContainer" containerID="18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"
Jul 17 16:29:13.434185 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434157    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2"} err="failed to get container status \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": rpc error: code = NotFound desc = could not find container \"18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2\": container with ID starting with 18af329a80331ce13721b54589301501dd1f9b0916417d51bfc8577848318da2 not found: ID does not exist"
Jul 17 16:29:13.434231 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434186    2561 scope.go:117] "RemoveContainer" containerID="22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"
Jul 17 16:29:13.434375 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434357    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e"} err="failed to get container status \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": rpc error: code = NotFound desc = could not find container \"22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e\": container with ID starting with 22444fe18b91baf8a0a6a5c6345f4be7d969a3e850144e17828955628a33f95e not found: ID does not exist"
Jul 17 16:29:13.434408 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434379    2561 scope.go:117] "RemoveContainer" containerID="1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"
Jul 17 16:29:13.434582 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434567    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984"} err="failed to get container status \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": rpc error: code = NotFound desc = could not find container \"1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984\": container with ID starting with 1d10d24b8df7eabd73dd447b4183e643a983c5b67a81d1083eec7345897e6984 not found: ID does not exist"
Jul 17 16:29:13.434628 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434582    2561 scope.go:117] "RemoveContainer" containerID="df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"
Jul 17 16:29:13.434759 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434741    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6"} err="failed to get container status \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": rpc error: code = NotFound desc = could not find container \"df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6\": container with ID starting with df57bd08413a9fde422c2f0f0ec528f7d17a5668613ae2f1270aa66c96e695b6 not found: ID does not exist"
Jul 17 16:29:13.434828 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434760    2561 scope.go:117] "RemoveContainer" containerID="22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"
Jul 17 16:29:13.434938 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434924    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c"} err="failed to get container status \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": rpc error: code = NotFound desc = could not find container \"22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c\": container with ID starting with 22289ef79bacf735ac035b4fb9d1f272079314c202714c77e2f7714a3c8e883c not found: ID does not exist"
Jul 17 16:29:13.434982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.434938    2561 scope.go:117] "RemoveContainer" containerID="08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"
Jul 17 16:29:13.435138 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.435120    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06"} err="failed to get container status \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": rpc error: code = NotFound desc = could not find container \"08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06\": container with ID starting with 08e9ec85c0ffd101d5f67450984281d27d92071bf63d9fd36f5dc02a8d4fbc06 not found: ID does not exist"
Jul 17 16:29:13.435211 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.435139    2561 scope.go:117] "RemoveContainer" containerID="a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"
Jul 17 16:29:13.435359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.435341    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa"} err="failed to get container status \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": rpc error: code = NotFound desc = could not find container \"a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa\": container with ID starting with a31f099cd1b438b7e6856757ea220e5b67df3facf044d6f5e0cc6755887678aa not found: ID does not exist"
Jul 17 16:29:13.468997 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.468976    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:13.469264 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469251    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="config-reloader"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469266    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="config-reloader"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469277    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-thanos"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469283    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-thanos"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469294    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469299    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469308    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="prometheus"
Jul 17 16:29:13.469314 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469314    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="prometheus"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469321    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="init-config-reloader"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469327    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="init-config-reloader"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469335    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="thanos-sidecar"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469339    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="thanos-sidecar"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469345    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-web"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469349    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-web"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469393    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="config-reloader"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469400    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-web"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469408    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="prometheus"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469414    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy-thanos"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469421    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="thanos-sidecar"
Jul 17 16:29:13.469506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.469426    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f61e205-340d-4f54-bebe-106d079e4447" containerName="kube-rbac-proxy"
Jul 17 16:29:13.474350 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.474336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.480590 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.480573    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 16:29:13.480777 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.480710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.481988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.482197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.483588    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.483802    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.483839    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.483957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.484035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 16:29:13.484386 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.484276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 16:29:13.487998 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.487972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 16:29:13.488124 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.487980    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kxfk8\""
Jul 17 16:29:13.488215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.488120    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bidrh4ndp8l84\""
Jul 17 16:29:13.488215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.488127    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 16:29:13.488215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.488124    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 16:29:13.491484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.491468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 16:29:13.506611 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.506592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:13.544157 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544157 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544315 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544315 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544315 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfxm\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-kube-api-access-plfxm\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544315 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544315 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-web-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config-out\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.544831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.544667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645500 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645500 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645627 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645627 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645627 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config-out\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.645773 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646068 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646068 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646068 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646068 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.645984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfxm\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-kube-api-access-plfxm\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646068 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-web-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646301 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.646575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.646558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.647478 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.647450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.647574 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.647457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.647633 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.647586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.649230 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.648780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config-out\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.649230 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.649121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.649230 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.649186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.649533 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.649513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.650155 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.650111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651151 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651277 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651331 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-web-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651331 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651467 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-config\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651822 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.651931 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.651913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.654473 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.654453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfxm\" (UniqueName: \"kubernetes.io/projected/74245a6a-bb1a-42f7-b94a-aaa2ba733b77-kube-api-access-plfxm\") pod \"prometheus-k8s-0\" (UID: \"74245a6a-bb1a-42f7-b94a-aaa2ba733b77\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.787783 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.787750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:13.925575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:13.925551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 16:29:13.927377 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:29:13.927344    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74245a6a_bb1a_42f7_b94a_aaa2ba733b77.slice/crio-7b6696df92c966c2db1ee6e394173b35c2f9aae680bb82cc1ccd776048c6461b WatchSource:0}: Error finding container 7b6696df92c966c2db1ee6e394173b35c2f9aae680bb82cc1ccd776048c6461b: Status 404 returned error can't find the container with id 7b6696df92c966c2db1ee6e394173b35c2f9aae680bb82cc1ccd776048c6461b
Jul 17 16:29:14.384600 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:14.384570    2561 generic.go:356] "Generic (PLEG): container finished" podID="74245a6a-bb1a-42f7-b94a-aaa2ba733b77" containerID="dba4745c8490418b02e49926353cfd22c116569d002e0a9fb7669334996265cd" exitCode=0
Jul 17 16:29:14.384731 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:14.384617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerDied","Data":"dba4745c8490418b02e49926353cfd22c116569d002e0a9fb7669334996265cd"}
Jul 17 16:29:14.384731 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:14.384669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"7b6696df92c966c2db1ee6e394173b35c2f9aae680bb82cc1ccd776048c6461b"}
Jul 17 16:29:14.514535 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:14.514513    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f61e205-340d-4f54-bebe-106d079e4447" path="/var/lib/kubelet/pods/6f61e205-340d-4f54-bebe-106d079e4447/volumes"
Jul 17 16:29:15.392986 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.392939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"18638f073d7357ec00127ff13ebb3f87cb1045727496960a3ba4ce2af4a62475"}
Jul 17 16:29:15.393150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.393025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"232f466b1b851b21971079f16e32bd4a9545bce45171db2f52e8bd1bc42b905d"}
Jul 17 16:29:15.393150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.393070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"9a9742698aad258854c88b43ff0df03feaf6d6db7a7ca920620a37e5bbe56bfa"}
Jul 17 16:29:15.393150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.393108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"52ff85f07461fcb9b72513d2932bc1ea09d069d02cf3ef627d4a8d640c31a398"}
Jul 17 16:29:15.393150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.393136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"bc8279ffe457d4a8c686d0abdafb4e64622b6796902a84ad485502400db2a924"}
Jul 17 16:29:15.393280 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.393161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74245a6a-bb1a-42f7-b94a-aaa2ba733b77","Type":"ContainerStarted","Data":"ecf9c18fb99540e5415fd8249a88c8e2be2c9a60e06e427cb6aa32f3b53847d8"}
Jul 17 16:29:15.430825 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.430783    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.413487312 podStartE2EDuration="2.430767469s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.280157ms" isStatefulPod=false podCreationTimestamp="2026-07-17 16:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:29:15.428269733 +0000 UTC m=+189.413357290" watchObservedRunningTime="2026-07-17 16:29:15.430767469 +0000 UTC m=+189.415855006"
Jul 17 16:29:15.832619 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.832593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:15.900071 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:15.900035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:18.788408 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:18.788375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:29:20.215728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:20.215696    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:20.282470 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:20.282440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:20.535106 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:20.535077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:20.535220 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:20.535130    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:20.539590 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:20.539572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:21.421552 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:21.421525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:29:21.472152 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:21.472120    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:29:21.588738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:21.588710    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:21.688472 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:21.688404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:21.749793 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:21.749762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:26.656673 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:26.656637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:26.716093 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:26.716064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:30.071297 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:30.071259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:30.132321 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:30.132287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:41.677256 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:41.677222    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:41.777632 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:41.777597    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:29:46.492465 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.492410    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66644b6b6f-wbrrn" podUID="9d939392-fd63-47ca-874d-65f3078c315b" containerName="console" containerID="cri-o://157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc" gracePeriod=15
Jul 17 16:29:46.740135 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.740114    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:29:46.893794 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.893956 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh24n\" (UniqueName: \"kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.893956 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.893956 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.893956 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893913    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.894207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.894207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.893991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert\") pod \"9d939392-fd63-47ca-874d-65f3078c315b\" (UID: \"9d939392-fd63-47ca-874d-65f3078c315b\") "
Jul 17 16:29:46.894313 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.894242    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config" (OuterVolumeSpecName: "console-config") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:46.894367 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.894304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:46.894531 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.894495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:46.894664 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.894642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:29:46.895972 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.895945    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:46.896123 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.895979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:29:46.896123 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.896001    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n" (OuterVolumeSpecName: "kube-api-access-fh24n") pod "9d939392-fd63-47ca-874d-65f3078c315b" (UID: "9d939392-fd63-47ca-874d-65f3078c315b"). InnerVolumeSpecName "kube-api-access-fh24n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:29:46.995115 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995088    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh24n\" (UniqueName: \"kubernetes.io/projected/9d939392-fd63-47ca-874d-65f3078c315b-kube-api-access-fh24n\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995115 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995115    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995292 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995129    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995292 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995140    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995292 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995152    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d939392-fd63-47ca-874d-65f3078c315b-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995292 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995164    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:46.995292 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:46.995177    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d939392-fd63-47ca-874d-65f3078c315b-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:29:47.530116 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.530080    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d939392-fd63-47ca-874d-65f3078c315b" containerID="157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc" exitCode=0
Jul 17 16:29:47.530602 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.530195    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66644b6b6f-wbrrn"
Jul 17 16:29:47.530602 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.530205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66644b6b6f-wbrrn" event={"ID":"9d939392-fd63-47ca-874d-65f3078c315b","Type":"ContainerDied","Data":"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"}
Jul 17 16:29:47.530602 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.530253    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66644b6b6f-wbrrn" event={"ID":"9d939392-fd63-47ca-874d-65f3078c315b","Type":"ContainerDied","Data":"07398d9e728f0b09992bf6dae4de990721e5be92e9a53d53ca33ff2f6590915a"}
Jul 17 16:29:47.530602 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.530280    2561 scope.go:117] "RemoveContainer" containerID="157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"
Jul 17 16:29:47.540150 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.540135    2561 scope.go:117] "RemoveContainer" containerID="157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"
Jul 17 16:29:47.540397 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:29:47.540379    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc\": container with ID starting with 157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc not found: ID does not exist" containerID="157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"
Jul 17 16:29:47.540437 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.540406    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc"} err="failed to get container status \"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc\": rpc error: code = NotFound desc = could not find container \"157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc\": container with ID starting with 157ad4ddbc78b22f3f46a493fe454cd7361d06de701ed7d3bc6caab6d0ebaadc not found: ID does not exist"
Jul 17 16:29:47.550847 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.550825    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:29:47.556373 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:47.556347    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66644b6b6f-wbrrn"]
Jul 17 16:29:48.513568 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:29:48.513525    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d939392-fd63-47ca-874d-65f3078c315b" path="/var/lib/kubelet/pods/9d939392-fd63-47ca-874d-65f3078c315b/volumes"
Jul 17 16:30:13.788567 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:13.788528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:30:13.803506 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:13.803483    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:30:14.644671 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:14.644644    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 16:30:14.931429 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:14.931347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:15.025875 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:15.025841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:41.708117 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:41.708026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:42.598879 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:42.598841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:45.326743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:45.326647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:45.436778 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:45.436739    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:46.769165 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:46.769134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:46.870598 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:46.870567    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:46.982478 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:46.982444    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:48.751850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:48.751819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:49.376391 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:49.376360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:52.115388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:52.115349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:52.181222 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:52.181195    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:57.429992 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:57.429950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:58.891450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:58.891415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:30:59.767756 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:30:59.767709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:05.135280 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:05.135240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:05.239845 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:05.239812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:06.348107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:06.348073    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:31:06.350928 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:06.350905    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:31:06.351840 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:06.351817    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 16:31:06.997584 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:06.997526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:23.931951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:23.931914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:24.044503 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.044458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:24.538686 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.538633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:24.737991 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.737945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:24.849882 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.849789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:24.894852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.894816    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"]
Jul 17 16:31:24.895198 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.895183    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d939392-fd63-47ca-874d-65f3078c315b" containerName="console"
Jul 17 16:31:24.895286 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.895201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d939392-fd63-47ca-874d-65f3078c315b" containerName="console"
Jul 17 16:31:24.895345 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.895299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d939392-fd63-47ca-874d-65f3078c315b" containerName="console"
Jul 17 16:31:24.898393 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.898363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:24.902872 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.902836    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4r79r\""
Jul 17 16:31:24.903014 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.902995    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 16:31:24.904076 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.904057    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 16:31:24.904176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.904103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 16:31:24.915165 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.915141    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 16:31:24.935534 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.935509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"]
Jul 17 16:31:24.959878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.959843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-tmp\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:24.960027 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.959909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:24.960027 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.959952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhdq\" (UniqueName: \"kubernetes.io/projected/9d57b6e2-07ea-4247-86e1-76ec712135cf-kube-api-access-xvhdq\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:24.960027 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.959974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-manifests\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:24.960027 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:24.960002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.060982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.060934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.061207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.060998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhdq\" (UniqueName: \"kubernetes.io/projected/9d57b6e2-07ea-4247-86e1-76ec712135cf-kube-api-access-xvhdq\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.061207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.061019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-manifests\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.061207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.061039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.061207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.061097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-tmp\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.061464 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.061446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-manifests\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.063431 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.063406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d57b6e2-07ea-4247-86e1-76ec712135cf-tmp\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.063575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.063556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.063618 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.063604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d57b6e2-07ea-4247-86e1-76ec712135cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.071161 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.071129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhdq\" (UniqueName: \"kubernetes.io/projected/9d57b6e2-07ea-4247-86e1-76ec712135cf-kube-api-access-xvhdq\") pod \"opendatahub-operator-controller-manager-5cc76599b9-bn7q6\" (UID: \"9d57b6e2-07ea-4247-86e1-76ec712135cf\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.094012 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.093985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.194949 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.194867    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.209346 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.209300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:25.299348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.299308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.372252 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.372228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"]
Jul 17 16:31:25.375123 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:31:25.375091    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d57b6e2_07ea_4247_86e1_76ec712135cf.slice/crio-47fbac3bbe52a61aa62b4f86ec7d73fb84bddf2049a3eb9090939b54cb36085c WatchSource:0}: Error finding container 47fbac3bbe52a61aa62b4f86ec7d73fb84bddf2049a3eb9090939b54cb36085c: Status 404 returned error can't find the container with id 47fbac3bbe52a61aa62b4f86ec7d73fb84bddf2049a3eb9090939b54cb36085c
Jul 17 16:31:25.376926 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.376905    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 16:31:25.421477 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.421441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.550553 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.550517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.651555 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.651522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:25.894580 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:25.894508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6" event={"ID":"9d57b6e2-07ea-4247-86e1-76ec712135cf","Type":"ContainerStarted","Data":"47fbac3bbe52a61aa62b4f86ec7d73fb84bddf2049a3eb9090939b54cb36085c"}
Jul 17 16:31:28.910415 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:28.910377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6" event={"ID":"9d57b6e2-07ea-4247-86e1-76ec712135cf","Type":"ContainerStarted","Data":"43d579fca7a8a56e72d0f26b1779d14135b8d65d335772680fa7757e7978343c"}
Jul 17 16:31:29.916636 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:29.916605    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d57b6e2-07ea-4247-86e1-76ec712135cf" containerID="43d579fca7a8a56e72d0f26b1779d14135b8d65d335772680fa7757e7978343c" exitCode=0
Jul 17 16:31:29.917071 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:29.916681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6" event={"ID":"9d57b6e2-07ea-4247-86e1-76ec712135cf","Type":"ContainerDied","Data":"43d579fca7a8a56e72d0f26b1779d14135b8d65d335772680fa7757e7978343c"}
Jul 17 16:31:30.922176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:30.922138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6" event={"ID":"9d57b6e2-07ea-4247-86e1-76ec712135cf","Type":"ContainerStarted","Data":"f5fb4ca8041b16043d139beeda8790530010a998c5d0c333d3fc718766ea8f20"}
Jul 17 16:31:30.922661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:30.922273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:30.943432 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:30.943385    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6" podStartSLOduration=2.5478983079999997 podStartE2EDuration="6.943370272s" totalImagesPullingTime="3.308993996s" totalInitContainerRuntime="1.086477968s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:31:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:31:30.941885633 +0000 UTC m=+324.926973216" watchObservedRunningTime="2026-07-17 16:31:30.943370272 +0000 UTC m=+324.928457809"
Jul 17 16:31:35.588878 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:35.588838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:35.701816 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:35.701779    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:36.763055 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:36.762990    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:36.902322 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:36.902296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:41.928458 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:41.928427    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cc76599b9-bn7q6"
Jul 17 16:31:43.765053 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:43.765017    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:43.876900 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:43.876870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:44.523192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:44.523159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:44.617448 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:44.617410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:44.895065 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:44.894962    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:44.972931 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:44.972898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:45.429610 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:45.429576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:52.345348 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.345314    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"]
Jul 17 16:31:52.358272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.358245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"]
Jul 17 16:31:52.358427 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.358360    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.361019 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.360998    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 16:31:52.361982 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.361962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-spj29\""
Jul 17 16:31:52.362126 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.361988    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 16:31:52.362126 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.362091    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 16:31:52.362249 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.362135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 16:31:52.362249 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.362181    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 16:31:52.495374 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.495339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23a5159d-08c1-4d41-aa5f-1a49207866cb-manager-config\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.495662 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.495641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95f8\" (UniqueName: \"kubernetes.io/projected/23a5159d-08c1-4d41-aa5f-1a49207866cb-kube-api-access-v95f8\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.495835 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.495802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-metrics-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.495963 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.495944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.597431 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.597347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.597594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.597424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23a5159d-08c1-4d41-aa5f-1a49207866cb-manager-config\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.597594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.597472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95f8\" (UniqueName: \"kubernetes.io/projected/23a5159d-08c1-4d41-aa5f-1a49207866cb-kube-api-access-v95f8\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.597594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.597497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-metrics-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.598175 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.598147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23a5159d-08c1-4d41-aa5f-1a49207866cb-manager-config\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.599981 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.599949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.600128 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.600083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23a5159d-08c1-4d41-aa5f-1a49207866cb-metrics-cert\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.615651 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.615626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95f8\" (UniqueName: \"kubernetes.io/projected/23a5159d-08c1-4d41-aa5f-1a49207866cb-kube-api-access-v95f8\") pod \"lws-controller-manager-64bdb655c8-t8pzh\" (UID: \"23a5159d-08c1-4d41-aa5f-1a49207866cb\") " pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.668886 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.668862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:52.794420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:52.794393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"]
Jul 17 16:31:52.797078 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:31:52.797030    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a5159d_08c1_4d41_aa5f_1a49207866cb.slice/crio-c57eb184ee52c3a788635bf4e5fc8c0d030349a1c7aa941dd3d810f52a809120 WatchSource:0}: Error finding container c57eb184ee52c3a788635bf4e5fc8c0d030349a1c7aa941dd3d810f52a809120: Status 404 returned error can't find the container with id c57eb184ee52c3a788635bf4e5fc8c0d030349a1c7aa941dd3d810f52a809120
Jul 17 16:31:53.012745 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:53.012676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh" event={"ID":"23a5159d-08c1-4d41-aa5f-1a49207866cb","Type":"ContainerStarted","Data":"c57eb184ee52c3a788635bf4e5fc8c0d030349a1c7aa941dd3d810f52a809120"}
Jul 17 16:31:54.120921 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.120883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:54.245121 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.245085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:54.684006 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.683973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:54.734178 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.734139    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"]
Jul 17 16:31:54.743382 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.743356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.746850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.746716    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 16:31:54.746850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.746729    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4mmj6\""
Jul 17 16:31:54.746850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.746738    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 16:31:54.747116 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.746975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 16:31:54.747300 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.747275    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 16:31:54.752900 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.752838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"]
Jul 17 16:31:54.780585 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.780516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:54.820742 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.820713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.820903 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.820753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tmp\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.820903 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.820843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6zf\" (UniqueName: \"kubernetes.io/projected/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-kube-api-access-pt6zf\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.921393 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.921359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tmp\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.921610 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.921461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6zf\" (UniqueName: \"kubernetes.io/projected/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-kube-api-access-pt6zf\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.921610 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.921533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.921750 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:31:54.921648    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 17 16:31:54.921750 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:31:54.921720    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs podName:7def5c34-5bbc-4f0f-970d-6e503f4cfceb nodeName:}" failed. No retries permitted until 2026-07-17 16:31:55.421698766 +0000 UTC m=+349.406786305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs") pod "kube-auth-proxy-55b8db87d7-d8vcb" (UID: "7def5c34-5bbc-4f0f-970d-6e503f4cfceb") : secret "kube-auth-proxy-tls" not found
Jul 17 16:31:54.924027 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.923979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tmp\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.931699 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.931634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6zf\" (UniqueName: \"kubernetes.io/projected/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-kube-api-access-pt6zf\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:54.932570 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:54.932524    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:55.425268 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.425231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:55.428318 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.428283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7def5c34-5bbc-4f0f-970d-6e503f4cfceb-tls-certs\") pod \"kube-auth-proxy-55b8db87d7-d8vcb\" (UID: \"7def5c34-5bbc-4f0f-970d-6e503f4cfceb\") " pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:55.453008 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.452964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:55.614342 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.614314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:55.658755 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.658736    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"
Jul 17 16:31:55.707134 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.707080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:55.799963 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:55.799884    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb"]
Jul 17 16:31:55.802000 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:31:55.801962    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7def5c34_5bbc_4f0f_970d_6e503f4cfceb.slice/crio-f1b07369b3a5e08161f5fb7d871af3e843b07f93a7935c805ae4ffe85f26b170 WatchSource:0}: Error finding container f1b07369b3a5e08161f5fb7d871af3e843b07f93a7935c805ae4ffe85f26b170: Status 404 returned error can't find the container with id f1b07369b3a5e08161f5fb7d871af3e843b07f93a7935c805ae4ffe85f26b170
Jul 17 16:31:56.030405 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:56.030375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb" event={"ID":"7def5c34-5bbc-4f0f-970d-6e503f4cfceb","Type":"ContainerStarted","Data":"f1b07369b3a5e08161f5fb7d871af3e843b07f93a7935c805ae4ffe85f26b170"}
Jul 17 16:31:56.031779 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:56.031747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh" event={"ID":"23a5159d-08c1-4d41-aa5f-1a49207866cb","Type":"ContainerStarted","Data":"ac577a57c8dfd3cb9eae09f40e8a9539a995fe447e6b5b042739d7eacbca7c28"}
Jul 17 16:31:56.031882 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:56.031871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:31:57.130866 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:57.130833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:57.271967 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:57.271929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:59.024450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:59.024418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:31:59.047334 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:59.047304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb" event={"ID":"7def5c34-5bbc-4f0f-970d-6e503f4cfceb","Type":"ContainerStarted","Data":"51c9ab107d2519a93f95a96836ca3913faad0d8d2c7f4362fb6e018a5a3ea9dd"}
Jul 17 16:31:59.074459 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:59.074417    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh" podStartSLOduration=4.239614692 podStartE2EDuration="7.074403367s" totalImagesPullingTime="2.834788675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:31:56.065660955 +0000 UTC m=+350.050748493" watchObservedRunningTime="2026-07-17 16:31:59.074403367 +0000 UTC m=+353.059490904"
Jul 17 16:31:59.074615 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:31:59.074594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55b8db87d7-d8vcb" podStartSLOduration=2.336616054 podStartE2EDuration="5.074587997s" totalImagesPullingTime="2.737971943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:31:59.074294493 +0000 UTC m=+353.059382031" watchObservedRunningTime="2026-07-17 16:31:59.074587997 +0000 UTC m=+353.059675534"
Jul 17 16:32:02.921183 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:02.921151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:03.020433 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:03.020399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:03.650318 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:03.650286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:03.750574 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:03.750540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:03.853829 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:03.853791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:07.039341 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:07.039313    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64bdb655c8-t8pzh"
Jul 17 16:32:44.653562 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:44.653525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:44.711460 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:44.711430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:45.317702 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:45.317535    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:45.413384 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:45.413354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:45.521361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:45.521328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:46.236249 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:46.236220    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:32:46.445987 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:32:46.445959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:03.138756 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.138720    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dff787bd-8lqsz"]
Jul 17 16:33:03.142092 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.142073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.150721 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.150696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.150850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.150822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-service-ca\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.150919 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.150899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-trusted-ca-bundle\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.150980 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.150967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.151038 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.150996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwzz\" (UniqueName: \"kubernetes.io/projected/7d572b0b-9c68-482e-ab4b-fba912fc463d-kube-api-access-mgwzz\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.151198 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.151171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-oauth-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.151303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.151247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-oauth-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.152336 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.152312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dff787bd-8lqsz"]
Jul 17 16:33:03.194385 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.194355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:03.251631 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-oauth-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-oauth-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-service-ca\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-trusted-ca-bundle\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.251805 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.251777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwzz\" (UniqueName: \"kubernetes.io/projected/7d572b0b-9c68-482e-ab4b-fba912fc463d-kube-api-access-mgwzz\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.252460 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.252439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-service-ca\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.252568 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.252504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-oauth-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.252568 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.252518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.252735 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.252700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d572b0b-9c68-482e-ab4b-fba912fc463d-trusted-ca-bundle\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.254165 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.254144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-oauth-config\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.254381 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.254361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d572b0b-9c68-482e-ab4b-fba912fc463d-console-serving-cert\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.259728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.259701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwzz\" (UniqueName: \"kubernetes.io/projected/7d572b0b-9c68-482e-ab4b-fba912fc463d-kube-api-access-mgwzz\") pod \"console-74dff787bd-8lqsz\" (UID: \"7d572b0b-9c68-482e-ab4b-fba912fc463d\") " pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.451479 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.451395    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:03.579320 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:03.579297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dff787bd-8lqsz"]
Jul 17 16:33:03.581212 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:33:03.581187    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d572b0b_9c68_482e_ab4b_fba912fc463d.slice/crio-aa9c36951f83734225e8eafe8a68ab7f67e703f183bbfcd5ebf71dfc77729101 WatchSource:0}: Error finding container aa9c36951f83734225e8eafe8a68ab7f67e703f183bbfcd5ebf71dfc77729101: Status 404 returned error can't find the container with id aa9c36951f83734225e8eafe8a68ab7f67e703f183bbfcd5ebf71dfc77729101
Jul 17 16:33:04.314363 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:04.314333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dff787bd-8lqsz" event={"ID":"7d572b0b-9c68-482e-ab4b-fba912fc463d","Type":"ContainerStarted","Data":"184a9d93aaf8a77e020645e7cad68e0f4ec417135b14bfa046c9fb5de2995ee3"}
Jul 17 16:33:04.314724 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:04.314376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dff787bd-8lqsz" event={"ID":"7d572b0b-9c68-482e-ab4b-fba912fc463d","Type":"ContainerStarted","Data":"aa9c36951f83734225e8eafe8a68ab7f67e703f183bbfcd5ebf71dfc77729101"}
Jul 17 16:33:04.333882 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:04.333834    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dff787bd-8lqsz" podStartSLOduration=1.333818483 podStartE2EDuration="1.333818483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:33:04.332445411 +0000 UTC m=+418.317532967" watchObservedRunningTime="2026-07-17 16:33:04.333818483 +0000 UTC m=+418.318906020"
Jul 17 16:33:13.451739 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:13.451703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:13.452162 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:13.451762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:13.460101 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:13.457195    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:14.361647 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:14.361616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dff787bd-8lqsz"
Jul 17 16:33:14.407965 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:14.407937    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:33:14.609287 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:14.609251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:14.664836 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:14.664766    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:16.305782 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:16.305747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:16.380543 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:16.380506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:18.166253 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:18.166215    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:18.274648 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:18.274621    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:31.701235 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:31.701149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:35.826899 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:35.826856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:38.190224 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:38.190187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:38.425913 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:38.425874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:38.804961 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:38.804920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:39.430786 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.430741    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cfb5c7f97-x4jgm" podUID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" containerName="console" containerID="cri-o://badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0" gracePeriod=15
Jul 17 16:33:39.687607 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.687558    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:33:39.761199 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761175    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srbps\" (UniqueName: \"kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761322    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761416 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761392    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config\") pod \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\" (UID: \"1d0388fc-ef73-47ad-9ff0-961f8c32e4af\") "
Jul 17 16:33:39.761740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761716    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:33:39.761740 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:33:39.761850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config" (OuterVolumeSpecName: "console-config") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:33:39.761850 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.761785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:33:39.763340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.763318    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:33:39.763575 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.763554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:33:39.763635 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.763581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps" (OuterVolumeSpecName: "kube-api-access-srbps") pod "1d0388fc-ef73-47ad-9ff0-961f8c32e4af" (UID: "1d0388fc-ef73-47ad-9ff0-961f8c32e4af"). InnerVolumeSpecName "kube-api-access-srbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:33:39.862005 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.861982    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-oauth-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862005 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862000    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbps\" (UniqueName: \"kubernetes.io/projected/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-kube-api-access-srbps\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862010    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862019    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-trusted-ca-bundle\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862027    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-oauth-config\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862035    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-console-serving-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:39.862156 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:39.862064    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0388fc-ef73-47ad-9ff0-961f8c32e4af-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:33:40.466285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.466248    2561 generic.go:356] "Generic (PLEG): container finished" podID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" containerID="badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0" exitCode=0
Jul 17 16:33:40.466661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.466305    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfb5c7f97-x4jgm"
Jul 17 16:33:40.466661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.466333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb5c7f97-x4jgm" event={"ID":"1d0388fc-ef73-47ad-9ff0-961f8c32e4af","Type":"ContainerDied","Data":"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"}
Jul 17 16:33:40.466661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.466412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfb5c7f97-x4jgm" event={"ID":"1d0388fc-ef73-47ad-9ff0-961f8c32e4af","Type":"ContainerDied","Data":"a6c5c2b2a9bc07c0f24312a3b7a8978691c769f3dfea2d4f581cd5ae448c0f66"}
Jul 17 16:33:40.466661 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.466440    2561 scope.go:117] "RemoveContainer" containerID="badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"
Jul 17 16:33:40.475134 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.475114    2561 scope.go:117] "RemoveContainer" containerID="badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"
Jul 17 16:33:40.475445 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:33:40.475418    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0\": container with ID starting with badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0 not found: ID does not exist" containerID="badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"
Jul 17 16:33:40.475542 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.475450    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0"} err="failed to get container status \"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0\": rpc error: code = NotFound desc = could not find container \"badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0\": container with ID starting with badb4cae3c1a7e907fff95d90f53781233c606908a25d7d502c25b69854989c0 not found: ID does not exist"
Jul 17 16:33:40.487494 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.487471    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:33:40.491331 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.491311    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cfb5c7f97-x4jgm"]
Jul 17 16:33:40.512777 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:40.512753    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" path="/var/lib/kubelet/pods/1d0388fc-ef73-47ad-9ff0-961f8c32e4af/volumes"
Jul 17 16:33:42.189514 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:42.189479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:43.258227 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:43.258193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:45.248091 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:45.248029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:47.055492 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:47.055462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:48.415155 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:48.415120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:48.539525 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:48.539493    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:50.389022 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:50.388981    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:50.533479 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:50.533451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:52.201831 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:52.201805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:52.300983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:52.300949    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:56.352939 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.352893    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"]
Jul 17 16:33:56.353407 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.353229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" containerName="console"
Jul 17 16:33:56.353407 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.353240    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" containerName="console"
Jul 17 16:33:56.353407 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.353289    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0388fc-ef73-47ad-9ff0-961f8c32e4af" containerName="console"
Jul 17 16:33:56.357589 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.357565    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.358531 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.358498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:56.359983 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.359961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 17 16:33:56.360298 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.360276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 16:33:56.360443 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.360426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6kglj\""
Jul 17 16:33:56.360549 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.360534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 17 16:33:56.360791 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.360775    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 16:33:56.364710 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.364685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"]
Jul 17 16:33:56.455714 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.455684    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:56.485809 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.485784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/282af9ea-3859-4b50-be9b-2f2e0ad62957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.485924 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.485819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx87v\" (UniqueName: \"kubernetes.io/projected/282af9ea-3859-4b50-be9b-2f2e0ad62957-kube-api-access-bx87v\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.485924 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.485846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/282af9ea-3859-4b50-be9b-2f2e0ad62957-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.586628 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.586603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/282af9ea-3859-4b50-be9b-2f2e0ad62957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.586743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.586639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx87v\" (UniqueName: \"kubernetes.io/projected/282af9ea-3859-4b50-be9b-2f2e0ad62957-kube-api-access-bx87v\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.586743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.586661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/282af9ea-3859-4b50-be9b-2f2e0ad62957-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.587328 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.587311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/282af9ea-3859-4b50-be9b-2f2e0ad62957-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.589067 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.589035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/282af9ea-3859-4b50-be9b-2f2e0ad62957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.598035 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.598016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx87v\" (UniqueName: \"kubernetes.io/projected/282af9ea-3859-4b50-be9b-2f2e0ad62957-kube-api-access-bx87v\") pod \"kuadrant-console-plugin-66b5774985-9kljd\" (UID: \"282af9ea-3859-4b50-be9b-2f2e0ad62957\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.685221 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.685172    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"
Jul 17 16:33:56.823624 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.823591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd"]
Jul 17 16:33:56.827495 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:33:56.827470    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod282af9ea_3859_4b50_be9b_2f2e0ad62957.slice/crio-5f8cf4da79d9c14dd458fe2de67d60ba74dad0b34862bfa255f674c4fd7fe0ae WatchSource:0}: Error finding container 5f8cf4da79d9c14dd458fe2de67d60ba74dad0b34862bfa255f674c4fd7fe0ae: Status 404 returned error can't find the container with id 5f8cf4da79d9c14dd458fe2de67d60ba74dad0b34862bfa255f674c4fd7fe0ae
Jul 17 16:33:56.846174 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.846150    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:56.986066 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:56.985999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:33:57.537947 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:33:57.537914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd" event={"ID":"282af9ea-3859-4b50-be9b-2f2e0ad62957","Type":"ContainerStarted","Data":"5f8cf4da79d9c14dd458fe2de67d60ba74dad0b34862bfa255f674c4fd7fe0ae"}
Jul 17 16:34:00.639176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:00.639138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:00.739918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:00.739885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:06.312699 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.312658    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"]
Jul 17 16:34:06.322145 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.322116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.327076 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.327029    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-st7cb\""
Jul 17 16:34:06.334407 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.334385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"]
Jul 17 16:34:06.373398 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.373357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.373551 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.373486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrxm\" (UniqueName: \"kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.474387 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.474347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrxm\" (UniqueName: \"kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.474592 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.474434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.474880 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.474854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.483237 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.483205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrxm\" (UniqueName: \"kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.634936 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.634848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:06.942743 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.942656    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"]
Jul 17 16:34:06.951361 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.951332    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"]
Jul 17 16:34:06.964795 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.964759    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:06.973601 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.973576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:06.978654 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.978621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlr9\" (UniqueName: \"kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:06.978810 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.978712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:06.984144 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:06.984077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:07.079550 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.079519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlr9\" (UniqueName: \"kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:07.079728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.079589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:07.080007 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.079988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:07.094123 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.094081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlr9\" (UniqueName: \"kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-v9d9t\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:07.221446 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.221340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:07.287004 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.286969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:07.324545 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:07.324504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:10.038498 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:10.032849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:10.135377 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:10.135336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:14.841146 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:14.841108    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:14.952902 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:14.952864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:18.400721 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:18.400679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:18.456003 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:18.455965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:19.450741 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:19.450714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:19.454403 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:34:19.454381    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d90e4b_c322_4ce2_96df_4274c477db25.slice/crio-8b58ec3ebb71b7cfb428e425130ffbfdb358f913fe83a942bf2d0c08b57b2bb8 WatchSource:0}: Error finding container 8b58ec3ebb71b7cfb428e425130ffbfdb358f913fe83a942bf2d0c08b57b2bb8: Status 404 returned error can't find the container with id 8b58ec3ebb71b7cfb428e425130ffbfdb358f913fe83a942bf2d0c08b57b2bb8
Jul 17 16:34:19.648801 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:19.648722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" event={"ID":"05d90e4b-c322-4ce2-96df-4274c477db25","Type":"ContainerStarted","Data":"8b58ec3ebb71b7cfb428e425130ffbfdb358f913fe83a942bf2d0c08b57b2bb8"}
Jul 17 16:34:19.650275 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:19.650250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd" event={"ID":"282af9ea-3859-4b50-be9b-2f2e0ad62957","Type":"ContainerStarted","Data":"8ac45794db2a01f6abe3d087e967212d4c76518c1936a55e2059471e9e29162d"}
Jul 17 16:34:21.856291 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:21.856234    2561 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-88xhh_kuadrant-system_e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8_0(6c793b59758d2a4cd44298062c0253eddee2da64097b651243aa6a30f2a949b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-88xhh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 17 16:34:21.856673 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:21.856325    2561 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-88xhh_kuadrant-system_e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8_0(6c793b59758d2a4cd44298062c0253eddee2da64097b651243aa6a30f2a949b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-88xhh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:22.665487 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.665449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:22.667431 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:22.667393    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:22.673916 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.673895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:22.675735 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:22.675699    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:22.728188 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.728158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume\") pod \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") "
Jul 17 16:34:22.728337 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.728212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfrxm\" (UniqueName: \"kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm\") pod \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\" (UID: \"e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8\") "
Jul 17 16:34:22.728431 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.728407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" (UID: "e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:34:22.730247 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.730214    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm" (OuterVolumeSpecName: "kube-api-access-hfrxm") pod "e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" (UID: "e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8"). InnerVolumeSpecName "kube-api-access-hfrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:34:22.829942 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.829903    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:34:22.829942 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:22.829939    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrxm\" (UniqueName: \"kubernetes.io/projected/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8-kube-api-access-hfrxm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:34:23.670312 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:23.670272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:23.672449 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:23.672412    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:23.680633 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:23.680603    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-88xhh\" is forbidden: User \"system:node:ip-10-0-137-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-172.ec2.internal' and this object" podUID="e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-88xhh"
Jul 17 16:34:24.513120 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:24.513088    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8" path="/var/lib/kubelet/pods/e3d9238c-b08a-4fc8-bb8b-d2597c2cdfd8/volumes"
Jul 17 16:34:24.675816 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:24.675784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" event={"ID":"05d90e4b-c322-4ce2-96df-4274c477db25","Type":"ContainerStarted","Data":"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"}
Jul 17 16:34:24.676207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:24.675867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:24.695594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:24.695550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9kljd" podStartSLOduration=6.119132076 podStartE2EDuration="28.695535362s" totalImagesPullingTime="22.576403286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:34:19.692265541 +0000 UTC m=+493.677353079" watchObservedRunningTime="2026-07-17 16:34:24.695535362 +0000 UTC m=+498.680622908"
Jul 17 16:34:24.695728 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:24.695637    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" podStartSLOduration=13.596793657 podStartE2EDuration="18.695627621s" totalImagesPullingTime="5.098833964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:34:24.694171142 +0000 UTC m=+498.679258681" watchObservedRunningTime="2026-07-17 16:34:24.695627621 +0000 UTC m=+498.680715160"
Jul 17 16:34:25.032542 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:25.032506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:25.126662 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:25.126632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:27.659318 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:27.659275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:28.211853 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:28.211826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:28.312502 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:28.312478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:32.821350 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:32.821313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:32.920890 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:32.920858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:33.022132 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:33.022093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:33.122038 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:33.121951    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:35.681596 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:35.681561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:38.228494 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:38.228455    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:38.327457 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:38.327421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:48.526964 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.526929    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:48.527480 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.527136    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" podUID="05d90e4b-c322-4ce2-96df-4274c477db25" containerName="manager" containerID="cri-o://d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841" gracePeriod=10
Jul 17 16:34:48.774084 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.774058    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:48.784218 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.784151    2561 generic.go:356] "Generic (PLEG): container finished" podID="05d90e4b-c322-4ce2-96df-4274c477db25" containerID="d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841" exitCode=0
Jul 17 16:34:48.784218 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.784202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"
Jul 17 16:34:48.784396 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.784223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" event={"ID":"05d90e4b-c322-4ce2-96df-4274c477db25","Type":"ContainerDied","Data":"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"}
Jul 17 16:34:48.784396 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.784263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t" event={"ID":"05d90e4b-c322-4ce2-96df-4274c477db25","Type":"ContainerDied","Data":"8b58ec3ebb71b7cfb428e425130ffbfdb358f913fe83a942bf2d0c08b57b2bb8"}
Jul 17 16:34:48.784396 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.784284    2561 scope.go:117] "RemoveContainer" containerID="d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"
Jul 17 16:34:48.793012 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.792995    2561 scope.go:117] "RemoveContainer" containerID="d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"
Jul 17 16:34:48.793319 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:34:48.793289    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841\": container with ID starting with d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841 not found: ID does not exist" containerID="d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"
Jul 17 16:34:48.793410 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.793328    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841"} err="failed to get container status \"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841\": rpc error: code = NotFound desc = could not find container \"d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841\": container with ID starting with d5351f5414bedb2b44468464ff9416480152632c64d61029bfff6641aac0a841 not found: ID does not exist"
Jul 17 16:34:48.849784 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.849755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume\") pod \"05d90e4b-c322-4ce2-96df-4274c477db25\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") "
Jul 17 16:34:48.849944 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.849846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xlr9\" (UniqueName: \"kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9\") pod \"05d90e4b-c322-4ce2-96df-4274c477db25\" (UID: \"05d90e4b-c322-4ce2-96df-4274c477db25\") "
Jul 17 16:34:48.850249 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.850227    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "05d90e4b-c322-4ce2-96df-4274c477db25" (UID: "05d90e4b-c322-4ce2-96df-4274c477db25"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:34:48.851858 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.851838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9" (OuterVolumeSpecName: "kube-api-access-5xlr9") pod "05d90e4b-c322-4ce2-96df-4274c477db25" (UID: "05d90e4b-c322-4ce2-96df-4274c477db25"). InnerVolumeSpecName "kube-api-access-5xlr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:34:48.950690 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.950650    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05d90e4b-c322-4ce2-96df-4274c477db25-extensions-socket-volume\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:34:48.950690 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:48.950685    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlr9\" (UniqueName: \"kubernetes.io/projected/05d90e4b-c322-4ce2-96df-4274c477db25-kube-api-access-5xlr9\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:34:49.111504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:49.111477    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:49.115993 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:49.115968    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-v9d9t"]
Jul 17 16:34:50.513524 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:50.513492    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05d90e4b-c322-4ce2-96df-4274c477db25" path="/var/lib/kubelet/pods/05d90e4b-c322-4ce2-96df-4274c477db25/volumes"
Jul 17 16:34:52.100092 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:52.100057    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:52.271629 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:52.271596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:54.603538 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:54.603493    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:54.707463 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:54.707419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:54.777836 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:54.777807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:55.083874 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.083843    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:34:55.084207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.084194    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d90e4b-c322-4ce2-96df-4274c477db25" containerName="manager"
Jul 17 16:34:55.084269 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.084209    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d90e4b-c322-4ce2-96df-4274c477db25" containerName="manager"
Jul 17 16:34:55.084309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.084279    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d90e4b-c322-4ce2-96df-4274c477db25" containerName="manager"
Jul 17 16:34:55.142987 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.142958    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:34:55.142987 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.142993    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:34:55.143189 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.143092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.146757 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.146734    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 16:34:55.201705 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.201671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3fcda4e9-23b3-4e3d-87e3-668211edcb11-config-file\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.201852 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.201718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbtr\" (UniqueName: \"kubernetes.io/projected/3fcda4e9-23b3-4e3d-87e3-668211edcb11-kube-api-access-kvbtr\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.303120 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.303095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbtr\" (UniqueName: \"kubernetes.io/projected/3fcda4e9-23b3-4e3d-87e3-668211edcb11-kube-api-access-kvbtr\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.303281 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.303257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3fcda4e9-23b3-4e3d-87e3-668211edcb11-config-file\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.303868 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.303850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3fcda4e9-23b3-4e3d-87e3-668211edcb11-config-file\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.311917 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.311900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbtr\" (UniqueName: \"kubernetes.io/projected/3fcda4e9-23b3-4e3d-87e3-668211edcb11-kube-api-access-kvbtr\") pod \"limitador-limitador-54d6d988d5-wx57w\" (UID: \"3fcda4e9-23b3-4e3d-87e3-668211edcb11\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.452897 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.452816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:34:55.471922 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.471891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:34:55.475674 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.475648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:34:55.478207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.478183    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5ldvq\""
Jul 17 16:34:55.486165 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.486137    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:34:55.502867 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.502828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:55.505959 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.505927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlk2f\" (UniqueName: \"kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f\") pod \"authorino-58cd955675-7ll2c\" (UID: \"f43eeebd-1d4b-4c55-99f2-a3a440a4d218\") " pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:34:55.566918 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.566877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:34:55.590334 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.590270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:34:55.592896 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:34:55.592870    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcda4e9_23b3_4e3d_87e3_668211edcb11.slice/crio-3495f7b9f16359ef72ad37abb42d5b9ae9166d2b5fcc539bd50eee9cafafe7ea WatchSource:0}: Error finding container 3495f7b9f16359ef72ad37abb42d5b9ae9166d2b5fcc539bd50eee9cafafe7ea: Status 404 returned error can't find the container with id 3495f7b9f16359ef72ad37abb42d5b9ae9166d2b5fcc539bd50eee9cafafe7ea
Jul 17 16:34:55.607192 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.607165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlk2f\" (UniqueName: \"kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f\") pod \"authorino-58cd955675-7ll2c\" (UID: \"f43eeebd-1d4b-4c55-99f2-a3a440a4d218\") " pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:34:55.616202 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.616180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlk2f\" (UniqueName: \"kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f\") pod \"authorino-58cd955675-7ll2c\" (UID: \"f43eeebd-1d4b-4c55-99f2-a3a440a4d218\") " pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:34:55.785984 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.785957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:34:55.820376 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.820343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w" event={"ID":"3fcda4e9-23b3-4e3d-87e3-668211edcb11","Type":"ContainerStarted","Data":"3495f7b9f16359ef72ad37abb42d5b9ae9166d2b5fcc539bd50eee9cafafe7ea"}
Jul 17 16:34:55.913626 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:55.913602    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:34:55.916079 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:34:55.916039    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43eeebd_1d4b_4c55_99f2_a3a440a4d218.slice/crio-7eeb3b7a9b128c590a6e678481336995d82cfd9a0f042c25a2054c9972af168b WatchSource:0}: Error finding container 7eeb3b7a9b128c590a6e678481336995d82cfd9a0f042c25a2054c9972af168b: Status 404 returned error can't find the container with id 7eeb3b7a9b128c590a6e678481336995d82cfd9a0f042c25a2054c9972af168b
Jul 17 16:34:56.828967 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:56.828913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7ll2c" event={"ID":"f43eeebd-1d4b-4c55-99f2-a3a440a4d218","Type":"ContainerStarted","Data":"7eeb3b7a9b128c590a6e678481336995d82cfd9a0f042c25a2054c9972af168b"}
Jul 17 16:34:59.626526 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:34:59.626448    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:35:00.848442 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.848408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7ll2c" event={"ID":"f43eeebd-1d4b-4c55-99f2-a3a440a4d218","Type":"ContainerStarted","Data":"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"}
Jul 17 16:35:00.848909 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.848455    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-7ll2c" podUID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" containerName="authorino" containerID="cri-o://23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325" gracePeriod=30
Jul 17 16:35:00.850015 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.849991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w" event={"ID":"3fcda4e9-23b3-4e3d-87e3-668211edcb11","Type":"ContainerStarted","Data":"b88e6780ff517808b7111735f531006ca84ba68947bc5b11c53d0ed41025fff3"}
Jul 17 16:35:00.850164 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.850128    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:35:00.874125 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.874080    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-7ll2c" podStartSLOduration=1.5367079380000002 podStartE2EDuration="5.874061931s" totalImagesPullingTime="4.337353993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:35:00.873356215 +0000 UTC m=+534.858443753" watchObservedRunningTime="2026-07-17 16:35:00.874061931 +0000 UTC m=+534.859149461"
Jul 17 16:35:00.906955 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:00.906903    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w" podStartSLOduration=1.248428936 podStartE2EDuration="5.906887413s" totalImagesPullingTime="4.658458477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:35:00.905741492 +0000 UTC m=+534.890829029" watchObservedRunningTime="2026-07-17 16:35:00.906887413 +0000 UTC m=+534.891974949"
Jul 17 16:35:01.109419 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.109358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:35:01.159639 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.159615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlk2f\" (UniqueName: \"kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f\") pod \"f43eeebd-1d4b-4c55-99f2-a3a440a4d218\" (UID: \"f43eeebd-1d4b-4c55-99f2-a3a440a4d218\") "
Jul 17 16:35:01.161552 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.161530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f" (OuterVolumeSpecName: "kube-api-access-qlk2f") pod "f43eeebd-1d4b-4c55-99f2-a3a440a4d218" (UID: "f43eeebd-1d4b-4c55-99f2-a3a440a4d218"). InnerVolumeSpecName "kube-api-access-qlk2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:35:01.260857 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.260820    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlk2f\" (UniqueName: \"kubernetes.io/projected/f43eeebd-1d4b-4c55-99f2-a3a440a4d218-kube-api-access-qlk2f\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:35:01.854911 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.854876    2561 generic.go:356] "Generic (PLEG): container finished" podID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" containerID="23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325" exitCode=0
Jul 17 16:35:01.854911 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.854913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7ll2c"
Jul 17 16:35:01.855450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.854950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7ll2c" event={"ID":"f43eeebd-1d4b-4c55-99f2-a3a440a4d218","Type":"ContainerDied","Data":"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"}
Jul 17 16:35:01.855450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.854987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7ll2c" event={"ID":"f43eeebd-1d4b-4c55-99f2-a3a440a4d218","Type":"ContainerDied","Data":"7eeb3b7a9b128c590a6e678481336995d82cfd9a0f042c25a2054c9972af168b"}
Jul 17 16:35:01.855450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.855011    2561 scope.go:117] "RemoveContainer" containerID="23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"
Jul 17 16:35:01.863039 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.863022    2561 scope.go:117] "RemoveContainer" containerID="23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"
Jul 17 16:35:01.863294 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:35:01.863274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325\": container with ID starting with 23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325 not found: ID does not exist" containerID="23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"
Jul 17 16:35:01.863359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.863304    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325"} err="failed to get container status \"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325\": rpc error: code = NotFound desc = could not find container \"23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325\": container with ID starting with 23d2eb7e6485ce1b39e2b75c7fe9c80e27b9815b0acf37e2a1f74a5499dff325 not found: ID does not exist"
Jul 17 16:35:01.878287 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.878263    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:35:01.881207 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:01.881182    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-7ll2c"]
Jul 17 16:35:02.513890 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:02.513851    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" path="/var/lib/kubelet/pods/f43eeebd-1d4b-4c55-99f2-a3a440a4d218/volumes"
Jul 17 16:35:08.727431 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:08.727394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:08.823861 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:08.823831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:11.856803 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:11.856770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wx57w"
Jul 17 16:35:15.770450 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:15.770417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:15.867702 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:15.867674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:16.207228 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:16.207198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:25.477387 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:25.477351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:27.595735 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:27.595703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:27.660409 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:27.660380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:28.255195 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.255160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:28.503340 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.503309    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dshn6"]
Jul 17 16:35:28.503679 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.503662    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" containerName="authorino"
Jul 17 16:35:28.503759 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.503681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" containerName="authorino"
Jul 17 16:35:28.503813 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.503772    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43eeebd-1d4b-4c55-99f2-a3a440a4d218" containerName="authorino"
Jul 17 16:35:28.507516 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.507465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:28.509599 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.509578    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5ldvq\""
Jul 17 16:35:28.514717 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.514609    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dshn6"]
Jul 17 16:35:28.679559 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.679530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw99\" (UniqueName: \"kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99\") pod \"authorino-5d4b67c446-dshn6\" (UID: \"c67dacbe-29df-4cb2-bbf6-febb576cad04\") " pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:28.726955 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.726929    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dshn6"]
Jul 17 16:35:28.727154 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:35:28.727136    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-stw99], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-dshn6" podUID="c67dacbe-29df-4cb2-bbf6-febb576cad04"
Jul 17 16:35:28.754742 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.754717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:28.779874 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.779846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw99\" (UniqueName: \"kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99\") pod \"authorino-5d4b67c446-dshn6\" (UID: \"c67dacbe-29df-4cb2-bbf6-febb576cad04\") " pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:28.787914 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.787895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw99\" (UniqueName: \"kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99\") pod \"authorino-5d4b67c446-dshn6\" (UID: \"c67dacbe-29df-4cb2-bbf6-febb576cad04\") " pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:28.964167 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.964141    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:35:28.967288 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.967273    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:28.969758 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.969740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 16:35:28.974562 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.974543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:35:28.979440 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.979422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:28.985379 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:28.985363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:29.082383 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.082327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stw99\" (UniqueName: \"kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99\") pod \"c67dacbe-29df-4cb2-bbf6-febb576cad04\" (UID: \"c67dacbe-29df-4cb2-bbf6-febb576cad04\") "
Jul 17 16:35:29.082484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.082469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.082535 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.082505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8l6t\" (UniqueName: \"kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.084289 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.084263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99" (OuterVolumeSpecName: "kube-api-access-stw99") pod "c67dacbe-29df-4cb2-bbf6-febb576cad04" (UID: "c67dacbe-29df-4cb2-bbf6-febb576cad04"). InnerVolumeSpecName "kube-api-access-stw99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:35:29.183532 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.183508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.183619 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.183550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8l6t\" (UniqueName: \"kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.183662 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.183630    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stw99\" (UniqueName: \"kubernetes.io/projected/c67dacbe-29df-4cb2-bbf6-febb576cad04-kube-api-access-stw99\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:35:29.185737 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.185720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.191729 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.191710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8l6t\" (UniqueName: \"kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t\") pod \"authorino-84958fc94f-vbbj4\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") " pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.275979 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.275959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:35:29.399597 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.399574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:35:29.402116 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:35:29.402086    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc455ff4_9e24_4612_880b_0545e61d48dc.slice/crio-8609ea9b219e0e526322368412fa122192e49d03c04426fff84b859991a28ab8 WatchSource:0}: Error finding container 8609ea9b219e0e526322368412fa122192e49d03c04426fff84b859991a28ab8: Status 404 returned error can't find the container with id 8609ea9b219e0e526322368412fa122192e49d03c04426fff84b859991a28ab8
Jul 17 16:35:29.989985 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.989956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-dshn6"
Jul 17 16:35:29.990389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.989995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84958fc94f-vbbj4" event={"ID":"dc455ff4-9e24-4612-880b-0545e61d48dc","Type":"ContainerStarted","Data":"8a1b4ffec543c1df9c55e5977190479326db500bdf99073a522a1e79fd4c0ecc"}
Jul 17 16:35:29.990389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:29.990066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84958fc94f-vbbj4" event={"ID":"dc455ff4-9e24-4612-880b-0545e61d48dc","Type":"ContainerStarted","Data":"8609ea9b219e0e526322368412fa122192e49d03c04426fff84b859991a28ab8"}
Jul 17 16:35:30.010482 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:30.010444    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84958fc94f-vbbj4" podStartSLOduration=1.6495551050000001 podStartE2EDuration="2.010429796s" totalImagesPullingTime="360.874691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:35:30.009813468 +0000 UTC m=+563.994901005" watchObservedRunningTime="2026-07-17 16:35:30.010429796 +0000 UTC m=+563.995517334"
Jul 17 16:35:30.034722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:30.034694    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dshn6"]
Jul 17 16:35:30.042439 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:30.042417    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-dshn6"]
Jul 17 16:35:30.514093 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:30.514011    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67dacbe-29df-4cb2-bbf6-febb576cad04" path="/var/lib/kubelet/pods/c67dacbe-29df-4cb2-bbf6-febb576cad04/volumes"
Jul 17 16:35:34.590458 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:34.590420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:34.687738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:34.687704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:34.773560 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:34.773531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:35.416738 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:35.416707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:38.028841 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:38.028802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:38.138223 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:38.138190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:53.474471 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:53.474427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:35:53.535005 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:35:53.534957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:06.375811 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:06.375781    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:36:06.376697 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:06.376678    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:36:09.657239 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:09.657203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:10.778577 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:10.778538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:36:11.049504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:11.049432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:11.110162 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:11.110118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:22.191185 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:22.191146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:22.287571 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:22.287539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:33.626986 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:33.626904    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:34.058737 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:34.058701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:36:40.443830 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:40.443791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:41.262210 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:41.262179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:36:51.334600 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:51.334563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:36:52.275769 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:36:52.275735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:37:11.228153 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:11.228112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:37:12.918356 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:12.918324    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:37:14.686907 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.686877    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"]
Jul 17 16:37:14.690750 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.690732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.693827 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.693809    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 16:37:14.693951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.693811    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jpw69\""
Jul 17 16:37:14.693951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.693811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 16:37:14.693951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.693869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 16:37:14.700446 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.700418    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"]
Jul 17 16:37:14.719951 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.719924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:37:14.778096 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm427\" (UniqueName: \"kubernetes.io/projected/a84c6f71-029c-40e5-a60d-f5b6bf978e73-kube-api-access-nm427\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.778195 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.778195 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.778195 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.778195 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.778337 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.778241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878668 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878787 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878787 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm427\" (UniqueName: \"kubernetes.io/projected/a84c6f71-029c-40e5-a60d-f5b6bf978e73-kube-api-access-nm427\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.878901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.878845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.879176 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.879155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.879255 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.879174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.879255 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.879227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.880824 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.880804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a84c6f71-029c-40e5-a60d-f5b6bf978e73-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.880939 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.880908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84c6f71-029c-40e5-a60d-f5b6bf978e73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:14.889167 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:14.889143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm427\" (UniqueName: \"kubernetes.io/projected/a84c6f71-029c-40e5-a60d-f5b6bf978e73-kube-api-access-nm427\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-q2svx\" (UID: \"a84c6f71-029c-40e5-a60d-f5b6bf978e73\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:15.009916 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:15.009861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:15.142629 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:15.142603    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"]
Jul 17 16:37:15.144257 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:37:15.144233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84c6f71_029c_40e5_a60d_f5b6bf978e73.slice/crio-fe6cd1ef70f2e82d48fd0d4bdd28c800e6b887b58806ed675ba04df590cf4e53 WatchSource:0}: Error finding container fe6cd1ef70f2e82d48fd0d4bdd28c800e6b887b58806ed675ba04df590cf4e53: Status 404 returned error can't find the container with id fe6cd1ef70f2e82d48fd0d4bdd28c800e6b887b58806ed675ba04df590cf4e53
Jul 17 16:37:15.145505 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:15.145490    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 16:37:15.475844 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:15.475772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx" event={"ID":"a84c6f71-029c-40e5-a60d-f5b6bf978e73","Type":"ContainerStarted","Data":"fe6cd1ef70f2e82d48fd0d4bdd28c800e6b887b58806ed675ba04df590cf4e53"}
Jul 17 16:37:16.482389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:16.482300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx" event={"ID":"a84c6f71-029c-40e5-a60d-f5b6bf978e73","Type":"ContainerStarted","Data":"7d3e74d433fd476dd3b4b443d69aeeadadb3b409fa67dfe991d66793746be1f3"}
Jul 17 16:37:16.503588 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:16.503543    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx" podStartSLOduration=1.490896494 podStartE2EDuration="2.503530383s" totalImagesPullingTime="1.012633889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:37:16.501056949 +0000 UTC m=+670.486144489" watchObservedRunningTime="2026-07-17 16:37:16.503530383 +0000 UTC m=+670.488617920"
Jul 17 16:37:22.665461 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:22.665425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:37:25.010381 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:25.010347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:25.010748 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:25.010398    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:25.022454 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:25.022426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:25.534583 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:25.534553    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-q2svx"
Jul 17 16:37:26.699303 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.699269    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"]
Jul 17 16:37:26.703032 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.703011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.705358 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.705338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 16:37:26.713119 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.713099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"]
Jul 17 16:37:26.739633 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.739591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:37:26.781708 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.781823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.781823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.781823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.781823 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.782033 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.781845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcqq\" (UniqueName: \"kubernetes.io/projected/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-kube-api-access-5kcqq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.882719 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcqq\" (UniqueName: \"kubernetes.io/projected/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-kube-api-access-5kcqq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.882853 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.882853 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.882853 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.882853 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.883026 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.882881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.883248 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.883219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.883351 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.883255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.883351 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.883310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.884830 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.884811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.885162 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.885145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:26.891713 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:26.891690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcqq\" (UniqueName: \"kubernetes.io/projected/ecc54fa1-e216-40b7-a715-1e14d3ebb39c-kube-api-access-5kcqq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl\" (UID: \"ecc54fa1-e216-40b7-a715-1e14d3ebb39c\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:27.011937 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:27.011883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:27.136864 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:27.136707    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"]
Jul 17 16:37:27.139497 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:37:27.139471    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc54fa1_e216_40b7_a715_1e14d3ebb39c.slice/crio-9da88e490189525399899ea3b4faeb59fd576063b065e95e1cf8fc2cd13a8a2d WatchSource:0}: Error finding container 9da88e490189525399899ea3b4faeb59fd576063b065e95e1cf8fc2cd13a8a2d: Status 404 returned error can't find the container with id 9da88e490189525399899ea3b4faeb59fd576063b065e95e1cf8fc2cd13a8a2d
Jul 17 16:37:27.533013 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:27.532980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl" event={"ID":"ecc54fa1-e216-40b7-a715-1e14d3ebb39c","Type":"ContainerStarted","Data":"bfdfc0f2b4d15a8d6e42c31698b5ea4cfd2caee1f401b94f446daad1612f8023"}
Jul 17 16:37:27.533185 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:27.533021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl" event={"ID":"ecc54fa1-e216-40b7-a715-1e14d3ebb39c","Type":"ContainerStarted","Data":"9da88e490189525399899ea3b4faeb59fd576063b065e95e1cf8fc2cd13a8a2d"}
Jul 17 16:37:27.551725 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:27.551681    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl" podStartSLOduration=1.362415553 podStartE2EDuration="1.551667559s" totalImagesPullingTime="189.252006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:37:27.550535149 +0000 UTC m=+681.535622685" watchObservedRunningTime="2026-07-17 16:37:27.551667559 +0000 UTC m=+681.536755095"
Jul 17 16:37:28.859758 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:28.859717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:37:37.012507 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:37.012469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:37.012940 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:37.012559    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:37.024594 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:37.024570    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:37:37.586487 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:37:37.586461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl"
Jul 17 16:38:42.236274 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:42.236241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:38:42.333536 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:42.333502    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:38:43.362065 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.362015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:38:43.459863 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.459825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:38:43.735010 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.734928    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:38:43.735193 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.735161    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-84958fc94f-vbbj4" podUID="dc455ff4-9e24-4612-880b-0545e61d48dc" containerName="authorino" containerID="cri-o://8a1b4ffec543c1df9c55e5977190479326db500bdf99073a522a1e79fd4c0ecc" gracePeriod=30
Jul 17 16:38:43.872157 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.872128    2561 generic.go:356] "Generic (PLEG): container finished" podID="dc455ff4-9e24-4612-880b-0545e61d48dc" containerID="8a1b4ffec543c1df9c55e5977190479326db500bdf99073a522a1e79fd4c0ecc" exitCode=0
Jul 17 16:38:43.872291 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.872196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84958fc94f-vbbj4" event={"ID":"dc455ff4-9e24-4612-880b-0545e61d48dc","Type":"ContainerDied","Data":"8a1b4ffec543c1df9c55e5977190479326db500bdf99073a522a1e79fd4c0ecc"}
Jul 17 16:38:43.981842 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:43.981820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:38:44.041826 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.041806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8l6t\" (UniqueName: \"kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t\") pod \"dc455ff4-9e24-4612-880b-0545e61d48dc\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") "
Jul 17 16:38:44.041946 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.041874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert\") pod \"dc455ff4-9e24-4612-880b-0545e61d48dc\" (UID: \"dc455ff4-9e24-4612-880b-0545e61d48dc\") "
Jul 17 16:38:44.043822 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.043793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t" (OuterVolumeSpecName: "kube-api-access-q8l6t") pod "dc455ff4-9e24-4612-880b-0545e61d48dc" (UID: "dc455ff4-9e24-4612-880b-0545e61d48dc"). InnerVolumeSpecName "kube-api-access-q8l6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:38:44.050722 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.050701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dc455ff4-9e24-4612-880b-0545e61d48dc" (UID: "dc455ff4-9e24-4612-880b-0545e61d48dc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:38:44.142687 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.142662    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8l6t\" (UniqueName: \"kubernetes.io/projected/dc455ff4-9e24-4612-880b-0545e61d48dc-kube-api-access-q8l6t\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:38:44.142774 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.142688    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc455ff4-9e24-4612-880b-0545e61d48dc-tls-cert\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:38:44.877162 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.877137    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84958fc94f-vbbj4"
Jul 17 16:38:44.877551 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.877133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84958fc94f-vbbj4" event={"ID":"dc455ff4-9e24-4612-880b-0545e61d48dc","Type":"ContainerDied","Data":"8609ea9b219e0e526322368412fa122192e49d03c04426fff84b859991a28ab8"}
Jul 17 16:38:44.877551 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.877268    2561 scope.go:117] "RemoveContainer" containerID="8a1b4ffec543c1df9c55e5977190479326db500bdf99073a522a1e79fd4c0ecc"
Jul 17 16:38:44.894248 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.894222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:38:44.896412 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:44.896393    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84958fc94f-vbbj4"]
Jul 17 16:38:46.513713 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:38:46.513676    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc455ff4-9e24-4612-880b-0545e61d48dc" path="/var/lib/kubelet/pods/dc455ff4-9e24-4612-880b-0545e61d48dc/volumes"
Jul 17 16:39:18.972976 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:39:18.972931    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:39:28.060448 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:39:28.060412    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:39:38.374782 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:39:38.374750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:40:54.141085 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:40:54.141032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:40:55.332227 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:40:55.332200    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:06.401504 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:06.401423    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:41:06.402198 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:06.402176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:41:18.158503 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:18.158469    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:41:26.554275 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:26.554242    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:41:28.514378 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:28.514335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:29.761892 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:29.761864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:34.509383 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:34.509332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:34.611793 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:34.611764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:34.714359 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:34.714317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:34.838879 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:34.838848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:40.517701 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:40.517660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:40.615084 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:40.615022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:53.654338 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:53.654289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:53.715085 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:53.715026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:41:58.156865 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:41:58.156828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:07.354511 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:07.354470    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:17.764204 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:17.764168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:23.562220 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:23.562191    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:33.866490 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:33.866418    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:43.761571 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:43.761533    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:54.665189 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:54.665160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:42:55.764756 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:42:55.764718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:43:03.966434 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:43:03.966399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:43:29.629349 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:43:29.629314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:43:29.980372 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:43:29.980276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:43:41.062516 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:43:41.062479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:43:56.193691 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:43:56.193653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:44:03.461305 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:44:03.461225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:44:15.961734 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:44:15.961698    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:44:31.558010 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:44:31.557969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:44:42.067890 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:44:42.067857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:44:51.980197 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:44:51.980167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:45:01.565016 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:45:01.564980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:45:11.065453 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:45:11.065422    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:46:06.427441 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:46:06.427364    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:46:06.429350 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:46:06.429329    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:47:00.867194 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:00.867109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:47:12.366679 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:12.366644    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:47:20.869441 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:20.869402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:47:33.028564 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:33.028527    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:47:43.054332 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:43.054290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:47:52.158128 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:47:52.158095    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:02.966628 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:02.966586    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:12.669272 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:12.669232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:24.673495 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:24.673465    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:33.395017 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:33.394938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:46.575032 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:46.574992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:48:51.835311 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:48:51.835260    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:49:07.260484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:49:07.260449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:49:16.563406 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:49:16.563368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:49:26.456900 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:49:26.456858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:49:36.878703 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:49:36.878670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:49:56.462837 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:49:56.462803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:50:08.251408 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:08.251375    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:50:17.148228 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:17.148196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:50:17.664615 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:17.664571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:50:26.861222 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:26.861190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:50:36.658009 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:36.657971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:50:46.457847 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:50:46.457810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:00.060609 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:00.060572    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:06.452834 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:06.452807    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:51:06.456611 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:06.456589    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:51:09.760620 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:09.760584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:24.053543 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:24.053512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:28.935593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:28.935510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:51:32.262393 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:32.262356    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:43.859484 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:43.859449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:51:59.675389 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:51:59.675357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:52:11.874502 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:11.874465    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:52:24.278238 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:24.278198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:52:26.035422 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:26.035383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:33.429792 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:33.429758    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:35.666965 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:35.666927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:52:41.675164 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:41.675131    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:46.793715 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:46.793678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:48.770039 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:48.770003    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:52:49.663688 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:49.663653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:49.756307 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:49.756277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:50.120285 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:50.120252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:55.987383 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:55.987349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:56.064599 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:56.064571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:56.216520 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:56.216479    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:56.313943 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:56.313912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:57.289131 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:57.289100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:57.389600 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:57.389560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:57.489090 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:57.489053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:57.695245 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:57.695155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:58.064655 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:58.064625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:52:58.156300 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:52:58.156267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:53:00.176394 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:53:00.176358    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:53:01.657508 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:53:01.657466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 16:55:30.527395 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:55:30.527360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:55:31.501953 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:55:31.501920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:56:06.478708 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:06.478676    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:56:06.483402 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:06.483382    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 16:56:17.083512 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:17.083475    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:56:17.222757 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:17.222723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:56:23.466216 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:23.466167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:56:23.567779 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:56:23.567752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:58:04.307111 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.307012    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:04.309605 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.307405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc455ff4-9e24-4612-880b-0545e61d48dc" containerName="authorino"
Jul 17 16:58:04.309605 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.307417    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc455ff4-9e24-4612-880b-0545e61d48dc" containerName="authorino"
Jul 17 16:58:04.309605 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.307481    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc455ff4-9e24-4612-880b-0545e61d48dc" containerName="authorino"
Jul 17 16:58:04.310569 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.310554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:04.312941 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.312910    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 16:58:04.313089 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.312921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 16:58:04.327098 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.327068    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:04.377495 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.377461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skhnh\" (UID: \"c8662b3f-de67-459a-9ce0-a1d1de5b0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:04.479064 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.479016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skhnh\" (UID: \"c8662b3f-de67-459a-9ce0-a1d1de5b0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:04.479660 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.479638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skhnh\" (UID: \"c8662b3f-de67-459a-9ce0-a1d1de5b0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:04.620029 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.619940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:04.957641 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.957620    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:04.959459 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:58:04.959431    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8662b3f_de67_459a_9ce0_a1d1de5b0867.slice/crio-dec3b9c14de43a215125c7825cc4ea1edcf66445daeff21d40c8f400a8ff6b66 WatchSource:0}: Error finding container dec3b9c14de43a215125c7825cc4ea1edcf66445daeff21d40c8f400a8ff6b66: Status 404 returned error can't find the container with id dec3b9c14de43a215125c7825cc4ea1edcf66445daeff21d40c8f400a8ff6b66
Jul 17 16:58:04.961215 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:04.961198    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 16:58:05.942919 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:05.942882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerStarted","Data":"dec3b9c14de43a215125c7825cc4ea1edcf66445daeff21d40c8f400a8ff6b66"}
Jul 17 16:58:14.984275 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:14.984200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerStarted","Data":"adbafa4fc3b3a930695af53108d1f7eb2aced14a43f08aedaa53bd0890d49ae5"}
Jul 17 16:58:14.999123 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:14.999075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" podStartSLOduration=1.329840101 podStartE2EDuration="10.999035173s" totalImagesPullingTime="9.669195072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:58:14.998606652 +0000 UTC m=+1928.983694264" watchObservedRunningTime="2026-07-17 16:58:14.999035173 +0000 UTC m=+1928.984122713"
Jul 17 16:58:15.923235 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:15.923199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:58:15.989865 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:15.989832    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerID="adbafa4fc3b3a930695af53108d1f7eb2aced14a43f08aedaa53bd0890d49ae5" exitCode=7
Jul 17 16:58:15.990241 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:15.989915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerDied","Data":"adbafa4fc3b3a930695af53108d1f7eb2aced14a43f08aedaa53bd0890d49ae5"}
Jul 17 16:58:15.990241 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:15.990088    2561 scope.go:117] "RemoveContainer" containerID="adbafa4fc3b3a930695af53108d1f7eb2aced14a43f08aedaa53bd0890d49ae5"
Jul 17 16:58:16.010608 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:16.010580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 16:58:16.996064 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:16.995967    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerID="f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178" exitCode=7
Jul 17 16:58:16.996064 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:16.996004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerDied","Data":"f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178"}
Jul 17 16:58:16.996559 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:16.996085    2561 scope.go:117] "RemoveContainer" containerID="adbafa4fc3b3a930695af53108d1f7eb2aced14a43f08aedaa53bd0890d49ae5"
Jul 17 16:58:16.996559 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:16.996252    2561 scope.go:117] "RemoveContainer" containerID="f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178"
Jul 17 16:58:16.996559 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:58:16.996474    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skhnh_odh-ai-gateway-infra(c8662b3f-de67-459a-9ce0-a1d1de5b0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867"
Jul 17 16:58:18.002617 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:18.002589    2561 scope.go:117] "RemoveContainer" containerID="f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178"
Jul 17 16:58:18.003003 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:58:18.002764    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skhnh_odh-ai-gateway-infra(c8662b3f-de67-459a-9ce0-a1d1de5b0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867"
Jul 17 16:58:32.508444 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:32.508412    2561 scope.go:117] "RemoveContainer" containerID="f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178"
Jul 17 16:58:33.070745 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:33.070671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerStarted","Data":"e34504b5a559db13924dfc4ee0a9176f74d58a42d209ea62a43daf167277180c"}
Jul 17 16:58:34.077132 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:34.077099    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerID="e34504b5a559db13924dfc4ee0a9176f74d58a42d209ea62a43daf167277180c" exitCode=7
Jul 17 16:58:34.077598 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:34.077178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerDied","Data":"e34504b5a559db13924dfc4ee0a9176f74d58a42d209ea62a43daf167277180c"}
Jul 17 16:58:34.077598 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:34.077242    2561 scope.go:117] "RemoveContainer" containerID="f943881b58ea10a6048cec29685d1f8ada1552628e98b72fc898cc78efb17178"
Jul 17 16:58:34.077598 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:34.077467    2561 scope.go:117] "RemoveContainer" containerID="e34504b5a559db13924dfc4ee0a9176f74d58a42d209ea62a43daf167277180c"
Jul 17 16:58:34.077725 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:58:34.077662    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skhnh_odh-ai-gateway-infra(c8662b3f-de67-459a-9ce0-a1d1de5b0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867"
Jul 17 16:58:34.104640 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:34.104615    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:35.215796 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:35.215770    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:35.229319 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:35.229295    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca\") pod \"c8662b3f-de67-459a-9ce0-a1d1de5b0867\" (UID: \"c8662b3f-de67-459a-9ce0-a1d1de5b0867\") "
Jul 17 16:58:35.229624 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:35.229600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8662b3f-de67-459a-9ce0-a1d1de5b0867" (UID: "c8662b3f-de67-459a-9ce0-a1d1de5b0867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:58:35.330420 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:35.330386    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8662b3f-de67-459a-9ce0-a1d1de5b0867-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:58:36.089474 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.089442    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"
Jul 17 16:58:36.089474 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.089470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh" event={"ID":"c8662b3f-de67-459a-9ce0-a1d1de5b0867","Type":"ContainerDied","Data":"dec3b9c14de43a215125c7825cc4ea1edcf66445daeff21d40c8f400a8ff6b66"}
Jul 17 16:58:36.089750 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.089529    2561 scope.go:117] "RemoveContainer" containerID="e34504b5a559db13924dfc4ee0a9176f74d58a42d209ea62a43daf167277180c"
Jul 17 16:58:36.115350 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.115327    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:36.122426 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.122404    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skhnh"]
Jul 17 16:58:36.519126 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:58:36.515260    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" path="/var/lib/kubelet/pods/c8662b3f-de67-459a-9ce0-a1d1de5b0867/volumes"
Jul 17 16:59:14.391984 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.391951    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392283    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392295    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392319    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392324    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392370    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392377    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.392388 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.392384    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 16:59:14.395320 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.395305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:14.400113 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.400081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 16:59:14.400283 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.400264    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 16:59:14.408196 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.408176    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:14.473549 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.473514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-h7bxz\" (UID: \"ff079470-a2b6-4f67-b2f1-2444af7c8f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:14.575901 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.575857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-h7bxz\" (UID: \"ff079470-a2b6-4f67-b2f1-2444af7c8f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:14.576593 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.576569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-h7bxz\" (UID: \"ff079470-a2b6-4f67-b2f1-2444af7c8f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:14.705175 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.705096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:14.849943 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:14.849918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:14.850585 ip-10-0-137-172 kubenswrapper[2561]: W0717 16:59:14.850563    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff079470_a2b6_4f67_b2f1_2444af7c8f2e.slice/crio-6cdab9ac7f2246dc74a9f50086689ba038bfadc407fe5c50f568a46750968c31 WatchSource:0}: Error finding container 6cdab9ac7f2246dc74a9f50086689ba038bfadc407fe5c50f568a46750968c31: Status 404 returned error can't find the container with id 6cdab9ac7f2246dc74a9f50086689ba038bfadc407fe5c50f568a46750968c31
Jul 17 16:59:15.261807 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:15.261771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerStarted","Data":"56e631bd58444a3001a8b61a95b4b30ba5389c98e14445a7314c83d40681607a"}
Jul 17 16:59:15.261978 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:15.261821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerStarted","Data":"6cdab9ac7f2246dc74a9f50086689ba038bfadc407fe5c50f568a46750968c31"}
Jul 17 16:59:15.283029 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:15.282983    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" podStartSLOduration=1.282968959 podStartE2EDuration="1.282968959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:59:15.28208096 +0000 UTC m=+1989.267168509" watchObservedRunningTime="2026-07-17 16:59:15.282968959 +0000 UTC m=+1989.268056496"
Jul 17 16:59:16.267464 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:16.267432    2561 generic.go:356] "Generic (PLEG): container finished" podID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerID="56e631bd58444a3001a8b61a95b4b30ba5389c98e14445a7314c83d40681607a" exitCode=7
Jul 17 16:59:16.267873 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:16.267482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerDied","Data":"56e631bd58444a3001a8b61a95b4b30ba5389c98e14445a7314c83d40681607a"}
Jul 17 16:59:16.267873 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:16.267718    2561 scope.go:117] "RemoveContainer" containerID="56e631bd58444a3001a8b61a95b4b30ba5389c98e14445a7314c83d40681607a"
Jul 17 16:59:17.273999 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:17.273964    2561 generic.go:356] "Generic (PLEG): container finished" podID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerID="713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c" exitCode=7
Jul 17 16:59:17.274481 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:17.274059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerDied","Data":"713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c"}
Jul 17 16:59:17.274481 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:17.274117    2561 scope.go:117] "RemoveContainer" containerID="56e631bd58444a3001a8b61a95b4b30ba5389c98e14445a7314c83d40681607a"
Jul 17 16:59:17.274481 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:17.274344    2561 scope.go:117] "RemoveContainer" containerID="713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c"
Jul 17 16:59:17.274631 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:59:17.274534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-h7bxz_odh-ai-gateway-infra(ff079470-a2b6-4f67-b2f1-2444af7c8f2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e"
Jul 17 16:59:18.280619 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:18.280594    2561 scope.go:117] "RemoveContainer" containerID="713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c"
Jul 17 16:59:18.281004 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:59:18.280785    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-h7bxz_odh-ai-gateway-infra(ff079470-a2b6-4f67-b2f1-2444af7c8f2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e"
Jul 17 16:59:32.508770 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:32.508732    2561 scope.go:117] "RemoveContainer" containerID="713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c"
Jul 17 16:59:33.350865 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:33.350834    2561 generic.go:356] "Generic (PLEG): container finished" podID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerID="7bb5304037ac8154e20003e25573f1b03930173899be569281ea0d949914e9d0" exitCode=7
Jul 17 16:59:33.351023 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:33.350884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerDied","Data":"7bb5304037ac8154e20003e25573f1b03930173899be569281ea0d949914e9d0"}
Jul 17 16:59:33.351023 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:33.350939    2561 scope.go:117] "RemoveContainer" containerID="713e974da2eb2f6346d2240626bd6531245c3494db6fadbed312c37c946ce91c"
Jul 17 16:59:33.351274 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:33.351255    2561 scope.go:117] "RemoveContainer" containerID="7bb5304037ac8154e20003e25573f1b03930173899be569281ea0d949914e9d0"
Jul 17 16:59:33.351530 ip-10-0-137-172 kubenswrapper[2561]: E0717 16:59:33.351507    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-h7bxz_odh-ai-gateway-infra(ff079470-a2b6-4f67-b2f1-2444af7c8f2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e"
Jul 17 16:59:34.381664 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:34.381637    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:34.511107 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:34.511082    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:34.654725 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:34.654652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca\") pod \"ff079470-a2b6-4f67-b2f1-2444af7c8f2e\" (UID: \"ff079470-a2b6-4f67-b2f1-2444af7c8f2e\") "
Jul 17 16:59:34.655309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:34.655065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff079470-a2b6-4f67-b2f1-2444af7c8f2e" (UID: "ff079470-a2b6-4f67-b2f1-2444af7c8f2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:59:34.655309 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:34.655189    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff079470-a2b6-4f67-b2f1-2444af7c8f2e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 16:59:35.362781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:35.362743    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"
Jul 17 16:59:35.362781 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:35.362754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz" event={"ID":"ff079470-a2b6-4f67-b2f1-2444af7c8f2e","Type":"ContainerDied","Data":"6cdab9ac7f2246dc74a9f50086689ba038bfadc407fe5c50f568a46750968c31"}
Jul 17 16:59:35.363026 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:35.362815    2561 scope.go:117] "RemoveContainer" containerID="7bb5304037ac8154e20003e25573f1b03930173899be569281ea0d949914e9d0"
Jul 17 16:59:35.386457 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:35.386435    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:35.388710 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:35.388685    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-h7bxz"]
Jul 17 16:59:36.518753 ip-10-0-137-172 kubenswrapper[2561]: I0717 16:59:36.518721    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" path="/var/lib/kubelet/pods/ff079470-a2b6-4f67-b2f1-2444af7c8f2e/volumes"
Jul 17 17:00:00.157499 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157803    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157814    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8662b3f-de67-459a-9ce0-a1d1de5b0867" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157822    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157828    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157851    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157857    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157907    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.157978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.157916    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.160446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.160425    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"
Jul 17 17:00:00.161156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.161136    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"]
Jul 17 17:00:00.161435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.161420    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.161503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.161435    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.161539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.161507    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff079470-a2b6-4f67-b2f1-2444af7c8f2e" containerName="revoke-keys"
Jul 17 17:00:00.162774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.162757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:00:00.164463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.164447    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"
Jul 17 17:00:00.170508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.170493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"
Jul 17 17:00:00.174039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.173907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:00.177833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.177811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"]
Jul 17 17:00:00.179766 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.179748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"
Jul 17 17:00:00.321019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.320988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:00.323546 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:00:00.323518    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f17ae7d_9cb9_4396_aa95_29b4be4c8353.slice/crio-3f75b12f738a9771d147c9bb5c48d338d2f5934aeb05471a613319e404b782ac WatchSource:0}: Error finding container 3f75b12f738a9771d147c9bb5c48d338d2f5934aeb05471a613319e404b782ac: Status 404 returned error can't find the container with id 3f75b12f738a9771d147c9bb5c48d338d2f5934aeb05471a613319e404b782ac
Jul 17 17:00:00.338253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.338233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"]
Jul 17 17:00:00.342005 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:00:00.341971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d9799e_1c73_449b_9f7b_ef03dac9952f.slice/crio-1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460 WatchSource:0}: Error finding container 1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460: Status 404 returned error can't find the container with id 1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460
Jul 17 17:00:00.478877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.478846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq" event={"ID":"85d9799e-1c73-449b-9f7b-ef03dac9952f","Type":"ContainerStarted","Data":"1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460"}
Jul 17 17:00:00.479895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:00.479868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" event={"ID":"6f17ae7d-9cb9-4396-aa95-29b4be4c8353","Type":"ContainerStarted","Data":"3f75b12f738a9771d147c9bb5c48d338d2f5934aeb05471a613319e404b782ac"}
Jul 17 17:00:01.486224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:01.486192    2561 generic.go:356] "Generic (PLEG): container finished" podID="85d9799e-1c73-449b-9f7b-ef03dac9952f" containerID="8883b53a407c8129f9264c8083f4964844cc8dafc49cccf8fee07f1a6709efb3" exitCode=0
Jul 17 17:00:01.486658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:01.486267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq" event={"ID":"85d9799e-1c73-449b-9f7b-ef03dac9952f","Type":"ContainerDied","Data":"8883b53a407c8129f9264c8083f4964844cc8dafc49cccf8fee07f1a6709efb3"}
Jul 17 17:00:01.487708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:01.487686    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerID="e08b657179b149fb3d2937c9413a89e15c081caf89a49b3c9d6611c432b81a69" exitCode=7
Jul 17 17:00:01.487809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:01.487753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" event={"ID":"6f17ae7d-9cb9-4396-aa95-29b4be4c8353","Type":"ContainerDied","Data":"e08b657179b149fb3d2937c9413a89e15c081caf89a49b3c9d6611c432b81a69"}
Jul 17 17:00:01.487887 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:01.487874    2561 scope.go:117] "RemoveContainer" containerID="e08b657179b149fb3d2937c9413a89e15c081caf89a49b3c9d6611c432b81a69"
Jul 17 17:00:02.494332 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:02.494232    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerID="1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227" exitCode=7
Jul 17 17:00:02.494332 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:02.494315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" event={"ID":"6f17ae7d-9cb9-4396-aa95-29b4be4c8353","Type":"ContainerDied","Data":"1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227"}
Jul 17 17:00:02.494878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:02.494406    2561 scope.go:117] "RemoveContainer" containerID="e08b657179b149fb3d2937c9413a89e15c081caf89a49b3c9d6611c432b81a69"
Jul 17 17:00:02.494878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:02.494543    2561 scope.go:117] "RemoveContainer" containerID="1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227"
Jul 17 17:00:02.494878 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:02.494787    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m_odh-ai-gateway-infra(6f17ae7d-9cb9-4396-aa95-29b4be4c8353)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353"
Jul 17 17:00:02.631534 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:02.631504    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"
Jul 17 17:00:03.500226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:03.500200    2561 scope.go:117] "RemoveContainer" containerID="1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227"
Jul 17 17:00:03.500620 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:03.500417    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m_odh-ai-gateway-infra(6f17ae7d-9cb9-4396-aa95-29b4be4c8353)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353"
Jul 17 17:00:03.501407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:03.501390    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq"
Jul 17 17:00:03.501496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:03.501387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738460-gk8dq" event={"ID":"85d9799e-1c73-449b-9f7b-ef03dac9952f","Type":"ContainerDied","Data":"1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460"}
Jul 17 17:00:03.501557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:03.501528    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a2e615457a6ec0d5d4afc72faa3dda182478ea04cbafea00e5406dd35d04460"
Jul 17 17:00:14.432386 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.432345    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:14.432787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.432719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d9799e-1c73-449b-9f7b-ef03dac9952f" containerName="cleanup"
Jul 17 17:00:14.432787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.432730    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d9799e-1c73-449b-9f7b-ef03dac9952f" containerName="cleanup"
Jul 17 17:00:14.432891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.432790    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d9799e-1c73-449b-9f7b-ef03dac9952f" containerName="cleanup"
Jul 17 17:00:14.435794 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.435778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:14.439061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.439020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:00:14.447160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.447140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:14.499728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.499697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-78z57\" (UID: \"17233509-795d-4b75-a5b9-9e890b77f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:14.600527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.600447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-78z57\" (UID: \"17233509-795d-4b75-a5b9-9e890b77f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:14.601091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.601073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-78z57\" (UID: \"17233509-795d-4b75-a5b9-9e890b77f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:14.745017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.744979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:14.881971 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:14.881896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:14.884826 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:00:14.884788    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17233509_795d_4b75_a5b9_9e890b77f1e3.slice/crio-3903eff1e0b6a33d20d25af1a58ed14ba3dc531b5ee4e9c6464628e1955c3154 WatchSource:0}: Error finding container 3903eff1e0b6a33d20d25af1a58ed14ba3dc531b5ee4e9c6464628e1955c3154: Status 404 returned error can't find the container with id 3903eff1e0b6a33d20d25af1a58ed14ba3dc531b5ee4e9c6464628e1955c3154
Jul 17 17:00:15.554979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:15.554948    2561 generic.go:356] "Generic (PLEG): container finished" podID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerID="341f0c5f5fdf9188e208bffd3d1c30ee31287127820e254ea3f337de4075afeb" exitCode=7
Jul 17 17:00:15.555307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:15.555006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" event={"ID":"17233509-795d-4b75-a5b9-9e890b77f1e3","Type":"ContainerDied","Data":"341f0c5f5fdf9188e208bffd3d1c30ee31287127820e254ea3f337de4075afeb"}
Jul 17 17:00:15.555307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:15.555063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" event={"ID":"17233509-795d-4b75-a5b9-9e890b77f1e3","Type":"ContainerStarted","Data":"3903eff1e0b6a33d20d25af1a58ed14ba3dc531b5ee4e9c6464628e1955c3154"}
Jul 17 17:00:15.555307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:15.555193    2561 scope.go:117] "RemoveContainer" containerID="341f0c5f5fdf9188e208bffd3d1c30ee31287127820e254ea3f337de4075afeb"
Jul 17 17:00:16.560777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:16.560740    2561 generic.go:356] "Generic (PLEG): container finished" podID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerID="3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a" exitCode=7
Jul 17 17:00:16.561268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:16.560780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" event={"ID":"17233509-795d-4b75-a5b9-9e890b77f1e3","Type":"ContainerDied","Data":"3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a"}
Jul 17 17:00:16.561268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:16.560838    2561 scope.go:117] "RemoveContainer" containerID="341f0c5f5fdf9188e208bffd3d1c30ee31287127820e254ea3f337de4075afeb"
Jul 17 17:00:16.561268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:16.561009    2561 scope.go:117] "RemoveContainer" containerID="3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a"
Jul 17 17:00:16.561268 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:16.561201    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-78z57_odh-ai-gateway-infra(17233509-795d-4b75-a5b9-9e890b77f1e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3"
Jul 17 17:00:17.508404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:17.508373    2561 scope.go:117] "RemoveContainer" containerID="1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227"
Jul 17 17:00:17.567252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:17.567230    2561 scope.go:117] "RemoveContainer" containerID="3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a"
Jul 17 17:00:17.567574 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:17.567413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-78z57_odh-ai-gateway-infra(17233509-795d-4b75-a5b9-9e890b77f1e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3"
Jul 17 17:00:18.572675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:18.572643    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerID="e30b650d18f36b734d786e256576e0674f1782e3d25aa920bf94783a6d56d3c1" exitCode=7
Jul 17 17:00:18.573122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:18.572715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" event={"ID":"6f17ae7d-9cb9-4396-aa95-29b4be4c8353","Type":"ContainerDied","Data":"e30b650d18f36b734d786e256576e0674f1782e3d25aa920bf94783a6d56d3c1"}
Jul 17 17:00:18.573122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:18.572772    2561 scope.go:117] "RemoveContainer" containerID="1c46ae3fbc89a237c1908c36c59a78255328467ef04cb735484a558c6fa6f227"
Jul 17 17:00:18.573122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:18.573065    2561 scope.go:117] "RemoveContainer" containerID="e30b650d18f36b734d786e256576e0674f1782e3d25aa920bf94783a6d56d3c1"
Jul 17 17:00:18.573288 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:18.573269    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m_odh-ai-gateway-infra(6f17ae7d-9cb9-4396-aa95-29b4be4c8353)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353"
Jul 17 17:00:19.613242 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:19.613211    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:19.761385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:19.761357    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"
Jul 17 17:00:20.584773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:20.584747    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"
Jul 17 17:00:20.584948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:20.584746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m" event={"ID":"6f17ae7d-9cb9-4396-aa95-29b4be4c8353","Type":"ContainerDied","Data":"3f75b12f738a9771d147c9bb5c48d338d2f5934aeb05471a613319e404b782ac"}
Jul 17 17:00:20.584948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:20.584879    2561 scope.go:117] "RemoveContainer" containerID="e30b650d18f36b734d786e256576e0674f1782e3d25aa920bf94783a6d56d3c1"
Jul 17 17:00:20.600724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:20.600702    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:20.604791 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:20.604773    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738460-ngj4m"]
Jul 17 17:00:22.514205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:22.514172    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" path="/var/lib/kubelet/pods/6f17ae7d-9cb9-4396-aa95-29b4be4c8353/volumes"
Jul 17 17:00:29.508765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:29.508687    2561 scope.go:117] "RemoveContainer" containerID="3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a"
Jul 17 17:00:30.634383 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:30.634353    2561 generic.go:356] "Generic (PLEG): container finished" podID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerID="36ff01aaa620f34c7309c3b2bb12db5f9858a0e45b3446684dccd2fa1a4ea9ef" exitCode=7
Jul 17 17:00:30.634806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:30.634409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" event={"ID":"17233509-795d-4b75-a5b9-9e890b77f1e3","Type":"ContainerDied","Data":"36ff01aaa620f34c7309c3b2bb12db5f9858a0e45b3446684dccd2fa1a4ea9ef"}
Jul 17 17:00:30.634806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:30.634453    2561 scope.go:117] "RemoveContainer" containerID="3c6c45caf2428fa458d1f06a73e2968ef3353d0b75e31b5cf5d6c357789b3f2a"
Jul 17 17:00:30.634806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:30.634715    2561 scope.go:117] "RemoveContainer" containerID="36ff01aaa620f34c7309c3b2bb12db5f9858a0e45b3446684dccd2fa1a4ea9ef"
Jul 17 17:00:30.634959 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:00:30.634887    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-78z57_odh-ai-gateway-infra(17233509-795d-4b75-a5b9-9e890b77f1e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3"
Jul 17 17:00:31.660062 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:31.660016    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:31.791231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:31.791210    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:31.958711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:31.958638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca\") pod \"17233509-795d-4b75-a5b9-9e890b77f1e3\" (UID: \"17233509-795d-4b75-a5b9-9e890b77f1e3\") "
Jul 17 17:00:31.958990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:31.958968    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17233509-795d-4b75-a5b9-9e890b77f1e3" (UID: "17233509-795d-4b75-a5b9-9e890b77f1e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:00:32.060202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.060178    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17233509-795d-4b75-a5b9-9e890b77f1e3-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:00:32.645375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.645345    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"
Jul 17 17:00:32.645547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.645378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57" event={"ID":"17233509-795d-4b75-a5b9-9e890b77f1e3","Type":"ContainerDied","Data":"3903eff1e0b6a33d20d25af1a58ed14ba3dc531b5ee4e9c6464628e1955c3154"}
Jul 17 17:00:32.645547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.645426    2561 scope.go:117] "RemoveContainer" containerID="36ff01aaa620f34c7309c3b2bb12db5f9858a0e45b3446684dccd2fa1a4ea9ef"
Jul 17 17:00:32.662202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.662176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:32.668888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:32.668863    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-78z57"]
Jul 17 17:00:34.513736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:00:34.513705    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" path="/var/lib/kubelet/pods/17233509-795d-4b75-a5b9-9e890b77f1e3/volumes"
Jul 17 17:01:04.479077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479035    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479568    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479589    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479605    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479615    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479633    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479656    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.479668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479664    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479675    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479683    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479789    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479802    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479810    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.480098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.479820    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f17ae7d-9cb9-4396-aa95-29b4be4c8353" containerName="cleanup"
Jul 17 17:01:04.480445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.480367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:04.483314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.483294    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:04.485820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.485788    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:01:04.486109 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.486092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:01:04.498557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.498539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:04.522760 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.522730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9cdnn\" (UID: \"551751f0-75e5-4eae-a6bb-667e07c5cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:04.623697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.623670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9cdnn\" (UID: \"551751f0-75e5-4eae-a6bb-667e07c5cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:04.624335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.624312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9cdnn\" (UID: \"551751f0-75e5-4eae-a6bb-667e07c5cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:04.792577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.792553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:04.932479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:04.932455    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:04.935907 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:01:04.935875    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551751f0_75e5_4eae_a6bb_667e07c5cbcf.slice/crio-fd9617cf0e714b7323ad6a053592d39b96bcdc94d492e9e6a54ecc43f397d177 WatchSource:0}: Error finding container fd9617cf0e714b7323ad6a053592d39b96bcdc94d492e9e6a54ecc43f397d177: Status 404 returned error can't find the container with id fd9617cf0e714b7323ad6a053592d39b96bcdc94d492e9e6a54ecc43f397d177
Jul 17 17:01:05.799667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:05.799635    2561 generic.go:356] "Generic (PLEG): container finished" podID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerID="a300d8104ae2a4f4c8b8ff82813564b73916c081217e9e7436a36a852d5d03b5" exitCode=7
Jul 17 17:01:05.800063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:05.799724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerDied","Data":"a300d8104ae2a4f4c8b8ff82813564b73916c081217e9e7436a36a852d5d03b5"}
Jul 17 17:01:05.800063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:05.799783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerStarted","Data":"fd9617cf0e714b7323ad6a053592d39b96bcdc94d492e9e6a54ecc43f397d177"}
Jul 17 17:01:05.800063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:05.799848    2561 scope.go:117] "RemoveContainer" containerID="a300d8104ae2a4f4c8b8ff82813564b73916c081217e9e7436a36a852d5d03b5"
Jul 17 17:01:06.507025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.507000    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:01:06.513935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.513914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:01:06.806271 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.806242    2561 generic.go:356] "Generic (PLEG): container finished" podID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerID="e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069" exitCode=7
Jul 17 17:01:06.806686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.806297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerDied","Data":"e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069"}
Jul 17 17:01:06.806686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.806341    2561 scope.go:117] "RemoveContainer" containerID="a300d8104ae2a4f4c8b8ff82813564b73916c081217e9e7436a36a852d5d03b5"
Jul 17 17:01:06.806686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:06.806529    2561 scope.go:117] "RemoveContainer" containerID="e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069"
Jul 17 17:01:06.806835 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:01:06.806736    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9cdnn_odh-ai-gateway-infra(551751f0-75e5-4eae-a6bb-667e07c5cbcf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf"
Jul 17 17:01:07.191186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:07.191093    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:01:07.588268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:07.588240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:07.691031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:07.691005    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:07.812723 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:07.812699    2561 scope.go:117] "RemoveContainer" containerID="e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069"
Jul 17 17:01:07.813109 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:01:07.812882    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9cdnn_odh-ai-gateway-infra(551751f0-75e5-4eae-a6bb-667e07c5cbcf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf"
Jul 17 17:01:17.933437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:17.933402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:18.033126 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:18.033098    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:18.660495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:18.660466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:01:22.509165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:22.509131    2561 scope.go:117] "RemoveContainer" containerID="e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069"
Jul 17 17:01:22.879389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:22.879363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerStarted","Data":"a6f3b46dc5caaf8c7bddcd24c9e0ca11a3b7110cd94896eff2348eb36a2f2f02"}
Jul 17 17:01:22.894853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:22.894809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" podStartSLOduration=18.89479406 podStartE2EDuration="18.89479406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:01:22.892977392 +0000 UTC m=+2116.878064930" watchObservedRunningTime="2026-07-17 17:01:22.89479406 +0000 UTC m=+2116.879881597"
Jul 17 17:01:23.885874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:23.885840    2561 generic.go:356] "Generic (PLEG): container finished" podID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerID="a6f3b46dc5caaf8c7bddcd24c9e0ca11a3b7110cd94896eff2348eb36a2f2f02" exitCode=7
Jul 17 17:01:23.886294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:23.885914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerDied","Data":"a6f3b46dc5caaf8c7bddcd24c9e0ca11a3b7110cd94896eff2348eb36a2f2f02"}
Jul 17 17:01:23.886294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:23.885968    2561 scope.go:117] "RemoveContainer" containerID="e86a541ce3f87168ab83aa8e25cbd37779411436898860b062dd597f677a2069"
Jul 17 17:01:23.886294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:23.886259    2561 scope.go:117] "RemoveContainer" containerID="a6f3b46dc5caaf8c7bddcd24c9e0ca11a3b7110cd94896eff2348eb36a2f2f02"
Jul 17 17:01:23.886461 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:01:23.886438    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9cdnn_odh-ai-gateway-infra(551751f0-75e5-4eae-a6bb-667e07c5cbcf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf"
Jul 17 17:01:23.902164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:23.902137    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:25.029335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.029311    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:25.199946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.199872    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca\") pod \"551751f0-75e5-4eae-a6bb-667e07c5cbcf\" (UID: \"551751f0-75e5-4eae-a6bb-667e07c5cbcf\") "
Jul 17 17:01:25.200281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.200260    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "551751f0-75e5-4eae-a6bb-667e07c5cbcf" (UID: "551751f0-75e5-4eae-a6bb-667e07c5cbcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:01:25.300720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.300690    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551751f0-75e5-4eae-a6bb-667e07c5cbcf-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:01:25.793816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.793783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:25.854148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.854107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:25.899917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.899890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn" event={"ID":"551751f0-75e5-4eae-a6bb-667e07c5cbcf","Type":"ContainerDied","Data":"fd9617cf0e714b7323ad6a053592d39b96bcdc94d492e9e6a54ecc43f397d177"}
Jul 17 17:01:25.900111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.899947    2561 scope.go:117] "RemoveContainer" containerID="a6f3b46dc5caaf8c7bddcd24c9e0ca11a3b7110cd94896eff2348eb36a2f2f02"
Jul 17 17:01:25.900111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.899903    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"
Jul 17 17:01:25.922530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.922508    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:25.926891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:25.926871    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9cdnn"]
Jul 17 17:01:26.514887 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:26.514854    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" path="/var/lib/kubelet/pods/551751f0-75e5-4eae-a6bb-667e07c5cbcf/volumes"
Jul 17 17:01:27.770974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:27.770940    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:27.894787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:27.894754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:01:29.478667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:01:29.478639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:02:04.517612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517535    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517840    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517851    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517865    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517871    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517881    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517886    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517901    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517906    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517961    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17233509-795d-4b75-a5b9-9e890b77f1e3" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517967    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.518098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.517975    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:04.520597 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.520582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:04.522950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.522930    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:02:04.523078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.522973    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:02:04.528017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.527987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:04.618333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.618300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-kjvpp\" (UID: \"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:04.719165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.719138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-kjvpp\" (UID: \"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:04.719730 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.719712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-kjvpp\" (UID: \"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:04.833582 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.833559    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:04.961560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:04.961526    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:04.964407 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:02:04.964380    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35ba1cc_60bc_42f1_a4e8_eb1205ecc700.slice/crio-7ed9c0c84969978201902f3dbcb4b121c32d3f4127d3af53111c73b9e5c83156 WatchSource:0}: Error finding container 7ed9c0c84969978201902f3dbcb4b121c32d3f4127d3af53111c73b9e5c83156: Status 404 returned error can't find the container with id 7ed9c0c84969978201902f3dbcb4b121c32d3f4127d3af53111c73b9e5c83156
Jul 17 17:02:05.091091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:05.090990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" event={"ID":"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700","Type":"ContainerStarted","Data":"7ed9c0c84969978201902f3dbcb4b121c32d3f4127d3af53111c73b9e5c83156"}
Jul 17 17:02:06.097276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:06.097241    2561 generic.go:356] "Generic (PLEG): container finished" podID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerID="3931f50d86735df52046da03857b53664a1a758a43445b3b5b0dbe7d04219795" exitCode=7
Jul 17 17:02:06.097664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:06.097321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" event={"ID":"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700","Type":"ContainerDied","Data":"3931f50d86735df52046da03857b53664a1a758a43445b3b5b0dbe7d04219795"}
Jul 17 17:02:06.097664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:06.097479    2561 scope.go:117] "RemoveContainer" containerID="3931f50d86735df52046da03857b53664a1a758a43445b3b5b0dbe7d04219795"
Jul 17 17:02:07.109211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:07.109177    2561 generic.go:356] "Generic (PLEG): container finished" podID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerID="94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f" exitCode=7
Jul 17 17:02:07.109695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:07.109230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" event={"ID":"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700","Type":"ContainerDied","Data":"94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f"}
Jul 17 17:02:07.109695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:07.109364    2561 scope.go:117] "RemoveContainer" containerID="3931f50d86735df52046da03857b53664a1a758a43445b3b5b0dbe7d04219795"
Jul 17 17:02:07.109817 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:07.109716    2561 scope.go:117] "RemoveContainer" containerID="94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f"
Jul 17 17:02:07.109953 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:02:07.109931    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-kjvpp_odh-ai-gateway-infra(f35ba1cc-60bc-42f1-a4e8-eb1205ecc700)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700"
Jul 17 17:02:08.116548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:08.116521    2561 scope.go:117] "RemoveContainer" containerID="94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f"
Jul 17 17:02:08.117065 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:02:08.116762    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-kjvpp_odh-ai-gateway-infra(f35ba1cc-60bc-42f1-a4e8-eb1205ecc700)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700"
Jul 17 17:02:18.509213 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:18.509178    2561 scope.go:117] "RemoveContainer" containerID="94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f"
Jul 17 17:02:19.167937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:19.167906    2561 generic.go:356] "Generic (PLEG): container finished" podID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerID="d0efc41815038f61ce277efdecaf60f330f2a0dfdf04d0adb60f7870ce84f6ed" exitCode=7
Jul 17 17:02:19.168082 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:19.167950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" event={"ID":"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700","Type":"ContainerDied","Data":"d0efc41815038f61ce277efdecaf60f330f2a0dfdf04d0adb60f7870ce84f6ed"}
Jul 17 17:02:19.168082 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:19.168011    2561 scope.go:117] "RemoveContainer" containerID="94177faf8905034a33f2680c2c8c7116b842b9c1173994733cf2c63fc63c757f"
Jul 17 17:02:19.168377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:19.168357    2561 scope.go:117] "RemoveContainer" containerID="d0efc41815038f61ce277efdecaf60f330f2a0dfdf04d0adb60f7870ce84f6ed"
Jul 17 17:02:19.168554 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:02:19.168534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-kjvpp_odh-ai-gateway-infra(f35ba1cc-60bc-42f1-a4e8-eb1205ecc700)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700"
Jul 17 17:02:20.192862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:20.192828    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:20.321029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:20.321005    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:20.449660 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:20.449586    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca\") pod \"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700\" (UID: \"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700\") "
Jul 17 17:02:20.449985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:20.449966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" (UID: "f35ba1cc-60bc-42f1-a4e8-eb1205ecc700"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:02:20.551114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:20.551084    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:02:21.179446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:21.179420    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"
Jul 17 17:02:21.179606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:21.179419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp" event={"ID":"f35ba1cc-60bc-42f1-a4e8-eb1205ecc700","Type":"ContainerDied","Data":"7ed9c0c84969978201902f3dbcb4b121c32d3f4127d3af53111c73b9e5c83156"}
Jul 17 17:02:21.179606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:21.179551    2561 scope.go:117] "RemoveContainer" containerID="d0efc41815038f61ce277efdecaf60f330f2a0dfdf04d0adb60f7870ce84f6ed"
Jul 17 17:02:21.194755 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:21.194732    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:21.203558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:21.203536    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-kjvpp"]
Jul 17 17:02:22.513944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:22.513903    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" path="/var/lib/kubelet/pods/f35ba1cc-60bc-42f1-a4e8-eb1205ecc700/volumes"
Jul 17 17:02:54.555997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.555963    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556343    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556357    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556365    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556378    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556386    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556446    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556456    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556463    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="551751f0-75e5-4eae-a6bb-667e07c5cbcf" containerName="revoke-keys"
Jul 17 17:02:54.556490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.556469    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35ba1cc-60bc-42f1-a4e8-eb1205ecc700" containerName="revoke-keys"
Jul 17 17:02:54.559366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.559346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:02:54.561864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.561832    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:02:54.561981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.561870    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:02:54.567825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.567800    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:02:54.644359 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.644320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-7bpj9\" (UID: \"20c38b80-1b01-4ea4-9199-e9cd343a0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:02:54.745810 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.745774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-7bpj9\" (UID: \"20c38b80-1b01-4ea4-9199-e9cd343a0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:02:54.746444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.746424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-7bpj9\" (UID: \"20c38b80-1b01-4ea4-9199-e9cd343a0867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:02:54.870322 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:54.870241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:02:55.010888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:55.010809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:02:55.013199 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:02:55.013171    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c38b80_1b01_4ea4_9199_e9cd343a0867.slice/crio-f0aa03bcebd3933802dee466345186f7c16c56a69d770935004719a2ca978064 WatchSource:0}: Error finding container f0aa03bcebd3933802dee466345186f7c16c56a69d770935004719a2ca978064: Status 404 returned error can't find the container with id f0aa03bcebd3933802dee466345186f7c16c56a69d770935004719a2ca978064
Jul 17 17:02:55.328281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:55.328246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerStarted","Data":"794550dd8bebb3c3aeb94c4c13f9a01bfb05944e91816f34ec71d221966c58e7"}
Jul 17 17:02:55.328463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:55.328311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerStarted","Data":"f0aa03bcebd3933802dee466345186f7c16c56a69d770935004719a2ca978064"}
Jul 17 17:02:55.346935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:55.346880    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" podStartSLOduration=1.346865873 podStartE2EDuration="1.346865873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:02:55.344989822 +0000 UTC m=+2209.330077359" watchObservedRunningTime="2026-07-17 17:02:55.346865873 +0000 UTC m=+2209.331953410"
Jul 17 17:02:56.334169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:56.334134    2561 generic.go:356] "Generic (PLEG): container finished" podID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerID="794550dd8bebb3c3aeb94c4c13f9a01bfb05944e91816f34ec71d221966c58e7" exitCode=7
Jul 17 17:02:56.334169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:56.334171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerDied","Data":"794550dd8bebb3c3aeb94c4c13f9a01bfb05944e91816f34ec71d221966c58e7"}
Jul 17 17:02:56.334667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:56.334363    2561 scope.go:117] "RemoveContainer" containerID="794550dd8bebb3c3aeb94c4c13f9a01bfb05944e91816f34ec71d221966c58e7"
Jul 17 17:02:57.340012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:57.339979    2561 generic.go:356] "Generic (PLEG): container finished" podID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerID="7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2" exitCode=7
Jul 17 17:02:57.340497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:57.340078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerDied","Data":"7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2"}
Jul 17 17:02:57.340497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:57.340159    2561 scope.go:117] "RemoveContainer" containerID="794550dd8bebb3c3aeb94c4c13f9a01bfb05944e91816f34ec71d221966c58e7"
Jul 17 17:02:57.340497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:57.340390    2561 scope.go:117] "RemoveContainer" containerID="7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2"
Jul 17 17:02:57.340628 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:02:57.340600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-7bpj9_odh-ai-gateway-infra(20c38b80-1b01-4ea4-9199-e9cd343a0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867"
Jul 17 17:02:58.346628 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:58.346602    2561 scope.go:117] "RemoveContainer" containerID="7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2"
Jul 17 17:02:58.347192 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:02:58.346828    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-7bpj9_odh-ai-gateway-infra(20c38b80-1b01-4ea4-9199-e9cd343a0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867"
Jul 17 17:02:58.676439 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:02:58.676346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:03:13.508590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:13.508560    2561 scope.go:117] "RemoveContainer" containerID="7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2"
Jul 17 17:03:14.423234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:14.423147    2561 generic.go:356] "Generic (PLEG): container finished" podID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerID="2d7c614b6df4efd256468ab9073d61c6de5a78eafeb904cdf7b04a8b3a879750" exitCode=7
Jul 17 17:03:14.423234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:14.423223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerDied","Data":"2d7c614b6df4efd256468ab9073d61c6de5a78eafeb904cdf7b04a8b3a879750"}
Jul 17 17:03:14.423426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:14.423319    2561 scope.go:117] "RemoveContainer" containerID="7c90372ca1b0b11d8ac39318bdd72c8bc20249315a9db3ce55a1d7e3ed58ccc2"
Jul 17 17:03:14.423560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:14.423545    2561 scope.go:117] "RemoveContainer" containerID="2d7c614b6df4efd256468ab9073d61c6de5a78eafeb904cdf7b04a8b3a879750"
Jul 17 17:03:14.423778 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:03:14.423761    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-7bpj9_odh-ai-gateway-infra(20c38b80-1b01-4ea4-9199-e9cd343a0867)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867"
Jul 17 17:03:15.453849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:15.450389    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:03:15.588980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:15.588951    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:03:15.739974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:15.739890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca\") pod \"20c38b80-1b01-4ea4-9199-e9cd343a0867\" (UID: \"20c38b80-1b01-4ea4-9199-e9cd343a0867\") "
Jul 17 17:03:15.740285 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:15.740262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20c38b80-1b01-4ea4-9199-e9cd343a0867" (UID: "20c38b80-1b01-4ea4-9199-e9cd343a0867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:03:15.840845 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:15.840817    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20c38b80-1b01-4ea4-9199-e9cd343a0867-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:03:16.434940 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.434905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9" event={"ID":"20c38b80-1b01-4ea4-9199-e9cd343a0867","Type":"ContainerDied","Data":"f0aa03bcebd3933802dee466345186f7c16c56a69d770935004719a2ca978064"}
Jul 17 17:03:16.435157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.434963    2561 scope.go:117] "RemoveContainer" containerID="2d7c614b6df4efd256468ab9073d61c6de5a78eafeb904cdf7b04a8b3a879750"
Jul 17 17:03:16.435157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.434918    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"
Jul 17 17:03:16.457227 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.457202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:03:16.462104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.462081    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-7bpj9"]
Jul 17 17:03:16.513489 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:16.513462    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" path="/var/lib/kubelet/pods/20c38b80-1b01-4ea4-9199-e9cd343a0867/volumes"
Jul 17 17:03:54.605400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605315    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605802    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605820    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605844    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605855    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605936    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.605970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.605949    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:03:54.608601 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.608581    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:03:54.610637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.610611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:03:54.610748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.610647    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:03:54.615078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.615038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:03:54.768782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.768743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qn4d4\" (UID: \"83f18784-6427-48f8-8b9b-6da8bf7caa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:03:54.869595 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.869506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qn4d4\" (UID: \"83f18784-6427-48f8-8b9b-6da8bf7caa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:03:54.870215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.870192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qn4d4\" (UID: \"83f18784-6427-48f8-8b9b-6da8bf7caa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:03:54.919552 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:54.919511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:03:55.047862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:55.047831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:03:55.051429 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:03:55.051393    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f18784_6427_48f8_8b9b_6da8bf7caa86.slice/crio-739348bfe86ad6785ecb9c1c47ec182d4eb32fdcfb08db1cf7c82cdf348ca119 WatchSource:0}: Error finding container 739348bfe86ad6785ecb9c1c47ec182d4eb32fdcfb08db1cf7c82cdf348ca119: Status 404 returned error can't find the container with id 739348bfe86ad6785ecb9c1c47ec182d4eb32fdcfb08db1cf7c82cdf348ca119
Jul 17 17:03:55.622108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:55.621619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerStarted","Data":"01bc00ae796e6ed8aab2ae68ea4e11e7bcd5df72eb60e7269925273f9e910957"}
Jul 17 17:03:55.622108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:55.621710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerStarted","Data":"739348bfe86ad6785ecb9c1c47ec182d4eb32fdcfb08db1cf7c82cdf348ca119"}
Jul 17 17:03:55.641685 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:55.641645    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" podStartSLOduration=1.641631085 podStartE2EDuration="1.641631085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:03:55.639886948 +0000 UTC m=+2269.624974488" watchObservedRunningTime="2026-07-17 17:03:55.641631085 +0000 UTC m=+2269.626718624"
Jul 17 17:03:56.627152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:56.627113    2561 generic.go:356] "Generic (PLEG): container finished" podID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerID="01bc00ae796e6ed8aab2ae68ea4e11e7bcd5df72eb60e7269925273f9e910957" exitCode=7
Jul 17 17:03:56.627553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:56.627165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerDied","Data":"01bc00ae796e6ed8aab2ae68ea4e11e7bcd5df72eb60e7269925273f9e910957"}
Jul 17 17:03:56.627553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:56.627350    2561 scope.go:117] "RemoveContainer" containerID="01bc00ae796e6ed8aab2ae68ea4e11e7bcd5df72eb60e7269925273f9e910957"
Jul 17 17:03:57.633478 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:57.633447    2561 generic.go:356] "Generic (PLEG): container finished" podID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerID="1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2" exitCode=7
Jul 17 17:03:57.633863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:57.633507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerDied","Data":"1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2"}
Jul 17 17:03:57.633863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:57.633570    2561 scope.go:117] "RemoveContainer" containerID="01bc00ae796e6ed8aab2ae68ea4e11e7bcd5df72eb60e7269925273f9e910957"
Jul 17 17:03:57.633863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:57.633764    2561 scope.go:117] "RemoveContainer" containerID="1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2"
Jul 17 17:03:57.633984 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:03:57.633943    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qn4d4_odh-ai-gateway-infra(83f18784-6427-48f8-8b9b-6da8bf7caa86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86"
Jul 17 17:03:58.639506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:03:58.639478    2561 scope.go:117] "RemoveContainer" containerID="1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2"
Jul 17 17:03:58.639878 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:03:58.639688    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qn4d4_odh-ai-gateway-infra(83f18784-6427-48f8-8b9b-6da8bf7caa86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86"
Jul 17 17:04:10.508260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:10.508222    2561 scope.go:117] "RemoveContainer" containerID="1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2"
Jul 17 17:04:11.696589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:11.696556    2561 generic.go:356] "Generic (PLEG): container finished" podID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerID="dd76074adfa12269bc361cc7d9799206d13b807bf3cfbff831a242f1ae027f4b" exitCode=7
Jul 17 17:04:11.697068 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:11.696626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerDied","Data":"dd76074adfa12269bc361cc7d9799206d13b807bf3cfbff831a242f1ae027f4b"}
Jul 17 17:04:11.697068 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:11.696687    2561 scope.go:117] "RemoveContainer" containerID="1d43db8912ca09818a2af8640d19ed55158b0691e72df893c78df0cfb952bea2"
Jul 17 17:04:11.697202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:11.697074    2561 scope.go:117] "RemoveContainer" containerID="dd76074adfa12269bc361cc7d9799206d13b807bf3cfbff831a242f1ae027f4b"
Jul 17 17:04:11.697307 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:11.697288    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qn4d4_odh-ai-gateway-infra(83f18784-6427-48f8-8b9b-6da8bf7caa86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86"
Jul 17 17:04:12.720420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:12.720384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:04:12.850281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:12.850258    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:04:12.908680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:12.908642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca\") pod \"83f18784-6427-48f8-8b9b-6da8bf7caa86\" (UID: \"83f18784-6427-48f8-8b9b-6da8bf7caa86\") "
Jul 17 17:04:12.908994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:12.908973    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83f18784-6427-48f8-8b9b-6da8bf7caa86" (UID: "83f18784-6427-48f8-8b9b-6da8bf7caa86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:04:13.009674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.009585    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f18784-6427-48f8-8b9b-6da8bf7caa86-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:04:13.708822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.708781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4" event={"ID":"83f18784-6427-48f8-8b9b-6da8bf7caa86","Type":"ContainerDied","Data":"739348bfe86ad6785ecb9c1c47ec182d4eb32fdcfb08db1cf7c82cdf348ca119"}
Jul 17 17:04:13.709008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.708842    2561 scope.go:117] "RemoveContainer" containerID="dd76074adfa12269bc361cc7d9799206d13b807bf3cfbff831a242f1ae027f4b"
Jul 17 17:04:13.709008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.708791    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"
Jul 17 17:04:13.730877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.730849    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:04:13.737354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:13.737322    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qn4d4"]
Jul 17 17:04:14.513800 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.513768    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" path="/var/lib/kubelet/pods/83f18784-6427-48f8-8b9b-6da8bf7caa86/volumes"
Jul 17 17:04:14.779591 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779559    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:14.779938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779923    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.779938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779937    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.780026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779950    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:04:14.780026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779956    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:04:14.780026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779963    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.780026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.779968    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.780026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.780023    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c38b80-1b01-4ea4-9199-e9cd343a0867" containerName="revoke-keys"
Jul 17 17:04:14.780193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.780033    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.780193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.780052    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.780193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.780062    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:14.784312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.784294    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:14.786484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.786464    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:04:14.786484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.786475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:04:14.789670 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.789651    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:14.823494 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.823469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb\" (UID: \"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:14.924345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.924312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb\" (UID: \"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:14.924934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:14.924913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb\" (UID: \"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:15.095371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:15.095289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:15.223437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:15.223408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:15.226641 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:04:15.226614    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ba8ae1_5499_4845_8ffb_cbb5dcb9f1a6.slice/crio-bbbb9a8c20981d7322966727350c83d994bdfd2941213fa5d0ca8ab58a4d2fde WatchSource:0}: Error finding container bbbb9a8c20981d7322966727350c83d994bdfd2941213fa5d0ca8ab58a4d2fde: Status 404 returned error can't find the container with id bbbb9a8c20981d7322966727350c83d994bdfd2941213fa5d0ca8ab58a4d2fde
Jul 17 17:04:15.719907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:15.719818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerStarted","Data":"336d0b54ac1a09e0164848ebca46386ecfe9c0701c7ce242eba334b6c693a438"}
Jul 17 17:04:15.719907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:15.719877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerStarted","Data":"bbbb9a8c20981d7322966727350c83d994bdfd2941213fa5d0ca8ab58a4d2fde"}
Jul 17 17:04:15.737389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:15.737342    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" podStartSLOduration=1.737328458 podStartE2EDuration="1.737328458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:04:15.735015304 +0000 UTC m=+2289.720102842" watchObservedRunningTime="2026-07-17 17:04:15.737328458 +0000 UTC m=+2289.722415995"
Jul 17 17:04:16.725077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:16.725028    2561 generic.go:356] "Generic (PLEG): container finished" podID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerID="336d0b54ac1a09e0164848ebca46386ecfe9c0701c7ce242eba334b6c693a438" exitCode=7
Jul 17 17:04:16.725077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:16.725072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerDied","Data":"336d0b54ac1a09e0164848ebca46386ecfe9c0701c7ce242eba334b6c693a438"}
Jul 17 17:04:16.725503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:16.725317    2561 scope.go:117] "RemoveContainer" containerID="336d0b54ac1a09e0164848ebca46386ecfe9c0701c7ce242eba334b6c693a438"
Jul 17 17:04:17.731626 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:17.731538    2561 generic.go:356] "Generic (PLEG): container finished" podID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerID="98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57" exitCode=7
Jul 17 17:04:17.732084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:17.731623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerDied","Data":"98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57"}
Jul 17 17:04:17.732084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:17.731677    2561 scope.go:117] "RemoveContainer" containerID="336d0b54ac1a09e0164848ebca46386ecfe9c0701c7ce242eba334b6c693a438"
Jul 17 17:04:17.732084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:17.731877    2561 scope.go:117] "RemoveContainer" containerID="98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57"
Jul 17 17:04:17.732254 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:17.732165    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb_odh-ai-gateway-infra(00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6"
Jul 17 17:04:18.738331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:18.738301    2561 scope.go:117] "RemoveContainer" containerID="98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57"
Jul 17 17:04:18.738724 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:18.738510    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb_odh-ai-gateway-infra(00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6"
Jul 17 17:04:32.508578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:32.508546    2561 scope.go:117] "RemoveContainer" containerID="98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57"
Jul 17 17:04:32.806104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:32.806069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerStarted","Data":"6d6bfb48bda6339c3937597317d56247334b76e837ad6587718f9a1b482d2d0e"}
Jul 17 17:04:33.812985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:33.812955    2561 generic.go:356] "Generic (PLEG): container finished" podID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerID="6d6bfb48bda6339c3937597317d56247334b76e837ad6587718f9a1b482d2d0e" exitCode=7
Jul 17 17:04:33.813393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:33.813028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerDied","Data":"6d6bfb48bda6339c3937597317d56247334b76e837ad6587718f9a1b482d2d0e"}
Jul 17 17:04:33.813393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:33.813102    2561 scope.go:117] "RemoveContainer" containerID="98afda84bfedb389cbba4ccd5002ad8fd5eecf95888d29ae621905476b720b57"
Jul 17 17:04:33.813492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:33.813422    2561 scope.go:117] "RemoveContainer" containerID="6d6bfb48bda6339c3937597317d56247334b76e837ad6587718f9a1b482d2d0e"
Jul 17 17:04:33.813630 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:33.813609    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb_odh-ai-gateway-infra(00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6"
Jul 17 17:04:33.853344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:33.853315    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:34.954719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:34.954688    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:35.103005 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.102914    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca\") pod \"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6\" (UID: \"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6\") "
Jul 17 17:04:35.103301 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.103279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" (UID: "00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:04:35.204513 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.204478    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:04:35.827117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.827086    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"
Jul 17 17:04:35.827117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.827100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb" event={"ID":"00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6","Type":"ContainerDied","Data":"bbbb9a8c20981d7322966727350c83d994bdfd2941213fa5d0ca8ab58a4d2fde"}
Jul 17 17:04:35.827366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.827159    2561 scope.go:117] "RemoveContainer" containerID="6d6bfb48bda6339c3937597317d56247334b76e837ad6587718f9a1b482d2d0e"
Jul 17 17:04:35.849720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.849679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:35.851757 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:35.851730    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-m7pkb"]
Jul 17 17:04:36.520579 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:36.520545    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" path="/var/lib/kubelet/pods/00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6/volumes"
Jul 17 17:04:44.644736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.644705    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645180    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645195    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f18784-6427-48f8-8b9b-6da8bf7caa86" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645214    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645220    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645225    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645233    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645238    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645296    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645306    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.645525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.645312    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ba8ae1-5499-4845-8ffb-cbb5dcb9f1a6" containerName="revoke-keys"
Jul 17 17:04:44.648207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.648191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:04:44.650491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.650466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:04:44.650632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.650492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:04:44.656026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.656005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:04:44.790930 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.790897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2gdks\" (UID: \"95875113-272d-437a-a422-827c7eb196f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:04:44.891511 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.891481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2gdks\" (UID: \"95875113-272d-437a-a422-827c7eb196f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:04:44.892110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.892091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2gdks\" (UID: \"95875113-272d-437a-a422-827c7eb196f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:04:44.962567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:44.962498    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:04:45.095083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:45.095058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:04:45.097651 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:04:45.097626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95875113_272d_437a_a422_827c7eb196f2.slice/crio-8ef0e87372e8fb721a9417013a6dc7e208d4bf4c079ecc158a6c636f552290df WatchSource:0}: Error finding container 8ef0e87372e8fb721a9417013a6dc7e208d4bf4c079ecc158a6c636f552290df: Status 404 returned error can't find the container with id 8ef0e87372e8fb721a9417013a6dc7e208d4bf4c079ecc158a6c636f552290df
Jul 17 17:04:45.875296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:45.875259    2561 generic.go:356] "Generic (PLEG): container finished" podID="95875113-272d-437a-a422-827c7eb196f2" containerID="ab9c89e983913d3ccb5747b350f5a8cb79a6f27317a02cf3c1bad1d0b327ba88" exitCode=7
Jul 17 17:04:45.875797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:45.875342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerDied","Data":"ab9c89e983913d3ccb5747b350f5a8cb79a6f27317a02cf3c1bad1d0b327ba88"}
Jul 17 17:04:45.875797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:45.875395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerStarted","Data":"8ef0e87372e8fb721a9417013a6dc7e208d4bf4c079ecc158a6c636f552290df"}
Jul 17 17:04:45.875797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:45.875550    2561 scope.go:117] "RemoveContainer" containerID="ab9c89e983913d3ccb5747b350f5a8cb79a6f27317a02cf3c1bad1d0b327ba88"
Jul 17 17:04:46.880944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:46.880888    2561 generic.go:356] "Generic (PLEG): container finished" podID="95875113-272d-437a-a422-827c7eb196f2" containerID="b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d" exitCode=7
Jul 17 17:04:46.880944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:46.880948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerDied","Data":"b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d"}
Jul 17 17:04:46.881464 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:46.880997    2561 scope.go:117] "RemoveContainer" containerID="ab9c89e983913d3ccb5747b350f5a8cb79a6f27317a02cf3c1bad1d0b327ba88"
Jul 17 17:04:46.881464 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:46.881233    2561 scope.go:117] "RemoveContainer" containerID="b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d"
Jul 17 17:04:46.881579 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:46.881481    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2gdks_odh-ai-gateway-infra(95875113-272d-437a-a422-827c7eb196f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" podUID="95875113-272d-437a-a422-827c7eb196f2"
Jul 17 17:04:47.886823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:04:47.886790    2561 scope.go:117] "RemoveContainer" containerID="b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d"
Jul 17 17:04:47.887220 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:04:47.886951    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2gdks_odh-ai-gateway-infra(95875113-272d-437a-a422-827c7eb196f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" podUID="95875113-272d-437a-a422-827c7eb196f2"
Jul 17 17:05:00.508596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:00.508515    2561 scope.go:117] "RemoveContainer" containerID="b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d"
Jul 17 17:05:00.951354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:00.951311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerStarted","Data":"9dedea49026e356af89e107ea5b6abf1d6f17322d23e4aa9cb4259408c91a2c6"}
Jul 17 17:05:00.967533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:00.967482    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" podStartSLOduration=16.967468437 podStartE2EDuration="16.967468437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:05:00.965983043 +0000 UTC m=+2334.951070598" watchObservedRunningTime="2026-07-17 17:05:00.967468437 +0000 UTC m=+2334.952555952"
Jul 17 17:05:01.957344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:01.957313    2561 generic.go:356] "Generic (PLEG): container finished" podID="95875113-272d-437a-a422-827c7eb196f2" containerID="9dedea49026e356af89e107ea5b6abf1d6f17322d23e4aa9cb4259408c91a2c6" exitCode=7
Jul 17 17:05:01.957797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:01.957367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerDied","Data":"9dedea49026e356af89e107ea5b6abf1d6f17322d23e4aa9cb4259408c91a2c6"}
Jul 17 17:05:01.957797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:01.957417    2561 scope.go:117] "RemoveContainer" containerID="b9dd68ec42e8f62ca9039cbe1f63035f112fda20acfb17a14785a8b7dc9ca66d"
Jul 17 17:05:01.957797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:01.957643    2561 scope.go:117] "RemoveContainer" containerID="9dedea49026e356af89e107ea5b6abf1d6f17322d23e4aa9cb4259408c91a2c6"
Jul 17 17:05:01.957965 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:01.957876    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2gdks_odh-ai-gateway-infra(95875113-272d-437a-a422-827c7eb196f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" podUID="95875113-272d-437a-a422-827c7eb196f2"
Jul 17 17:05:01.982442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:01.982409    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:05:03.096137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.096115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:05:03.140790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.140759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca\") pod \"95875113-272d-437a-a422-827c7eb196f2\" (UID: \"95875113-272d-437a-a422-827c7eb196f2\") "
Jul 17 17:05:03.141159 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.141137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95875113-272d-437a-a422-827c7eb196f2" (UID: "95875113-272d-437a-a422-827c7eb196f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:05:03.242196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.242090    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95875113-272d-437a-a422-827c7eb196f2-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:05:03.970935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.970899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks" event={"ID":"95875113-272d-437a-a422-827c7eb196f2","Type":"ContainerDied","Data":"8ef0e87372e8fb721a9417013a6dc7e208d4bf4c079ecc158a6c636f552290df"}
Jul 17 17:05:03.970935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.970923    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"
Jul 17 17:05:03.971283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.970963    2561 scope.go:117] "RemoveContainer" containerID="9dedea49026e356af89e107ea5b6abf1d6f17322d23e4aa9cb4259408c91a2c6"
Jul 17 17:05:03.994312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:03.994283    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:05:04.000813 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:04.000786    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2gdks"]
Jul 17 17:05:04.514594 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:04.514561    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95875113-272d-437a-a422-827c7eb196f2" path="/var/lib/kubelet/pods/95875113-272d-437a-a422-827c7eb196f2/volumes"
Jul 17 17:05:14.852841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.852804    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853170    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853182    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853202    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853209    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853266    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853275    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.853280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.853281    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:14.856124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.856108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:14.858470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.858442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:05:14.858596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.858556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:05:14.864308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.864273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:14.949630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:14.949587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk\" (UID: \"1dd925d5-bbee-4ec5-946d-a108ab161f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:15.050343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:15.050308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk\" (UID: \"1dd925d5-bbee-4ec5-946d-a108ab161f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:15.050959 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:15.050938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk\" (UID: \"1dd925d5-bbee-4ec5-946d-a108ab161f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:15.166984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:15.166896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:15.296890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:15.296855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:15.298976 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:05:15.298948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd925d5_bbee_4ec5_946d_a108ab161f5e.slice/crio-ff502b1b935a1bb9acdf6ac0c66ce80c649d7c8dc16c2278f4d7827193b0a695 WatchSource:0}: Error finding container ff502b1b935a1bb9acdf6ac0c66ce80c649d7c8dc16c2278f4d7827193b0a695: Status 404 returned error can't find the container with id ff502b1b935a1bb9acdf6ac0c66ce80c649d7c8dc16c2278f4d7827193b0a695
Jul 17 17:05:16.027349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:16.027323    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerID="73882f907bf57b61050ca2ce317e0bfafd2e7f3d35305249cf1918ab3538df39" exitCode=7
Jul 17 17:05:16.027661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:16.027400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerDied","Data":"73882f907bf57b61050ca2ce317e0bfafd2e7f3d35305249cf1918ab3538df39"}
Jul 17 17:05:16.027661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:16.027452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerStarted","Data":"ff502b1b935a1bb9acdf6ac0c66ce80c649d7c8dc16c2278f4d7827193b0a695"}
Jul 17 17:05:16.027661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:16.027580    2561 scope.go:117] "RemoveContainer" containerID="73882f907bf57b61050ca2ce317e0bfafd2e7f3d35305249cf1918ab3538df39"
Jul 17 17:05:17.034146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:17.034097    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerID="0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834" exitCode=7
Jul 17 17:05:17.034562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:17.034161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerDied","Data":"0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834"}
Jul 17 17:05:17.034562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:17.034221    2561 scope.go:117] "RemoveContainer" containerID="73882f907bf57b61050ca2ce317e0bfafd2e7f3d35305249cf1918ab3538df39"
Jul 17 17:05:17.034562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:17.034370    2561 scope.go:117] "RemoveContainer" containerID="0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834"
Jul 17 17:05:17.034689 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:17.034608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk_odh-ai-gateway-infra(1dd925d5-bbee-4ec5-946d-a108ab161f5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e"
Jul 17 17:05:18.040111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:18.040082    2561 scope.go:117] "RemoveContainer" containerID="0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834"
Jul 17 17:05:18.040489 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:18.040284    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk_odh-ai-gateway-infra(1dd925d5-bbee-4ec5-946d-a108ab161f5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e"
Jul 17 17:05:30.509164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:30.509128    2561 scope.go:117] "RemoveContainer" containerID="0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834"
Jul 17 17:05:31.099361 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:31.099323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerStarted","Data":"0041ef5e4b092e079246c4a0c8153ce5985686f61b1078f0d2f5076f7008bd3f"}
Jul 17 17:05:31.116355 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:31.116300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" podStartSLOduration=17.116284604 podStartE2EDuration="17.116284604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:05:31.114544447 +0000 UTC m=+2365.099631986" watchObservedRunningTime="2026-07-17 17:05:31.116284604 +0000 UTC m=+2365.101372141"
Jul 17 17:05:32.106685 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:32.106652    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerID="0041ef5e4b092e079246c4a0c8153ce5985686f61b1078f0d2f5076f7008bd3f" exitCode=7
Jul 17 17:05:32.107186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:32.106725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerDied","Data":"0041ef5e4b092e079246c4a0c8153ce5985686f61b1078f0d2f5076f7008bd3f"}
Jul 17 17:05:32.107186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:32.106789    2561 scope.go:117] "RemoveContainer" containerID="0686dc451dd2058dfe0ffa2a0ad05b03a888e946a6b3bbf75008937187235834"
Jul 17 17:05:32.107186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:32.107118    2561 scope.go:117] "RemoveContainer" containerID="0041ef5e4b092e079246c4a0c8153ce5985686f61b1078f0d2f5076f7008bd3f"
Jul 17 17:05:32.107394 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:32.107373    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk_odh-ai-gateway-infra(1dd925d5-bbee-4ec5-946d-a108ab161f5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e"
Jul 17 17:05:32.123562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:32.123538    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:33.250933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:33.250902    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:33.414834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:33.414746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca\") pod \"1dd925d5-bbee-4ec5-946d-a108ab161f5e\" (UID: \"1dd925d5-bbee-4ec5-946d-a108ab161f5e\") "
Jul 17 17:05:33.415127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:33.415104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dd925d5-bbee-4ec5-946d-a108ab161f5e" (UID: "1dd925d5-bbee-4ec5-946d-a108ab161f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:05:33.516114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:33.516086    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd925d5-bbee-4ec5-946d-a108ab161f5e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:05:34.118490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.118454    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"
Jul 17 17:05:34.118490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.118478    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk" event={"ID":"1dd925d5-bbee-4ec5-946d-a108ab161f5e","Type":"ContainerDied","Data":"ff502b1b935a1bb9acdf6ac0c66ce80c649d7c8dc16c2278f4d7827193b0a695"}
Jul 17 17:05:34.118759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.118577    2561 scope.go:117] "RemoveContainer" containerID="0041ef5e4b092e079246c4a0c8153ce5985686f61b1078f0d2f5076f7008bd3f"
Jul 17 17:05:34.141853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.141826    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:34.147802 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.147778    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-d7xvk"]
Jul 17 17:05:34.514133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.514037    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" path="/var/lib/kubelet/pods/1dd925d5-bbee-4ec5-946d-a108ab161f5e/volumes"
Jul 17 17:05:34.722756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.722722    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:34.723166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723148    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:34.723166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723168    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95875113-272d-437a-a422-827c7eb196f2" containerName="revoke-keys"
Jul 17 17:05:34.723331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723185    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.723331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723194    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.723331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723217    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.723331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723226    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.723331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723324    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.723570 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.723338    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:05:34.727739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.727718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:34.730203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.730187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:05:34.730294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.730274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:05:34.737699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.737674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:34.824989 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.824949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-98wxc\" (UID: \"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:34.925960 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.925921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-98wxc\" (UID: \"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:34.926601 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:34.926582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-98wxc\" (UID: \"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:35.039379 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:35.039338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:35.169426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:35.169401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:35.171478 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:05:35.171444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d7be0e_f5b9_44a3_8c3c_bbe07bc132bd.slice/crio-c45c24cad46ed56e1f7caa8755f2f095cadac210f587bf76dc57ce5559ee5b30 WatchSource:0}: Error finding container c45c24cad46ed56e1f7caa8755f2f095cadac210f587bf76dc57ce5559ee5b30: Status 404 returned error can't find the container with id c45c24cad46ed56e1f7caa8755f2f095cadac210f587bf76dc57ce5559ee5b30
Jul 17 17:05:36.131188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:36.131093    2561 generic.go:356] "Generic (PLEG): container finished" podID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerID="b02a3aafa82ca65642965336467e24460c2fda47c34e6d3cf9fd812040fbbf12" exitCode=7
Jul 17 17:05:36.131623 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:36.131185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" event={"ID":"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd","Type":"ContainerDied","Data":"b02a3aafa82ca65642965336467e24460c2fda47c34e6d3cf9fd812040fbbf12"}
Jul 17 17:05:36.131623 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:36.131240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" event={"ID":"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd","Type":"ContainerStarted","Data":"c45c24cad46ed56e1f7caa8755f2f095cadac210f587bf76dc57ce5559ee5b30"}
Jul 17 17:05:36.131623 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:36.131359    2561 scope.go:117] "RemoveContainer" containerID="b02a3aafa82ca65642965336467e24460c2fda47c34e6d3cf9fd812040fbbf12"
Jul 17 17:05:37.137623 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:37.137587    2561 generic.go:356] "Generic (PLEG): container finished" podID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerID="8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d" exitCode=7
Jul 17 17:05:37.138083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:37.137672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" event={"ID":"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd","Type":"ContainerDied","Data":"8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d"}
Jul 17 17:05:37.138083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:37.137743    2561 scope.go:117] "RemoveContainer" containerID="b02a3aafa82ca65642965336467e24460c2fda47c34e6d3cf9fd812040fbbf12"
Jul 17 17:05:37.138083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:37.137867    2561 scope.go:117] "RemoveContainer" containerID="8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d"
Jul 17 17:05:37.138212 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:37.138104    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-98wxc_odh-ai-gateway-infra(62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd"
Jul 17 17:05:38.144025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:38.143996    2561 scope.go:117] "RemoveContainer" containerID="8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d"
Jul 17 17:05:38.144487 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:38.144244    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-98wxc_odh-ai-gateway-infra(62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd"
Jul 17 17:05:50.508544 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:50.508512    2561 scope.go:117] "RemoveContainer" containerID="8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d"
Jul 17 17:05:51.202911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:51.202887    2561 generic.go:356] "Generic (PLEG): container finished" podID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerID="36aeb7e874ffb023024d67aa071691e6eac751f5905e8b4774ccdf402fec57a1" exitCode=7
Jul 17 17:05:51.203028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:51.202961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" event={"ID":"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd","Type":"ContainerDied","Data":"36aeb7e874ffb023024d67aa071691e6eac751f5905e8b4774ccdf402fec57a1"}
Jul 17 17:05:51.203028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:51.203020    2561 scope.go:117] "RemoveContainer" containerID="8e0c28e61c7e9610c6ad9c625d6be979c02be71c5030f508d35a11aa0595b04d"
Jul 17 17:05:51.203289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:51.203268    2561 scope.go:117] "RemoveContainer" containerID="36aeb7e874ffb023024d67aa071691e6eac751f5905e8b4774ccdf402fec57a1"
Jul 17 17:05:51.203482 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:05:51.203461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-98wxc_odh-ai-gateway-infra(62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd"
Jul 17 17:05:52.230117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:52.230083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:52.361463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:52.361442    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:52.378909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:52.378885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca\") pod \"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd\" (UID: \"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd\") "
Jul 17 17:05:52.379293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:52.379271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" (UID: "62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:05:52.479840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:52.479800    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:05:53.214255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:53.214224    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"
Jul 17 17:05:53.214255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:53.214229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc" event={"ID":"62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd","Type":"ContainerDied","Data":"c45c24cad46ed56e1f7caa8755f2f095cadac210f587bf76dc57ce5559ee5b30"}
Jul 17 17:05:53.214506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:53.214284    2561 scope.go:117] "RemoveContainer" containerID="36aeb7e874ffb023024d67aa071691e6eac751f5905e8b4774ccdf402fec57a1"
Jul 17 17:05:53.230712 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:53.230689    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:53.236867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:53.236847    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-98wxc"]
Jul 17 17:05:54.513459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:05:54.513423    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" path="/var/lib/kubelet/pods/62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd/volumes"
Jul 17 17:06:04.895969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.895933    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896286    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896297    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896312    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896329    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896335    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896340    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896398    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd925d5-bbee-4ec5-946d-a108ab161f5e" containerName="revoke-keys"
Jul 17 17:06:04.896633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896407    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896413    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.896633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.896419    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d7be0e-f5b9-44a3-8c3c-bbe07bc132bd" containerName="revoke-keys"
Jul 17 17:06:04.900711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.900696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:04.903336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.903310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:06:04.903448 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.903317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:06:04.907112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.907081    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:04.995490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:04.995461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk\" (UID: \"7a54ee0d-acb8-4bf6-8c70-4b95786a2359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:05.096182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:05.096153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk\" (UID: \"7a54ee0d-acb8-4bf6-8c70-4b95786a2359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:05.096751 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:05.096731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk\" (UID: \"7a54ee0d-acb8-4bf6-8c70-4b95786a2359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:05.212366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:05.212303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:05.340057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:05.340015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:05.341627 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:06:05.341596    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a54ee0d_acb8_4bf6_8c70_4b95786a2359.slice/crio-b9de1aa6c8e0d7a3bb35250786f6fb9f7c277256a84b794186267183456c574b WatchSource:0}: Error finding container b9de1aa6c8e0d7a3bb35250786f6fb9f7c277256a84b794186267183456c574b: Status 404 returned error can't find the container with id b9de1aa6c8e0d7a3bb35250786f6fb9f7c277256a84b794186267183456c574b
Jul 17 17:06:06.279177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.279150    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerID="44d19c0932956d4e45f802f34d2469f0c63fdf78a04545af79c7e1e121727872" exitCode=7
Jul 17 17:06:06.279539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.279209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" event={"ID":"7a54ee0d-acb8-4bf6-8c70-4b95786a2359","Type":"ContainerDied","Data":"44d19c0932956d4e45f802f34d2469f0c63fdf78a04545af79c7e1e121727872"}
Jul 17 17:06:06.279539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.279246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" event={"ID":"7a54ee0d-acb8-4bf6-8c70-4b95786a2359","Type":"ContainerStarted","Data":"b9de1aa6c8e0d7a3bb35250786f6fb9f7c277256a84b794186267183456c574b"}
Jul 17 17:06:06.279539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.279389    2561 scope.go:117] "RemoveContainer" containerID="44d19c0932956d4e45f802f34d2469f0c63fdf78a04545af79c7e1e121727872"
Jul 17 17:06:06.535773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.535713    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:06:06.542638 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:06.542619    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:06:07.285078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:07.285030    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerID="d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054" exitCode=7
Jul 17 17:06:07.285537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:07.285116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" event={"ID":"7a54ee0d-acb8-4bf6-8c70-4b95786a2359","Type":"ContainerDied","Data":"d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054"}
Jul 17 17:06:07.285537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:07.285174    2561 scope.go:117] "RemoveContainer" containerID="44d19c0932956d4e45f802f34d2469f0c63fdf78a04545af79c7e1e121727872"
Jul 17 17:06:07.285537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:07.285396    2561 scope.go:117] "RemoveContainer" containerID="d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054"
Jul 17 17:06:07.285703 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:07.285627    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk_odh-ai-gateway-infra(7a54ee0d-acb8-4bf6-8c70-4b95786a2359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359"
Jul 17 17:06:08.293396 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:08.293368    2561 scope.go:117] "RemoveContainer" containerID="d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054"
Jul 17 17:06:08.293775 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:08.293550    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk_odh-ai-gateway-infra(7a54ee0d-acb8-4bf6-8c70-4b95786a2359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359"
Jul 17 17:06:18.895097 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:18.895040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:06:19.937632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:19.937532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:06:21.509012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:21.508981    2561 scope.go:117] "RemoveContainer" containerID="d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054"
Jul 17 17:06:22.361460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:22.361432    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerID="e77d6457e9aad45e9a57a5b40375999a439179afcfc53786bb953a41f21cd21e" exitCode=7
Jul 17 17:06:22.361653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:22.361490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" event={"ID":"7a54ee0d-acb8-4bf6-8c70-4b95786a2359","Type":"ContainerDied","Data":"e77d6457e9aad45e9a57a5b40375999a439179afcfc53786bb953a41f21cd21e"}
Jul 17 17:06:22.361653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:22.361549    2561 scope.go:117] "RemoveContainer" containerID="d2ec0d61df78536fb801bd9c55069affb9d59f9697f99642812b60e097b55054"
Jul 17 17:06:22.361812 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:22.361791    2561 scope.go:117] "RemoveContainer" containerID="e77d6457e9aad45e9a57a5b40375999a439179afcfc53786bb953a41f21cd21e"
Jul 17 17:06:22.362000 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:22.361977    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk_odh-ai-gateway-infra(7a54ee0d-acb8-4bf6-8c70-4b95786a2359)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359"
Jul 17 17:06:23.388315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:23.388282    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:23.515707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:23.515684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:23.661540 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:23.661479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca\") pod \"7a54ee0d-acb8-4bf6-8c70-4b95786a2359\" (UID: \"7a54ee0d-acb8-4bf6-8c70-4b95786a2359\") "
Jul 17 17:06:23.661834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:23.661799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a54ee0d-acb8-4bf6-8c70-4b95786a2359" (UID: "7a54ee0d-acb8-4bf6-8c70-4b95786a2359"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:06:23.762700 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:23.762672    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54ee0d-acb8-4bf6-8c70-4b95786a2359-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:06:24.372036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.372004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk" event={"ID":"7a54ee0d-acb8-4bf6-8c70-4b95786a2359","Type":"ContainerDied","Data":"b9de1aa6c8e0d7a3bb35250786f6fb9f7c277256a84b794186267183456c574b"}
Jul 17 17:06:24.372218 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.372037    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"
Jul 17 17:06:24.372218 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.372103    2561 scope.go:117] "RemoveContainer" containerID="e77d6457e9aad45e9a57a5b40375999a439179afcfc53786bb953a41f21cd21e"
Jul 17 17:06:24.393003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.392981    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:24.395772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.395747    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-7xzrk"]
Jul 17 17:06:24.513356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.513332    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" path="/var/lib/kubelet/pods/7a54ee0d-acb8-4bf6-8c70-4b95786a2359/volumes"
Jul 17 17:06:24.752114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752038    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:24.752380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752369    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752382    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752394    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752399    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752416    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752421    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752472    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.752596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.752487    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a54ee0d-acb8-4bf6-8c70-4b95786a2359" containerName="revoke-keys"
Jul 17 17:06:24.756721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.756703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:24.759075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.759039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:06:24.759183 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.759101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:06:24.762804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.762680    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:24.870980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.870944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-jrgbm\" (UID: \"a49f811c-8697-408c-a971-41763a184136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:24.972325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.972290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-jrgbm\" (UID: \"a49f811c-8697-408c-a971-41763a184136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:24.972913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:24.972894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-jrgbm\" (UID: \"a49f811c-8697-408c-a971-41763a184136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:25.068452 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:25.068426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:25.195781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:25.195750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:25.197692 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:06:25.197667    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49f811c_8697_408c_a971_41763a184136.slice/crio-712d993733034523625e632a5031295d09c6f65e4f00afd8fa924df04429869f WatchSource:0}: Error finding container 712d993733034523625e632a5031295d09c6f65e4f00afd8fa924df04429869f: Status 404 returned error can't find the container with id 712d993733034523625e632a5031295d09c6f65e4f00afd8fa924df04429869f
Jul 17 17:06:25.378413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:25.378328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" event={"ID":"a49f811c-8697-408c-a971-41763a184136","Type":"ContainerStarted","Data":"712d993733034523625e632a5031295d09c6f65e4f00afd8fa924df04429869f"}
Jul 17 17:06:25.461241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:25.461208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:06:25.564707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:25.564679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:06:26.384235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:26.384203    2561 generic.go:356] "Generic (PLEG): container finished" podID="a49f811c-8697-408c-a971-41763a184136" containerID="f8483a17db5dd1fe9115921855366f43962c991e7d6c8e1f2f9e384439b2c54a" exitCode=7
Jul 17 17:06:26.384410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:26.384287    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" event={"ID":"a49f811c-8697-408c-a971-41763a184136","Type":"ContainerDied","Data":"f8483a17db5dd1fe9115921855366f43962c991e7d6c8e1f2f9e384439b2c54a"}
Jul 17 17:06:26.384468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:26.384452    2561 scope.go:117] "RemoveContainer" containerID="f8483a17db5dd1fe9115921855366f43962c991e7d6c8e1f2f9e384439b2c54a"
Jul 17 17:06:27.391353 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:27.391321    2561 generic.go:356] "Generic (PLEG): container finished" podID="a49f811c-8697-408c-a971-41763a184136" containerID="f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd" exitCode=7
Jul 17 17:06:27.391844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:27.391406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" event={"ID":"a49f811c-8697-408c-a971-41763a184136","Type":"ContainerDied","Data":"f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd"}
Jul 17 17:06:27.391844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:27.391462    2561 scope.go:117] "RemoveContainer" containerID="f8483a17db5dd1fe9115921855366f43962c991e7d6c8e1f2f9e384439b2c54a"
Jul 17 17:06:27.391844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:27.391597    2561 scope.go:117] "RemoveContainer" containerID="f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd"
Jul 17 17:06:27.391985 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:27.391841    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-jrgbm_odh-ai-gateway-infra(a49f811c-8697-408c-a971-41763a184136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" podUID="a49f811c-8697-408c-a971-41763a184136"
Jul 17 17:06:28.397321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:28.397255    2561 scope.go:117] "RemoveContainer" containerID="f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd"
Jul 17 17:06:28.397659 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:28.397457    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-jrgbm_odh-ai-gateway-infra(a49f811c-8697-408c-a971-41763a184136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" podUID="a49f811c-8697-408c-a971-41763a184136"
Jul 17 17:06:42.508569 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:42.508538    2561 scope.go:117] "RemoveContainer" containerID="f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd"
Jul 17 17:06:43.465818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:43.465741    2561 generic.go:356] "Generic (PLEG): container finished" podID="a49f811c-8697-408c-a971-41763a184136" containerID="6ecaeafea50b245bc34d28e8bc7661bb0ee23befe4d130c5d10891824d1baf76" exitCode=7
Jul 17 17:06:43.465818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:43.465784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" event={"ID":"a49f811c-8697-408c-a971-41763a184136","Type":"ContainerDied","Data":"6ecaeafea50b245bc34d28e8bc7661bb0ee23befe4d130c5d10891824d1baf76"}
Jul 17 17:06:43.465995 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:43.465861    2561 scope.go:117] "RemoveContainer" containerID="f946cca135a33bf0594821f420406f2395d6abcfd021df95985ecb3ec0308bbd"
Jul 17 17:06:43.466194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:43.466168    2561 scope.go:117] "RemoveContainer" containerID="6ecaeafea50b245bc34d28e8bc7661bb0ee23befe4d130c5d10891824d1baf76"
Jul 17 17:06:43.466390 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:43.466367    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-jrgbm_odh-ai-gateway-infra(a49f811c-8697-408c-a971-41763a184136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" podUID="a49f811c-8697-408c-a971-41763a184136"
Jul 17 17:06:44.493711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:44.493679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:44.633215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:44.633190    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:44.744897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:44.744816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca\") pod \"a49f811c-8697-408c-a971-41763a184136\" (UID: \"a49f811c-8697-408c-a971-41763a184136\") "
Jul 17 17:06:44.745226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:44.745205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a49f811c-8697-408c-a971-41763a184136" (UID: "a49f811c-8697-408c-a971-41763a184136"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:06:44.845679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:44.845648    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f811c-8697-408c-a971-41763a184136-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:06:45.479401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:45.479366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm" event={"ID":"a49f811c-8697-408c-a971-41763a184136","Type":"ContainerDied","Data":"712d993733034523625e632a5031295d09c6f65e4f00afd8fa924df04429869f"}
Jul 17 17:06:45.479401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:45.479386    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"
Jul 17 17:06:45.479673 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:45.479428    2561 scope.go:117] "RemoveContainer" containerID="6ecaeafea50b245bc34d28e8bc7661bb0ee23befe4d130c5d10891824d1baf76"
Jul 17 17:06:45.501193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:45.501168    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:45.508972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:45.508929    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-jrgbm"]
Jul 17 17:06:46.513506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:46.513472    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49f811c-8697-408c-a971-41763a184136" path="/var/lib/kubelet/pods/a49f811c-8697-408c-a971-41763a184136/volumes"
Jul 17 17:06:54.939934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.939900    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940243    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940254    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940281    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940286    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940295    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940301    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940347    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.940407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.940357    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:06:54.944721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.944705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:06:54.947140 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.947110    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:06:54.947269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.947155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:06:54.951002 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:54.950961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:06:55.036431 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.036394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf\" (UID: \"91fa9416-5833-47e6-ab1f-cfa6a6b91e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:06:55.137860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.137820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf\" (UID: \"91fa9416-5833-47e6-ab1f-cfa6a6b91e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:06:55.138470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.138452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf\" (UID: \"91fa9416-5833-47e6-ab1f-cfa6a6b91e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:06:55.255984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.255901    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:06:55.388231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.388198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:06:55.392258 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:06:55.392233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fa9416_5833_47e6_ab1f_cfa6a6b91e00.slice/crio-e01d4fa99173046b63838ec9508c81735cf290990da3df002ebd73cd1af4b804 WatchSource:0}: Error finding container e01d4fa99173046b63838ec9508c81735cf290990da3df002ebd73cd1af4b804: Status 404 returned error can't find the container with id e01d4fa99173046b63838ec9508c81735cf290990da3df002ebd73cd1af4b804
Jul 17 17:06:55.526830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:55.526801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" event={"ID":"91fa9416-5833-47e6-ab1f-cfa6a6b91e00","Type":"ContainerStarted","Data":"e01d4fa99173046b63838ec9508c81735cf290990da3df002ebd73cd1af4b804"}
Jul 17 17:06:56.533232 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:56.533198    2561 generic.go:356] "Generic (PLEG): container finished" podID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerID="133951bfd7530eb4f46a4af54d05386a8e87982dbbcb52911e046cb59460b6d2" exitCode=7
Jul 17 17:06:56.533628 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:56.533247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" event={"ID":"91fa9416-5833-47e6-ab1f-cfa6a6b91e00","Type":"ContainerDied","Data":"133951bfd7530eb4f46a4af54d05386a8e87982dbbcb52911e046cb59460b6d2"}
Jul 17 17:06:56.533628 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:56.533456    2561 scope.go:117] "RemoveContainer" containerID="133951bfd7530eb4f46a4af54d05386a8e87982dbbcb52911e046cb59460b6d2"
Jul 17 17:06:57.539880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:57.539846    2561 generic.go:356] "Generic (PLEG): container finished" podID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerID="019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d" exitCode=7
Jul 17 17:06:57.540329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:57.539906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" event={"ID":"91fa9416-5833-47e6-ab1f-cfa6a6b91e00","Type":"ContainerDied","Data":"019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d"}
Jul 17 17:06:57.540329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:57.539951    2561 scope.go:117] "RemoveContainer" containerID="133951bfd7530eb4f46a4af54d05386a8e87982dbbcb52911e046cb59460b6d2"
Jul 17 17:06:57.540329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:57.540166    2561 scope.go:117] "RemoveContainer" containerID="019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d"
Jul 17 17:06:57.540456 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:57.540380    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf_odh-ai-gateway-infra(91fa9416-5833-47e6-ab1f-cfa6a6b91e00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00"
Jul 17 17:06:58.546506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:06:58.546477    2561 scope.go:117] "RemoveContainer" containerID="019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d"
Jul 17 17:06:58.546884 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:06:58.546691    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf_odh-ai-gateway-infra(91fa9416-5833-47e6-ab1f-cfa6a6b91e00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00"
Jul 17 17:07:09.508507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:09.508470    2561 scope.go:117] "RemoveContainer" containerID="019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d"
Jul 17 17:07:10.602575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:10.602540    2561 generic.go:356] "Generic (PLEG): container finished" podID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerID="d3daed5f95f55217bed27683cd931a9a855f04373bbd1ff1252e87fdd62d35d9" exitCode=7
Jul 17 17:07:10.603075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:10.602596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" event={"ID":"91fa9416-5833-47e6-ab1f-cfa6a6b91e00","Type":"ContainerDied","Data":"d3daed5f95f55217bed27683cd931a9a855f04373bbd1ff1252e87fdd62d35d9"}
Jul 17 17:07:10.603075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:10.602656    2561 scope.go:117] "RemoveContainer" containerID="019a374a22d091efb38a3864c25ae41d1ae54d40130140cee54debb9fca9995d"
Jul 17 17:07:10.603075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:10.602957    2561 scope.go:117] "RemoveContainer" containerID="d3daed5f95f55217bed27683cd931a9a855f04373bbd1ff1252e87fdd62d35d9"
Jul 17 17:07:10.603282 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:10.603201    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf_odh-ai-gateway-infra(91fa9416-5833-47e6-ab1f-cfa6a6b91e00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00"
Jul 17 17:07:11.626600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:11.626565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:07:11.759208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:11.759181    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:07:11.782492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:11.782459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca\") pod \"91fa9416-5833-47e6-ab1f-cfa6a6b91e00\" (UID: \"91fa9416-5833-47e6-ab1f-cfa6a6b91e00\") "
Jul 17 17:07:11.782791 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:11.782760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91fa9416-5833-47e6-ab1f-cfa6a6b91e00" (UID: "91fa9416-5833-47e6-ab1f-cfa6a6b91e00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:07:11.883774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:11.883698    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91fa9416-5833-47e6-ab1f-cfa6a6b91e00-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:07:12.614907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:12.614864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf" event={"ID":"91fa9416-5833-47e6-ab1f-cfa6a6b91e00","Type":"ContainerDied","Data":"e01d4fa99173046b63838ec9508c81735cf290990da3df002ebd73cd1af4b804"}
Jul 17 17:07:12.615116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:12.614923    2561 scope.go:117] "RemoveContainer" containerID="d3daed5f95f55217bed27683cd931a9a855f04373bbd1ff1252e87fdd62d35d9"
Jul 17 17:07:12.615116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:12.614882    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"
Jul 17 17:07:12.630847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:12.630822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:07:12.637731 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:12.637706    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-zsbtf"]
Jul 17 17:07:14.514900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:14.514864    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" path="/var/lib/kubelet/pods/91fa9416-5833-47e6-ab1f-cfa6a6b91e00/volumes"
Jul 17 17:07:14.941436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:14.941382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:15.142603 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:15.142554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:16.264984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.264945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:16.365039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.365000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:16.590759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.590727    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"]
Jul 17 17:07:16.591105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591107    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591125    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591190    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49f811c-8697-408c-a971-41763a184136" containerName="revoke-keys"
Jul 17 17:07:16.591260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591201    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591208    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.591260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.591214    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:16.594060 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.594022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.596581 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.596553    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 17:07:16.596710 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.596658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4cb0f3bd-serving-cert\""
Jul 17 17:07:16.597432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.597415    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cmvsl\""
Jul 17 17:07:16.597585 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.597445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 17:07:16.605741 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.605712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"]
Jul 17 17:07:16.625089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.625036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.625300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.625180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7249256d-4460-47ac-bb2e-cf16bf5030bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.625300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.625224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjxr\" (UniqueName: \"kubernetes.io/projected/7249256d-4460-47ac-bb2e-cf16bf5030bb-kube-api-access-6pjxr\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.630894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.630865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:16.725895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.725859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.726101 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.725899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7249256d-4460-47ac-bb2e-cf16bf5030bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.726101 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.725928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjxr\" (UniqueName: \"kubernetes.io/projected/7249256d-4460-47ac-bb2e-cf16bf5030bb-kube-api-access-6pjxr\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.726101 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:16.726009    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-serving-cert: secret "maas-api-e2e-ait-4cb0f3bd-serving-cert" not found
Jul 17 17:07:16.726101 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:16.726099    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls podName:7249256d-4460-47ac-bb2e-cf16bf5030bb nodeName:}" failed. No retries permitted until 2026-07-17 17:07:17.226080053 +0000 UTC m=+2471.211167574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls") pod "maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" (UID: "7249256d-4460-47ac-bb2e-cf16bf5030bb") : secret "maas-api-e2e-ait-4cb0f3bd-serving-cert" not found
Jul 17 17:07:16.726767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.726740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7249256d-4460-47ac-bb2e-cf16bf5030bb-openshift-service-ca\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.736716 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.736691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjxr\" (UniqueName: \"kubernetes.io/projected/7249256d-4460-47ac-bb2e-cf16bf5030bb-kube-api-access-6pjxr\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:16.873681 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:16.873602    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:17.108816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.108781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:17.210028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.209936    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:17.231164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.231131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:17.233584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.233562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7249256d-4460-47ac-bb2e-cf16bf5030bb-maas-api-tls\") pod \"maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s\" (UID: \"7249256d-4460-47ac-bb2e-cf16bf5030bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:17.507745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.507648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:17.645277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.645251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"]
Jul 17 17:07:17.647754 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:07:17.647725    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7249256d_4460_47ac_bb2e_cf16bf5030bb.slice/crio-3e6b01996be9bf249b78ba96d027bfef6a1ad6b2bfeb79c5a82422af4bd0def7 WatchSource:0}: Error finding container 3e6b01996be9bf249b78ba96d027bfef6a1ad6b2bfeb79c5a82422af4bd0def7: Status 404 returned error can't find the container with id 3e6b01996be9bf249b78ba96d027bfef6a1ad6b2bfeb79c5a82422af4bd0def7
Jul 17 17:07:17.649110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:17.649090    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:07:18.646251 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:18.646202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerStarted","Data":"3e6b01996be9bf249b78ba96d027bfef6a1ad6b2bfeb79c5a82422af4bd0def7"}
Jul 17 17:07:18.959847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:18.959761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:07:20.657829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:20.657797    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="36e202aaf121eb355b2b67981024a2909db626e09b5c40a08324eef5017fde2d" exitCode=1
Jul 17 17:07:20.658251 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:20.657871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"36e202aaf121eb355b2b67981024a2909db626e09b5c40a08324eef5017fde2d"}
Jul 17 17:07:20.658251 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:20.658105    2561 scope.go:117] "RemoveContainer" containerID="36e202aaf121eb355b2b67981024a2909db626e09b5c40a08324eef5017fde2d"
Jul 17 17:07:21.664016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:21.663918    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f" exitCode=1
Jul 17 17:07:21.664016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:21.664000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"}
Jul 17 17:07:21.664477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:21.664105    2561 scope.go:117] "RemoveContainer" containerID="36e202aaf121eb355b2b67981024a2909db626e09b5c40a08324eef5017fde2d"
Jul 17 17:07:21.664477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:21.664358    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:21.664611 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:21.664588    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:22.514548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:22.514518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:22.670528 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:22.670491    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:22.670900 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:22.670673    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:23.675157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:23.675130    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:23.675535 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:23.675343    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:24.800925 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.800873    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:24.801382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.801248    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:24.801382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.801261    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fa9416-5833-47e6-ab1f-cfa6a6b91e00" containerName="revoke-keys"
Jul 17 17:07:24.804318 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.804300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:24.806500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.806478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:07:24.810778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.810757    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:24.907343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:24.907303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-f2gj6\" (UID: \"ffe7c853-3432-4c64-942d-729c453d32d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:25.008718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.008682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-f2gj6\" (UID: \"ffe7c853-3432-4c64-942d-729c453d32d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:25.009315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.009295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-f2gj6\" (UID: \"ffe7c853-3432-4c64-942d-729c453d32d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:25.116100 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.115991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:25.251463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.251430    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:25.254656 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:07:25.254626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe7c853_3432_4c64_942d_729c453d32d1.slice/crio-8ea226ead75cccabbdb9b46f9cff2723a08b0b4b101aa9f02b4c1e1082fe938d WatchSource:0}: Error finding container 8ea226ead75cccabbdb9b46f9cff2723a08b0b4b101aa9f02b4c1e1082fe938d: Status 404 returned error can't find the container with id 8ea226ead75cccabbdb9b46f9cff2723a08b0b4b101aa9f02b4c1e1082fe938d
Jul 17 17:07:25.685139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.685031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerStarted","Data":"ad1a06f6a0fe7a1f05abe030f042e74c14718c879d3375e0820ef82fa7ae8229"}
Jul 17 17:07:25.685139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.685108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerStarted","Data":"8ea226ead75cccabbdb9b46f9cff2723a08b0b4b101aa9f02b4c1e1082fe938d"}
Jul 17 17:07:25.705767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:25.705715    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" podStartSLOduration=1.705697534 podStartE2EDuration="1.705697534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:07:25.703767487 +0000 UTC m=+2479.688855025" watchObservedRunningTime="2026-07-17 17:07:25.705697534 +0000 UTC m=+2479.690785073"
Jul 17 17:07:26.691805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:26.691771    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffe7c853-3432-4c64-942d-729c453d32d1" containerID="ad1a06f6a0fe7a1f05abe030f042e74c14718c879d3375e0820ef82fa7ae8229" exitCode=7
Jul 17 17:07:26.692219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:26.691820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerDied","Data":"ad1a06f6a0fe7a1f05abe030f042e74c14718c879d3375e0820ef82fa7ae8229"}
Jul 17 17:07:26.692219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:26.692110    2561 scope.go:117] "RemoveContainer" containerID="ad1a06f6a0fe7a1f05abe030f042e74c14718c879d3375e0820ef82fa7ae8229"
Jul 17 17:07:27.507794 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.507752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:27.508270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.508252    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:27.508472 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:27.508450    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:27.698072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.697955    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffe7c853-3432-4c64-942d-729c453d32d1" containerID="142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2" exitCode=7
Jul 17 17:07:27.698072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.698015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerDied","Data":"142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2"}
Jul 17 17:07:27.698550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.698088    2561 scope.go:117] "RemoveContainer" containerID="ad1a06f6a0fe7a1f05abe030f042e74c14718c879d3375e0820ef82fa7ae8229"
Jul 17 17:07:27.698550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:27.698284    2561 scope.go:117] "RemoveContainer" containerID="142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2"
Jul 17 17:07:27.698628 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:27.698549    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-f2gj6_odh-ai-gateway-infra(ffe7c853-3432-4c64-942d-729c453d32d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" podUID="ffe7c853-3432-4c64-942d-729c453d32d1"
Jul 17 17:07:28.703758 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:28.703731    2561 scope.go:117] "RemoveContainer" containerID="142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2"
Jul 17 17:07:28.704153 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:28.703911    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-f2gj6_odh-ai-gateway-infra(ffe7c853-3432-4c64-942d-729c453d32d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" podUID="ffe7c853-3432-4c64-942d-729c453d32d1"
Jul 17 17:07:31.522956 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:31.522926    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:31.623235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:31.623202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:35.163980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:35.163935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:39.508989 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:39.508957    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:40.761179 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:40.761147    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82" exitCode=1
Jul 17 17:07:40.761619 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:40.761206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"}
Jul 17 17:07:40.761619 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:40.761253    2561 scope.go:117] "RemoveContainer" containerID="02635961d90fbfe33f7a4eff1c977c9be3548470b03322f7f52f7e410bcba63f"
Jul 17 17:07:40.761738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:40.761628    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:07:40.761856 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:40.761837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:42.508971 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:42.508942    2561 scope.go:117] "RemoveContainer" containerID="142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2"
Jul 17 17:07:42.514984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:42.514957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:42.515369 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:42.515353    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:07:42.515566 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:42.515544    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:43.778858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:43.778829    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffe7c853-3432-4c64-942d-729c453d32d1" containerID="bf4e1df94fc4b470f8e4ca28ac0bf49c68272e60a1fdd94a404957bbe0f05640" exitCode=7
Jul 17 17:07:43.779347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:43.778882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerDied","Data":"bf4e1df94fc4b470f8e4ca28ac0bf49c68272e60a1fdd94a404957bbe0f05640"}
Jul 17 17:07:43.779347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:43.778941    2561 scope.go:117] "RemoveContainer" containerID="142a8ebba4e3caf15347fc8f18ead539b35c41c3e3a5b22fb430d45baa4256e2"
Jul 17 17:07:43.779347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:43.779258    2561 scope.go:117] "RemoveContainer" containerID="bf4e1df94fc4b470f8e4ca28ac0bf49c68272e60a1fdd94a404957bbe0f05640"
Jul 17 17:07:43.779492 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:43.779465    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-f2gj6_odh-ai-gateway-infra(ffe7c853-3432-4c64-942d-729c453d32d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" podUID="ffe7c853-3432-4c64-942d-729c453d32d1"
Jul 17 17:07:44.804376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.804341    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:44.941976 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.941952    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:44.981728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.981684    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:07:44.982072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982060    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982076    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982082    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982145    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982154    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.982226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.982162    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:07:44.985093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.985072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:07:44.992686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:44.992659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:07:45.097461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.097378    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca\") pod \"ffe7c853-3432-4c64-942d-729c453d32d1\" (UID: \"ffe7c853-3432-4c64-942d-729c453d32d1\") "
Jul 17 17:07:45.097606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.097510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9\" (UID: \"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:07:45.097748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.097726    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffe7c853-3432-4c64-942d-729c453d32d1" (UID: "ffe7c853-3432-4c64-942d-729c453d32d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:07:45.198870 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.198830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9\" (UID: \"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:07:45.199111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.198891    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe7c853-3432-4c64-942d-729c453d32d1-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:07:45.199554 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.199532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9\" (UID: \"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:07:45.296417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.296381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:07:45.426656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.426622    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:07:45.428884 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:07:45.428855    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3ba0ba_c9e0_4b30_9868_c27ce6da78fc.slice/crio-92f1d3be04da2b8e179f3c1d9762050eb6109d487c28ae9efee1074803168b61 WatchSource:0}: Error finding container 92f1d3be04da2b8e179f3c1d9762050eb6109d487c28ae9efee1074803168b61: Status 404 returned error can't find the container with id 92f1d3be04da2b8e179f3c1d9762050eb6109d487c28ae9efee1074803168b61
Jul 17 17:07:45.791408 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.791372    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"
Jul 17 17:07:45.791596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.791380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6" event={"ID":"ffe7c853-3432-4c64-942d-729c453d32d1","Type":"ContainerDied","Data":"8ea226ead75cccabbdb9b46f9cff2723a08b0b4b101aa9f02b4c1e1082fe938d"}
Jul 17 17:07:45.791596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.791528    2561 scope.go:117] "RemoveContainer" containerID="bf4e1df94fc4b470f8e4ca28ac0bf49c68272e60a1fdd94a404957bbe0f05640"
Jul 17 17:07:45.792799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.792775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerStarted","Data":"a04216ff1eba59be56e805ec5ca3135b333e6f8a12bd6e3e3cd1ae0ff577ab2f"}
Jul 17 17:07:45.792898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.792817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerStarted","Data":"92f1d3be04da2b8e179f3c1d9762050eb6109d487c28ae9efee1074803168b61"}
Jul 17 17:07:45.811061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.810999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" podStartSLOduration=1.8109850459999999 podStartE2EDuration="1.810985046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:07:45.809503846 +0000 UTC m=+2499.794591384" watchObservedRunningTime="2026-07-17 17:07:45.810985046 +0000 UTC m=+2499.796072583"
Jul 17 17:07:45.823943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.823920    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:45.829750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:45.829727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-f2gj6"]
Jul 17 17:07:46.513554 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:46.513521    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" path="/var/lib/kubelet/pods/ffe7c853-3432-4c64-942d-729c453d32d1/volumes"
Jul 17 17:07:46.799017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:46.798983    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerID="a04216ff1eba59be56e805ec5ca3135b333e6f8a12bd6e3e3cd1ae0ff577ab2f" exitCode=7
Jul 17 17:07:46.799202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:46.799072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerDied","Data":"a04216ff1eba59be56e805ec5ca3135b333e6f8a12bd6e3e3cd1ae0ff577ab2f"}
Jul 17 17:07:46.799291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:46.799276    2561 scope.go:117] "RemoveContainer" containerID="a04216ff1eba59be56e805ec5ca3135b333e6f8a12bd6e3e3cd1ae0ff577ab2f"
Jul 17 17:07:47.508486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.508393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:07:47.508957 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.508939    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:07:47.509179 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:47.509159    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:07:47.805865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.805836    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerID="ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3" exitCode=7
Jul 17 17:07:47.806027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.805919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerDied","Data":"ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3"}
Jul 17 17:07:47.806027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.805972    2561 scope.go:117] "RemoveContainer" containerID="a04216ff1eba59be56e805ec5ca3135b333e6f8a12bd6e3e3cd1ae0ff577ab2f"
Jul 17 17:07:47.806172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:47.806161    2561 scope.go:117] "RemoveContainer" containerID="ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3"
Jul 17 17:07:47.806400 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:47.806380    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9_odh-ai-gateway-infra(1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc"
Jul 17 17:07:48.812337 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:48.812314    2561 scope.go:117] "RemoveContainer" containerID="ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3"
Jul 17 17:07:48.812716 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:48.812532    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9_odh-ai-gateway-infra(1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc"
Jul 17 17:07:53.042632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:53.042596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:53.102256 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:53.102218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:07:59.509193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:07:59.509110    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:07:59.509707 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:07:59.509299    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:08:00.509185 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:00.509154    2561 scope.go:117] "RemoveContainer" containerID="ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3"
Jul 17 17:08:00.867216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:00.867188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerStarted","Data":"afd2c578f35f0021328bd6b6e04a2a6030da7a0569b2d35ca527eb313fafbd1d"}
Jul 17 17:08:01.873574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:01.873545    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerID="afd2c578f35f0021328bd6b6e04a2a6030da7a0569b2d35ca527eb313fafbd1d" exitCode=7
Jul 17 17:08:01.873975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:01.873603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerDied","Data":"afd2c578f35f0021328bd6b6e04a2a6030da7a0569b2d35ca527eb313fafbd1d"}
Jul 17 17:08:01.873975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:01.873652    2561 scope.go:117] "RemoveContainer" containerID="ac20323f5e0542792dc3918109c1662c55438a53a181b073875fb0576c190ff3"
Jul 17 17:08:01.873975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:01.873883    2561 scope.go:117] "RemoveContainer" containerID="afd2c578f35f0021328bd6b6e04a2a6030da7a0569b2d35ca527eb313fafbd1d"
Jul 17 17:08:01.874147 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:01.874076    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9_odh-ai-gateway-infra(1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc"
Jul 17 17:08:01.900393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:01.900363    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:08:03.009944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.009926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:08:03.057855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.057831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca\") pod \"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc\" (UID: \"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc\") "
Jul 17 17:08:03.058163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.058142    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" (UID: "1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:08:03.159270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.159213    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:08:03.884474 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.884437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9" event={"ID":"1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc","Type":"ContainerDied","Data":"92f1d3be04da2b8e179f3c1d9762050eb6109d487c28ae9efee1074803168b61"}
Jul 17 17:08:03.884474 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.884466    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"
Jul 17 17:08:03.884689 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.884526    2561 scope.go:117] "RemoveContainer" containerID="afd2c578f35f0021328bd6b6e04a2a6030da7a0569b2d35ca527eb313fafbd1d"
Jul 17 17:08:03.905153 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.905132    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:08:03.909426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:03.909406    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-j62w9"]
Jul 17 17:08:04.514445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:04.514414    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" path="/var/lib/kubelet/pods/1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc/volumes"
Jul 17 17:08:07.138468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:07.138434    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:08:14.508736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:14.508710    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:08:15.945530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:15.945497    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c" exitCode=1
Jul 17 17:08:15.945941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:15.945554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"}
Jul 17 17:08:15.945941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:15.945597    2561 scope.go:117] "RemoveContainer" containerID="9fa942afc45f29deecd305842f17ab7500ab80e28f77c8c4f4915ed60f1e2c82"
Jul 17 17:08:15.945941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:15.945878    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:08:15.946125 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:15.946102    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:08:17.508201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:17.508164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:08:17.508692 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:17.508270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:08:17.508692 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:17.508677    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:08:17.508915 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:17.508890    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:08:24.848467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848434    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848761    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848772    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848782    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848787    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848804    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:08:24.848819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848809    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe7c853-3432-4c64-942d-729c453d32d1" containerName="revoke-keys"
Jul 17 17:08:24.849036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848862    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.849036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.848869    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:24.853307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.853292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:24.855635 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.855617    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:08:24.859531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.859496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:24.934862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:24.934829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mhrvr\" (UID: \"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:25.036335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.036303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mhrvr\" (UID: \"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:25.036930 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.036907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mhrvr\" (UID: \"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:25.164398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.164340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:25.500743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.500713    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:25.502581 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:08:25.502537    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3ffefb_1e6a_409f_b9b2_39bb2569f24d.slice/crio-e366848ed5c5f2ca8be7296295cf0e912f9f7c01a994619eba97d83827721655 WatchSource:0}: Error finding container e366848ed5c5f2ca8be7296295cf0e912f9f7c01a994619eba97d83827721655: Status 404 returned error can't find the container with id e366848ed5c5f2ca8be7296295cf0e912f9f7c01a994619eba97d83827721655
Jul 17 17:08:25.989765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.989736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerStarted","Data":"83cdb3686560102bca6d03e91393663eecb2ba381356aceecda420e7f0977bfc"}
Jul 17 17:08:25.990127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:25.989784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerStarted","Data":"e366848ed5c5f2ca8be7296295cf0e912f9f7c01a994619eba97d83827721655"}
Jul 17 17:08:26.005965 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:26.005922    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" podStartSLOduration=2.00590676 podStartE2EDuration="2.00590676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:08:26.005264625 +0000 UTC m=+2539.990352189" watchObservedRunningTime="2026-07-17 17:08:26.00590676 +0000 UTC m=+2539.990994297"
Jul 17 17:08:26.995876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:26.995840    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerID="83cdb3686560102bca6d03e91393663eecb2ba381356aceecda420e7f0977bfc" exitCode=7
Jul 17 17:08:26.996281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:26.995885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerDied","Data":"83cdb3686560102bca6d03e91393663eecb2ba381356aceecda420e7f0977bfc"}
Jul 17 17:08:26.996281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:26.996110    2561 scope.go:117] "RemoveContainer" containerID="83cdb3686560102bca6d03e91393663eecb2ba381356aceecda420e7f0977bfc"
Jul 17 17:08:28.003275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:28.003199    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerID="b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374" exitCode=7
Jul 17 17:08:28.003275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:28.003273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerDied","Data":"b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374"}
Jul 17 17:08:28.003687 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:28.003327    2561 scope.go:117] "RemoveContainer" containerID="83cdb3686560102bca6d03e91393663eecb2ba381356aceecda420e7f0977bfc"
Jul 17 17:08:28.003687 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:28.003481    2561 scope.go:117] "RemoveContainer" containerID="b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374"
Jul 17 17:08:28.003687 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:28.003649    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mhrvr_odh-ai-gateway-infra(7e3ffefb-1e6a-409f-b9b2-39bb2569f24d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d"
Jul 17 17:08:29.009866 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:29.009838    2561 scope.go:117] "RemoveContainer" containerID="b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374"
Jul 17 17:08:29.010256 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:29.010033    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mhrvr_odh-ai-gateway-infra(7e3ffefb-1e6a-409f-b9b2-39bb2569f24d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d"
Jul 17 17:08:32.508529 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:32.508490    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:08:32.509127 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:32.508707    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:08:32.592381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:32.592347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:08:32.651184 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:32.651155    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:08:35.023950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.023921    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:35.024326 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.024294    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:35.024326 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.024306    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:35.024398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.024378    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a3ba0ba-c9e0-4b30-9868-c27ce6da78fc" containerName="revoke-keys"
Jul 17 17:08:35.028517 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.028497    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:35.038522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.038495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:35.215975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.215945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4\" (UID: \"4e3c0b49-c194-434e-8b62-47ab1a7dedbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:35.316907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.316842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4\" (UID: \"4e3c0b49-c194-434e-8b62-47ab1a7dedbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:35.317476 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.317458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4\" (UID: \"4e3c0b49-c194-434e-8b62-47ab1a7dedbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:35.339913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.339893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:35.473128 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:35.473097    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:35.476004 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:08:35.475963    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3c0b49_c194_434e_8b62_47ab1a7dedbb.slice/crio-cfd92fe2daf3b4de0630eca4a1a0edbafcffdba6130804b87ae9189bfa34998f WatchSource:0}: Error finding container cfd92fe2daf3b4de0630eca4a1a0edbafcffdba6130804b87ae9189bfa34998f: Status 404 returned error can't find the container with id cfd92fe2daf3b4de0630eca4a1a0edbafcffdba6130804b87ae9189bfa34998f
Jul 17 17:08:36.043272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:36.043243    2561 generic.go:356] "Generic (PLEG): container finished" podID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerID="db382b1016e73ee3727a87bca088e4bc895bde86bf3d79167987e18c21d2605e" exitCode=7
Jul 17 17:08:36.043568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:36.043331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerDied","Data":"db382b1016e73ee3727a87bca088e4bc895bde86bf3d79167987e18c21d2605e"}
Jul 17 17:08:36.043568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:36.043393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerStarted","Data":"cfd92fe2daf3b4de0630eca4a1a0edbafcffdba6130804b87ae9189bfa34998f"}
Jul 17 17:08:36.043568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:36.043511    2561 scope.go:117] "RemoveContainer" containerID="db382b1016e73ee3727a87bca088e4bc895bde86bf3d79167987e18c21d2605e"
Jul 17 17:08:37.049149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:37.049106    2561 generic.go:356] "Generic (PLEG): container finished" podID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerID="f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d" exitCode=7
Jul 17 17:08:37.049590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:37.049170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerDied","Data":"f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d"}
Jul 17 17:08:37.049590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:37.049238    2561 scope.go:117] "RemoveContainer" containerID="db382b1016e73ee3727a87bca088e4bc895bde86bf3d79167987e18c21d2605e"
Jul 17 17:08:37.049590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:37.049410    2561 scope.go:117] "RemoveContainer" containerID="f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d"
Jul 17 17:08:37.049757 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:37.049608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4_odh-ai-gateway-infra(4e3c0b49-c194-434e-8b62-47ab1a7dedbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb"
Jul 17 17:08:38.056429 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:38.056401    2561 scope.go:117] "RemoveContainer" containerID="f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d"
Jul 17 17:08:38.056802 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:38.056580    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4_odh-ai-gateway-infra(4e3c0b49-c194-434e-8b62-47ab1a7dedbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb"
Jul 17 17:08:41.508467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:41.508434    2561 scope.go:117] "RemoveContainer" containerID="b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374"
Jul 17 17:08:42.075723 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:42.075651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerStarted","Data":"b87dbcd5f71921c423c42b7363678c8d229dd5aeebde716c2c4f55f6d679fffe"}
Jul 17 17:08:43.082385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:43.082354    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerID="b87dbcd5f71921c423c42b7363678c8d229dd5aeebde716c2c4f55f6d679fffe" exitCode=7
Jul 17 17:08:43.082795 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:43.082391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerDied","Data":"b87dbcd5f71921c423c42b7363678c8d229dd5aeebde716c2c4f55f6d679fffe"}
Jul 17 17:08:43.082795 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:43.082437    2561 scope.go:117] "RemoveContainer" containerID="b4fc89faa8783faf9cada2d51738246a195a32b00510c9a7cf3dbb3512db9374"
Jul 17 17:08:43.082795 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:43.082663    2561 scope.go:117] "RemoveContainer" containerID="b87dbcd5f71921c423c42b7363678c8d229dd5aeebde716c2c4f55f6d679fffe"
Jul 17 17:08:43.082971 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:43.082839    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mhrvr_odh-ai-gateway-infra(7e3ffefb-1e6a-409f-b9b2-39bb2569f24d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d"
Jul 17 17:08:43.107602 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:43.107579    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:44.222841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:44.222820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:44.287635 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:44.287607    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca\") pod \"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d\" (UID: \"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d\") "
Jul 17 17:08:44.287924 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:44.287904    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" (UID: "7e3ffefb-1e6a-409f-b9b2-39bb2569f24d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:08:44.388377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:44.388315    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:08:45.095541 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:45.095517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"
Jul 17 17:08:45.095717 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:45.095559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr" event={"ID":"7e3ffefb-1e6a-409f-b9b2-39bb2569f24d","Type":"ContainerDied","Data":"e366848ed5c5f2ca8be7296295cf0e912f9f7c01a994619eba97d83827721655"}
Jul 17 17:08:45.095717 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:45.095621    2561 scope.go:117] "RemoveContainer" containerID="b87dbcd5f71921c423c42b7363678c8d229dd5aeebde716c2c4f55f6d679fffe"
Jul 17 17:08:45.109969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:45.109947    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:45.117911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:45.117890    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mhrvr"]
Jul 17 17:08:46.514308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:46.514272    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" path="/var/lib/kubelet/pods/7e3ffefb-1e6a-409f-b9b2-39bb2569f24d/volumes"
Jul 17 17:08:47.508587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:47.508559    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:08:47.508766 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:47.508733    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:08:52.508488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:52.508459    2561 scope.go:117] "RemoveContainer" containerID="f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d"
Jul 17 17:08:53.133988 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:53.133916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerStarted","Data":"b0ee40043ffb809552c01c354ba8eb37fdbddd3edfef3df9900fbca0ac388733"}
Jul 17 17:08:53.149652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:53.149605    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" podStartSLOduration=18.14959247 podStartE2EDuration="18.14959247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:08:53.147856513 +0000 UTC m=+2567.132944054" watchObservedRunningTime="2026-07-17 17:08:53.14959247 +0000 UTC m=+2567.134680008"
Jul 17 17:08:54.140390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:54.140349    2561 generic.go:356] "Generic (PLEG): container finished" podID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerID="b0ee40043ffb809552c01c354ba8eb37fdbddd3edfef3df9900fbca0ac388733" exitCode=7
Jul 17 17:08:54.140877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:54.140423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerDied","Data":"b0ee40043ffb809552c01c354ba8eb37fdbddd3edfef3df9900fbca0ac388733"}
Jul 17 17:08:54.140877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:54.140513    2561 scope.go:117] "RemoveContainer" containerID="f69613170dfbdfaac4af26e67176ff22921709903d8a6d5a9ff8fa520bd60e9d"
Jul 17 17:08:54.140877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:54.140773    2561 scope.go:117] "RemoveContainer" containerID="b0ee40043ffb809552c01c354ba8eb37fdbddd3edfef3df9900fbca0ac388733"
Jul 17 17:08:54.141100 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:08:54.140999    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4_odh-ai-gateway-infra(4e3c0b49-c194-434e-8b62-47ab1a7dedbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb"
Jul 17 17:08:54.157202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:54.157179    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:55.277479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:55.277456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:55.374114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:55.374074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca\") pod \"4e3c0b49-c194-434e-8b62-47ab1a7dedbb\" (UID: \"4e3c0b49-c194-434e-8b62-47ab1a7dedbb\") "
Jul 17 17:08:55.374380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:55.374360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e3c0b49-c194-434e-8b62-47ab1a7dedbb" (UID: "4e3c0b49-c194-434e-8b62-47ab1a7dedbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:08:55.475484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:55.475419    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3c0b49-c194-434e-8b62-47ab1a7dedbb-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:08:56.153167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.153139    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"
Jul 17 17:08:56.153167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.153159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4" event={"ID":"4e3c0b49-c194-434e-8b62-47ab1a7dedbb","Type":"ContainerDied","Data":"cfd92fe2daf3b4de0630eca4a1a0edbafcffdba6130804b87ae9189bfa34998f"}
Jul 17 17:08:56.153388 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.153215    2561 scope.go:117] "RemoveContainer" containerID="b0ee40043ffb809552c01c354ba8eb37fdbddd3edfef3df9900fbca0ac388733"
Jul 17 17:08:56.173311 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.173282    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:56.177520 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.177499    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-4pqb4"]
Jul 17 17:08:56.514404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:08:56.514334    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" path="/var/lib/kubelet/pods/4e3c0b49-c194-434e-8b62-47ab1a7dedbb/volumes"
Jul 17 17:09:00.508661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:00.508630    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:09:01.176847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:01.176820    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a" exitCode=1
Jul 17 17:09:01.176943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:01.176859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"}
Jul 17 17:09:01.176943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:01.176912    2561 scope.go:117] "RemoveContainer" containerID="ff7b76ad42f5297fb7ebeb595c5ba6db766b8a88b3449031a23658466a75518c"
Jul 17 17:09:01.177338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:01.177320    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:01.177551 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:01.177527    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:02.513787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:02.513762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:09:02.514185 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:02.514025    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:02.514185 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:02.514177    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:07.508400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:07.508359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:09:07.508882 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:07.508864    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:07.509140 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:07.509119    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:20.508534 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:20.508500    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:20.508919 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:20.508710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:24.892651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.892619    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.892954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.892975    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.892985    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.892992    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893026    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893032    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893064    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893072    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893089    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893097    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893168    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893177    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893186    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:24.893467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.893193    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:24.897663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.897647    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:24.899834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.899817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:09:24.903394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:24.903369    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:25.025798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:25.025756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9c7jt\" (UID: \"54b10dcb-965e-40a0-a763-7b77255435fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:25.126758 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:25.126721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9c7jt\" (UID: \"54b10dcb-965e-40a0-a763-7b77255435fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:25.127377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:25.127359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9c7jt\" (UID: \"54b10dcb-965e-40a0-a763-7b77255435fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:25.209341 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:25.209262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:25.338897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:25.338874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:25.340932 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:09:25.340905    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b10dcb_965e_40a0_a763_7b77255435fd.slice/crio-8a6dd3b445c3d67a692857206d66fee7791faee48f1cab29b045a6d0bcaee5be WatchSource:0}: Error finding container 8a6dd3b445c3d67a692857206d66fee7791faee48f1cab29b045a6d0bcaee5be: Status 404 returned error can't find the container with id 8a6dd3b445c3d67a692857206d66fee7791faee48f1cab29b045a6d0bcaee5be
Jul 17 17:09:26.288646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:26.288607    2561 generic.go:356] "Generic (PLEG): container finished" podID="54b10dcb-965e-40a0-a763-7b77255435fd" containerID="b55b795cbc2d1ec25ac4ac4fc0c09b302fe1e4438b0fa8ecd3480067c3754bdc" exitCode=7
Jul 17 17:09:26.289078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:26.288693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" event={"ID":"54b10dcb-965e-40a0-a763-7b77255435fd","Type":"ContainerDied","Data":"b55b795cbc2d1ec25ac4ac4fc0c09b302fe1e4438b0fa8ecd3480067c3754bdc"}
Jul 17 17:09:26.289078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:26.288781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" event={"ID":"54b10dcb-965e-40a0-a763-7b77255435fd","Type":"ContainerStarted","Data":"8a6dd3b445c3d67a692857206d66fee7791faee48f1cab29b045a6d0bcaee5be"}
Jul 17 17:09:26.289078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:26.288842    2561 scope.go:117] "RemoveContainer" containerID="b55b795cbc2d1ec25ac4ac4fc0c09b302fe1e4438b0fa8ecd3480067c3754bdc"
Jul 17 17:09:27.294293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:27.294261    2561 generic.go:356] "Generic (PLEG): container finished" podID="54b10dcb-965e-40a0-a763-7b77255435fd" containerID="e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713" exitCode=7
Jul 17 17:09:27.294671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:27.294350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" event={"ID":"54b10dcb-965e-40a0-a763-7b77255435fd","Type":"ContainerDied","Data":"e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713"}
Jul 17 17:09:27.294671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:27.294409    2561 scope.go:117] "RemoveContainer" containerID="b55b795cbc2d1ec25ac4ac4fc0c09b302fe1e4438b0fa8ecd3480067c3754bdc"
Jul 17 17:09:27.294671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:27.294608    2561 scope.go:117] "RemoveContainer" containerID="e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713"
Jul 17 17:09:27.294833 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:27.294820    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9c7jt_odh-ai-gateway-infra(54b10dcb-965e-40a0-a763-7b77255435fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" podUID="54b10dcb-965e-40a0-a763-7b77255435fd"
Jul 17 17:09:28.300741 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:28.300714    2561 scope.go:117] "RemoveContainer" containerID="e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713"
Jul 17 17:09:28.301159 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:28.300906    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9c7jt_odh-ai-gateway-infra(54b10dcb-965e-40a0-a763-7b77255435fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" podUID="54b10dcb-965e-40a0-a763-7b77255435fd"
Jul 17 17:09:33.508946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:33.508920    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:33.509386 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:33.509132    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:35.068667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.068627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:35.069203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.069182    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:35.069288 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.069206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:35.069344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.069308    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e3ffefb-1e6a-409f-b9b2-39bb2569f24d" containerName="revoke-keys"
Jul 17 17:09:35.069344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.069322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3c0b49-c194-434e-8b62-47ab1a7dedbb" containerName="revoke-keys"
Jul 17 17:09:35.073549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.073529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:35.078702 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.078673    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:35.116706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.116671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv\" (UID: \"60f01cfd-52b8-4bc9-992b-0ecced0a04ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:35.217470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.217439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv\" (UID: \"60f01cfd-52b8-4bc9-992b-0ecced0a04ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:35.218015 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.217998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv\" (UID: \"60f01cfd-52b8-4bc9-992b-0ecced0a04ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:35.385035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.384923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:35.509103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:35.509079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:35.511705 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:09:35.511679    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f01cfd_52b8_4bc9_992b_0ecced0a04ce.slice/crio-47ff727334f73ee183e1b4876aca2296af1f9798006828a216f662bfeab2c9f5 WatchSource:0}: Error finding container 47ff727334f73ee183e1b4876aca2296af1f9798006828a216f662bfeab2c9f5: Status 404 returned error can't find the container with id 47ff727334f73ee183e1b4876aca2296af1f9798006828a216f662bfeab2c9f5
Jul 17 17:09:36.337459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:36.337423    2561 generic.go:356] "Generic (PLEG): container finished" podID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerID="2e37182a89b6a8450937d4e31b3f0214a975ad904feb36d43576a2eeacf0dbaf" exitCode=7
Jul 17 17:09:36.337863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:36.337507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" event={"ID":"60f01cfd-52b8-4bc9-992b-0ecced0a04ce","Type":"ContainerDied","Data":"2e37182a89b6a8450937d4e31b3f0214a975ad904feb36d43576a2eeacf0dbaf"}
Jul 17 17:09:36.337863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:36.337559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" event={"ID":"60f01cfd-52b8-4bc9-992b-0ecced0a04ce","Type":"ContainerStarted","Data":"47ff727334f73ee183e1b4876aca2296af1f9798006828a216f662bfeab2c9f5"}
Jul 17 17:09:36.337863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:36.337722    2561 scope.go:117] "RemoveContainer" containerID="2e37182a89b6a8450937d4e31b3f0214a975ad904feb36d43576a2eeacf0dbaf"
Jul 17 17:09:37.343807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:37.343770    2561 generic.go:356] "Generic (PLEG): container finished" podID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerID="6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81" exitCode=7
Jul 17 17:09:37.344279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:37.343845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" event={"ID":"60f01cfd-52b8-4bc9-992b-0ecced0a04ce","Type":"ContainerDied","Data":"6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81"}
Jul 17 17:09:37.344279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:37.343905    2561 scope.go:117] "RemoveContainer" containerID="2e37182a89b6a8450937d4e31b3f0214a975ad904feb36d43576a2eeacf0dbaf"
Jul 17 17:09:37.344279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:37.344113    2561 scope.go:117] "RemoveContainer" containerID="6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81"
Jul 17 17:09:37.344493 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:37.344312    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv_odh-ai-gateway-infra(60f01cfd-52b8-4bc9-992b-0ecced0a04ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce"
Jul 17 17:09:38.350146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:38.350115    2561 scope.go:117] "RemoveContainer" containerID="6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81"
Jul 17 17:09:38.350661 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:38.350393    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv_odh-ai-gateway-infra(60f01cfd-52b8-4bc9-992b-0ecced0a04ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce"
Jul 17 17:09:38.508968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:38.508931    2561 scope.go:117] "RemoveContainer" containerID="e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713"
Jul 17 17:09:39.356241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:39.356210    2561 generic.go:356] "Generic (PLEG): container finished" podID="54b10dcb-965e-40a0-a763-7b77255435fd" containerID="664bf68bf03ef22c09fa9eafe1c39937c02bdd9eaac27272e0f94b510ab50857" exitCode=7
Jul 17 17:09:39.356661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:39.356289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" event={"ID":"54b10dcb-965e-40a0-a763-7b77255435fd","Type":"ContainerDied","Data":"664bf68bf03ef22c09fa9eafe1c39937c02bdd9eaac27272e0f94b510ab50857"}
Jul 17 17:09:39.356661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:39.356351    2561 scope.go:117] "RemoveContainer" containerID="e1eb59389bad60698e3427d8511578490fb965c9e68d5f8bd6287a09aa512713"
Jul 17 17:09:39.356661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:39.356588    2561 scope.go:117] "RemoveContainer" containerID="664bf68bf03ef22c09fa9eafe1c39937c02bdd9eaac27272e0f94b510ab50857"
Jul 17 17:09:39.356852 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:39.356782    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9c7jt_odh-ai-gateway-infra(54b10dcb-965e-40a0-a763-7b77255435fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" podUID="54b10dcb-965e-40a0-a763-7b77255435fd"
Jul 17 17:09:40.381953 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:40.381925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:40.512341 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:40.512309    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:40.562483 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:40.562449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca\") pod \"54b10dcb-965e-40a0-a763-7b77255435fd\" (UID: \"54b10dcb-965e-40a0-a763-7b77255435fd\") "
Jul 17 17:09:40.562783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:40.562762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54b10dcb-965e-40a0-a763-7b77255435fd" (UID: "54b10dcb-965e-40a0-a763-7b77255435fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:09:40.663580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:40.663514    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b10dcb-965e-40a0-a763-7b77255435fd-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:09:41.368435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:41.368401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt" event={"ID":"54b10dcb-965e-40a0-a763-7b77255435fd","Type":"ContainerDied","Data":"8a6dd3b445c3d67a692857206d66fee7791faee48f1cab29b045a6d0bcaee5be"}
Jul 17 17:09:41.368435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:41.368422    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"
Jul 17 17:09:41.368699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:41.368461    2561 scope.go:117] "RemoveContainer" containerID="664bf68bf03ef22c09fa9eafe1c39937c02bdd9eaac27272e0f94b510ab50857"
Jul 17 17:09:41.389660 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:41.389628    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:41.395548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:41.395528    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9c7jt"]
Jul 17 17:09:42.515180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:42.515147    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" path="/var/lib/kubelet/pods/54b10dcb-965e-40a0-a763-7b77255435fd/volumes"
Jul 17 17:09:47.508886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:47.508853    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:47.509319 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:47.509081    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:09:48.508495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:48.508462    2561 scope.go:117] "RemoveContainer" containerID="6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81"
Jul 17 17:09:49.408194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:49.408159    2561 generic.go:356] "Generic (PLEG): container finished" podID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerID="454de3a8da79d5e58237b020acc4a8a62dc49231ea394489b504c69a49901fe1" exitCode=7
Jul 17 17:09:49.408617 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:49.408228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" event={"ID":"60f01cfd-52b8-4bc9-992b-0ecced0a04ce","Type":"ContainerDied","Data":"454de3a8da79d5e58237b020acc4a8a62dc49231ea394489b504c69a49901fe1"}
Jul 17 17:09:49.408617 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:49.408290    2561 scope.go:117] "RemoveContainer" containerID="6ec5a689dcbe00d2ef277acb52482d7585186e989957422b080c05964632cd81"
Jul 17 17:09:49.408617 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:49.408573    2561 scope.go:117] "RemoveContainer" containerID="454de3a8da79d5e58237b020acc4a8a62dc49231ea394489b504c69a49901fe1"
Jul 17 17:09:49.408818 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:49.408799    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv_odh-ai-gateway-infra(60f01cfd-52b8-4bc9-992b-0ecced0a04ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce"
Jul 17 17:09:50.434719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:50.434686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:50.567806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:50.567780    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:50.653697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:50.653665    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca\") pod \"60f01cfd-52b8-4bc9-992b-0ecced0a04ce\" (UID: \"60f01cfd-52b8-4bc9-992b-0ecced0a04ce\") "
Jul 17 17:09:50.654080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:50.654033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60f01cfd-52b8-4bc9-992b-0ecced0a04ce" (UID: "60f01cfd-52b8-4bc9-992b-0ecced0a04ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:09:50.755039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:50.754955    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f01cfd-52b8-4bc9-992b-0ecced0a04ce-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:09:51.421889 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:51.421862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"
Jul 17 17:09:51.421889 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:51.421870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv" event={"ID":"60f01cfd-52b8-4bc9-992b-0ecced0a04ce","Type":"ContainerDied","Data":"47ff727334f73ee183e1b4876aca2296af1f9798006828a216f662bfeab2c9f5"}
Jul 17 17:09:51.422130 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:51.421924    2561 scope.go:117] "RemoveContainer" containerID="454de3a8da79d5e58237b020acc4a8a62dc49231ea394489b504c69a49901fe1"
Jul 17 17:09:51.442354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:51.442331    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:51.447393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:51.447371    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-tdvlv"]
Jul 17 17:09:52.513781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:52.513749    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" path="/var/lib/kubelet/pods/60f01cfd-52b8-4bc9-992b-0ecced0a04ce/volumes"
Jul 17 17:09:59.508787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:09:59.508755    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:09:59.509324 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:09:59.508919    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:11.508472 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:11.508440    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:10:11.508918 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:11.508709    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:14.936713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.936679    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937012    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937037    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937056    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937062    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937069    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937075    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937081    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937086    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937151    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937159    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.937432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937164    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937171    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f01cfd-52b8-4bc9-992b-0ecced0a04ce" containerName="revoke-keys"
Jul 17 17:10:14.937432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.937178    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:14.939991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.939976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:14.942255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.942237    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:10:14.948248 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:14.948225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:15.070783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.070756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-wd26x\" (UID: \"f7850852-7f09-402a-ab07-a2c813f55366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:15.172123 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.172096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-wd26x\" (UID: \"f7850852-7f09-402a-ab07-a2c813f55366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:15.172701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.172684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-wd26x\" (UID: \"f7850852-7f09-402a-ab07-a2c813f55366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:15.251006 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.250946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:15.380733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.377746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:15.538927 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:15.538900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" event={"ID":"f7850852-7f09-402a-ab07-a2c813f55366","Type":"ContainerStarted","Data":"37e614b6e20a8f1477808f985cb61fe042d7aa9b7989faaec710da148cf7f057"}
Jul 17 17:10:16.544182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:16.544142    2561 generic.go:356] "Generic (PLEG): container finished" podID="f7850852-7f09-402a-ab07-a2c813f55366" containerID="6e7f40132f7b40b76900a5a65460b78d2aadd1536fe1ef7f89a2b510e97081ab" exitCode=7
Jul 17 17:10:16.544559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:16.544207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" event={"ID":"f7850852-7f09-402a-ab07-a2c813f55366","Type":"ContainerDied","Data":"6e7f40132f7b40b76900a5a65460b78d2aadd1536fe1ef7f89a2b510e97081ab"}
Jul 17 17:10:16.544559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:16.544402    2561 scope.go:117] "RemoveContainer" containerID="6e7f40132f7b40b76900a5a65460b78d2aadd1536fe1ef7f89a2b510e97081ab"
Jul 17 17:10:17.550918 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:17.550878    2561 generic.go:356] "Generic (PLEG): container finished" podID="f7850852-7f09-402a-ab07-a2c813f55366" containerID="d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d" exitCode=7
Jul 17 17:10:17.551374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:17.550961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" event={"ID":"f7850852-7f09-402a-ab07-a2c813f55366","Type":"ContainerDied","Data":"d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d"}
Jul 17 17:10:17.551374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:17.551035    2561 scope.go:117] "RemoveContainer" containerID="6e7f40132f7b40b76900a5a65460b78d2aadd1536fe1ef7f89a2b510e97081ab"
Jul 17 17:10:17.551374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:17.551249    2561 scope.go:117] "RemoveContainer" containerID="d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d"
Jul 17 17:10:17.551544 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:17.551475    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-wd26x_odh-ai-gateway-infra(f7850852-7f09-402a-ab07-a2c813f55366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" podUID="f7850852-7f09-402a-ab07-a2c813f55366"
Jul 17 17:10:18.557306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:18.557277    2561 scope.go:117] "RemoveContainer" containerID="d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d"
Jul 17 17:10:18.557712 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:18.557476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-wd26x_odh-ai-gateway-infra(f7850852-7f09-402a-ab07-a2c813f55366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" podUID="f7850852-7f09-402a-ab07-a2c813f55366"
Jul 17 17:10:25.106322 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.106282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:25.106735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.106705    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:25.106735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.106718    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:25.106816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.106785    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b10dcb-965e-40a0-a763-7b77255435fd" containerName="revoke-keys"
Jul 17 17:10:25.111331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.111315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:25.117422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.117395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:25.159265 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.159224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx\" (UID: \"3d1c131e-00bd-47b3-9128-2443d2d9ba0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:25.259877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.259842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx\" (UID: \"3d1c131e-00bd-47b3-9128-2443d2d9ba0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:25.260488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.260461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx\" (UID: \"3d1c131e-00bd-47b3-9128-2443d2d9ba0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:25.422957 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.422873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:25.508880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.508851    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:10:25.563475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.563450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:25.565055 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:10:25.565013    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1c131e_00bd_47b3_9128_2443d2d9ba0c.slice/crio-f18e81e7327af94d5ba32c4a41aab9a123a56c6307b9fcad94f23d10ada5a8ba WatchSource:0}: Error finding container f18e81e7327af94d5ba32c4a41aab9a123a56c6307b9fcad94f23d10ada5a8ba: Status 404 returned error can't find the container with id f18e81e7327af94d5ba32c4a41aab9a123a56c6307b9fcad94f23d10ada5a8ba
Jul 17 17:10:25.589706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:25.589674    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" event={"ID":"3d1c131e-00bd-47b3-9128-2443d2d9ba0c","Type":"ContainerStarted","Data":"f18e81e7327af94d5ba32c4a41aab9a123a56c6307b9fcad94f23d10ada5a8ba"}
Jul 17 17:10:26.596235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.596203    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c" exitCode=1
Jul 17 17:10:26.596696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.596277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"}
Jul 17 17:10:26.596696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.596334    2561 scope.go:117] "RemoveContainer" containerID="5c6eea4f28f137530c605e814d12a166b186276be5e310dd4ab6f0e2af28638a"
Jul 17 17:10:26.596696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.596683    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:10:26.596922 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:26.596882    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:26.597897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.597876    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerID="c9d54415c86863893a0ed6d18bffd70a3cb3a150e9492d29d43df25e9bd2c110" exitCode=7
Jul 17 17:10:26.597978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.597932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" event={"ID":"3d1c131e-00bd-47b3-9128-2443d2d9ba0c","Type":"ContainerDied","Data":"c9d54415c86863893a0ed6d18bffd70a3cb3a150e9492d29d43df25e9bd2c110"}
Jul 17 17:10:26.598137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:26.598124    2561 scope.go:117] "RemoveContainer" containerID="c9d54415c86863893a0ed6d18bffd70a3cb3a150e9492d29d43df25e9bd2c110"
Jul 17 17:10:27.508106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.508020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:10:27.508106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.508085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:10:27.604580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.604554    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:10:27.604986 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:27.604760    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:27.606110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.606087    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerID="bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626" exitCode=7
Jul 17 17:10:27.606226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.606136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" event={"ID":"3d1c131e-00bd-47b3-9128-2443d2d9ba0c","Type":"ContainerDied","Data":"bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626"}
Jul 17 17:10:27.606226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.606188    2561 scope.go:117] "RemoveContainer" containerID="c9d54415c86863893a0ed6d18bffd70a3cb3a150e9492d29d43df25e9bd2c110"
Jul 17 17:10:27.606390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:27.606375    2561 scope.go:117] "RemoveContainer" containerID="bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626"
Jul 17 17:10:27.606599 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:27.606579    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx_odh-ai-gateway-infra(3d1c131e-00bd-47b3-9128-2443d2d9ba0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c"
Jul 17 17:10:28.611886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:28.611861    2561 scope.go:117] "RemoveContainer" containerID="bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626"
Jul 17 17:10:28.612286 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:28.612095    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx_odh-ai-gateway-infra(3d1c131e-00bd-47b3-9128-2443d2d9ba0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c"
Jul 17 17:10:28.612286 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:28.612100    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:10:28.612286 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:28.612233    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:31.508228 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:31.508197    2561 scope.go:117] "RemoveContainer" containerID="d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d"
Jul 17 17:10:32.632867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:32.632835    2561 generic.go:356] "Generic (PLEG): container finished" podID="f7850852-7f09-402a-ab07-a2c813f55366" containerID="26bb1da0d67e02708e5bad6763738d47a950da5aa36371529d9a760385de402e" exitCode=7
Jul 17 17:10:32.633366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:32.632894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" event={"ID":"f7850852-7f09-402a-ab07-a2c813f55366","Type":"ContainerDied","Data":"26bb1da0d67e02708e5bad6763738d47a950da5aa36371529d9a760385de402e"}
Jul 17 17:10:32.633366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:32.632964    2561 scope.go:117] "RemoveContainer" containerID="d5b4f5bc7672c933709b3df222cf7a087d705ba587cddffc733c262411110b8d"
Jul 17 17:10:32.633366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:32.633337    2561 scope.go:117] "RemoveContainer" containerID="26bb1da0d67e02708e5bad6763738d47a950da5aa36371529d9a760385de402e"
Jul 17 17:10:32.633565 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:32.633543    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-wd26x_odh-ai-gateway-infra(f7850852-7f09-402a-ab07-a2c813f55366)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" podUID="f7850852-7f09-402a-ab07-a2c813f55366"
Jul 17 17:10:33.658246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:33.658210    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:33.793969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:33.793943    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:33.824334 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:33.824307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca\") pod \"f7850852-7f09-402a-ab07-a2c813f55366\" (UID: \"f7850852-7f09-402a-ab07-a2c813f55366\") "
Jul 17 17:10:33.824678 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:33.824654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7850852-7f09-402a-ab07-a2c813f55366" (UID: "f7850852-7f09-402a-ab07-a2c813f55366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:10:33.924976 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:33.924881    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7850852-7f09-402a-ab07-a2c813f55366-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:10:34.646064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:34.646014    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"
Jul 17 17:10:34.646064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:34.646013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x" event={"ID":"f7850852-7f09-402a-ab07-a2c813f55366","Type":"ContainerDied","Data":"37e614b6e20a8f1477808f985cb61fe042d7aa9b7989faaec710da148cf7f057"}
Jul 17 17:10:34.646321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:34.646109    2561 scope.go:117] "RemoveContainer" containerID="26bb1da0d67e02708e5bad6763738d47a950da5aa36371529d9a760385de402e"
Jul 17 17:10:34.663468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:34.663439    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:34.671131 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:34.671103    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-wd26x"]
Jul 17 17:10:36.514597 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:36.514565    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7850852-7f09-402a-ab07-a2c813f55366" path="/var/lib/kubelet/pods/f7850852-7f09-402a-ab07-a2c813f55366/volumes"
Jul 17 17:10:41.508175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:41.508140    2561 scope.go:117] "RemoveContainer" containerID="bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626"
Jul 17 17:10:41.508607 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:41.508406    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:10:41.508607 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:41.508550    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:10:42.686648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:42.686610    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerID="254bf05b6f46198e40c53128fbcdb5662f3d826f6708422140b2c0eb24eda2f3" exitCode=7
Jul 17 17:10:42.687117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:42.686666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" event={"ID":"3d1c131e-00bd-47b3-9128-2443d2d9ba0c","Type":"ContainerDied","Data":"254bf05b6f46198e40c53128fbcdb5662f3d826f6708422140b2c0eb24eda2f3"}
Jul 17 17:10:42.687117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:42.686726    2561 scope.go:117] "RemoveContainer" containerID="bd831fb3ab438577d54f8c907e04adeaa944d35c914ae2b1f2b576a6b8dc8626"
Jul 17 17:10:42.687117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:42.686996    2561 scope.go:117] "RemoveContainer" containerID="254bf05b6f46198e40c53128fbcdb5662f3d826f6708422140b2c0eb24eda2f3"
Jul 17 17:10:42.687259 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:42.687234    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx_odh-ai-gateway-infra(3d1c131e-00bd-47b3-9128-2443d2d9ba0c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c"
Jul 17 17:10:43.711267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:43.711233    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:43.846432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:43.846405    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:43.911435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:43.911404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca\") pod \"3d1c131e-00bd-47b3-9128-2443d2d9ba0c\" (UID: \"3d1c131e-00bd-47b3-9128-2443d2d9ba0c\") "
Jul 17 17:10:43.911752 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:43.911731    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d1c131e-00bd-47b3-9128-2443d2d9ba0c" (UID: "3d1c131e-00bd-47b3-9128-2443d2d9ba0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:10:44.012379 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.012309    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1c131e-00bd-47b3-9128-2443d2d9ba0c-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:10:44.699818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.699787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"
Jul 17 17:10:44.699967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.699791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx" event={"ID":"3d1c131e-00bd-47b3-9128-2443d2d9ba0c","Type":"ContainerDied","Data":"f18e81e7327af94d5ba32c4a41aab9a123a56c6307b9fcad94f23d10ada5a8ba"}
Jul 17 17:10:44.699967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.699935    2561 scope.go:117] "RemoveContainer" containerID="254bf05b6f46198e40c53128fbcdb5662f3d826f6708422140b2c0eb24eda2f3"
Jul 17 17:10:44.715661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.715631    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:44.721869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:44.721850    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-97fhx"]
Jul 17 17:10:46.514948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:46.514915    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" path="/var/lib/kubelet/pods/3d1c131e-00bd-47b3-9128-2443d2d9ba0c/volumes"
Jul 17 17:10:56.511825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:10:56.511791    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:10:56.512269 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:10:56.511999    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:11:04.974668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974922    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974933    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974946    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974952    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974960    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974965    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974975    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974980    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974988    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.974994    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.975061    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.975074    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.975082    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.975090    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:04.975133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.975096    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:04.979095 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.979078    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:04.981312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.981290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:11:04.984843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:04.984819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:05.105201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.105165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-4kllh\" (UID: \"236f6844-9324-4f11-a938-fedd006303f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:05.205703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.205673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-4kllh\" (UID: \"236f6844-9324-4f11-a938-fedd006303f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:05.206489 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.206466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-4kllh\" (UID: \"236f6844-9324-4f11-a938-fedd006303f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:05.290899 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.290869    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:05.628589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.628563    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:05.631150 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:11:05.631122    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236f6844_9324_4f11_a938_fedd006303f6.slice/crio-59996070a3ba626f8bc4ac94e4529bbf10bb2e73cfe0226354e2955d7f085ce1 WatchSource:0}: Error finding container 59996070a3ba626f8bc4ac94e4529bbf10bb2e73cfe0226354e2955d7f085ce1: Status 404 returned error can't find the container with id 59996070a3ba626f8bc4ac94e4529bbf10bb2e73cfe0226354e2955d7f085ce1
Jul 17 17:11:05.805563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:05.805525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerStarted","Data":"59996070a3ba626f8bc4ac94e4529bbf10bb2e73cfe0226354e2955d7f085ce1"}
Jul 17 17:11:06.561807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:06.561780    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:11:06.571099 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:06.571019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:11:06.811928 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:06.811894    2561 generic.go:356] "Generic (PLEG): container finished" podID="236f6844-9324-4f11-a938-fedd006303f6" containerID="b1fcb3eed4dd57f51ff3b6af635f3ee4f069e8415657dac9ac3fcce5fe5f6a3f" exitCode=7
Jul 17 17:11:06.812111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:06.811976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerDied","Data":"b1fcb3eed4dd57f51ff3b6af635f3ee4f069e8415657dac9ac3fcce5fe5f6a3f"}
Jul 17 17:11:06.812241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:06.812228    2561 scope.go:117] "RemoveContainer" containerID="b1fcb3eed4dd57f51ff3b6af635f3ee4f069e8415657dac9ac3fcce5fe5f6a3f"
Jul 17 17:11:07.819760 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:07.819726    2561 generic.go:356] "Generic (PLEG): container finished" podID="236f6844-9324-4f11-a938-fedd006303f6" containerID="f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec" exitCode=7
Jul 17 17:11:07.820252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:07.819780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerDied","Data":"f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec"}
Jul 17 17:11:07.820252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:07.819832    2561 scope.go:117] "RemoveContainer" containerID="b1fcb3eed4dd57f51ff3b6af635f3ee4f069e8415657dac9ac3fcce5fe5f6a3f"
Jul 17 17:11:07.820252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:07.820038    2561 scope.go:117] "RemoveContainer" containerID="f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec"
Jul 17 17:11:07.820394 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:07.820321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-4kllh_odh-ai-gateway-infra(236f6844-9324-4f11-a938-fedd006303f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" podUID="236f6844-9324-4f11-a938-fedd006303f6"
Jul 17 17:11:08.825314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:08.825288    2561 scope.go:117] "RemoveContainer" containerID="f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec"
Jul 17 17:11:08.825661 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:08.825479    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-4kllh_odh-ai-gateway-infra(236f6844-9324-4f11-a938-fedd006303f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" podUID="236f6844-9324-4f11-a938-fedd006303f6"
Jul 17 17:11:09.508453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:09.508422    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:11:09.508635 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:09.508590    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:11:15.146418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.146388    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:15.146779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.146764    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:15.146779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.146778    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7850852-7f09-402a-ab07-a2c813f55366" containerName="revoke-keys"
Jul 17 17:11:15.146867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.146830    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1c131e-00bd-47b3-9128-2443d2d9ba0c" containerName="revoke-keys"
Jul 17 17:11:15.149790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.149771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:15.157946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.157921    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:15.297682 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.297640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx\" (UID: \"88af17e0-8eb1-4fed-9777-cde652a1be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:15.398279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.398199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx\" (UID: \"88af17e0-8eb1-4fed-9777-cde652a1be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:15.398761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.398743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx\" (UID: \"88af17e0-8eb1-4fed-9777-cde652a1be05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:15.460293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.460259    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:15.600116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.600088    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:15.602127 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:11:15.602096    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af17e0_8eb1_4fed_9777_cde652a1be05.slice/crio-a2005b184d07f98078964b2b4e8321e8ed7cb1e3832779796aaa65ef6488d494 WatchSource:0}: Error finding container a2005b184d07f98078964b2b4e8321e8ed7cb1e3832779796aaa65ef6488d494: Status 404 returned error can't find the container with id a2005b184d07f98078964b2b4e8321e8ed7cb1e3832779796aaa65ef6488d494
Jul 17 17:11:15.857838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:15.857805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerStarted","Data":"a2005b184d07f98078964b2b4e8321e8ed7cb1e3832779796aaa65ef6488d494"}
Jul 17 17:11:16.863634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:16.863600    2561 generic.go:356] "Generic (PLEG): container finished" podID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerID="7618809c0f9287af1fba5f1b99b11fe78b4409e1cbf3e0d1b2fc638cc2c71e0d" exitCode=7
Jul 17 17:11:16.864080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:16.863681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerDied","Data":"7618809c0f9287af1fba5f1b99b11fe78b4409e1cbf3e0d1b2fc638cc2c71e0d"}
Jul 17 17:11:16.864080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:16.863850    2561 scope.go:117] "RemoveContainer" containerID="7618809c0f9287af1fba5f1b99b11fe78b4409e1cbf3e0d1b2fc638cc2c71e0d"
Jul 17 17:11:17.870548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:17.870504    2561 generic.go:356] "Generic (PLEG): container finished" podID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerID="b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3" exitCode=7
Jul 17 17:11:17.871030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:17.870565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerDied","Data":"b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3"}
Jul 17 17:11:17.871030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:17.870619    2561 scope.go:117] "RemoveContainer" containerID="7618809c0f9287af1fba5f1b99b11fe78b4409e1cbf3e0d1b2fc638cc2c71e0d"
Jul 17 17:11:17.871030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:17.870778    2561 scope.go:117] "RemoveContainer" containerID="b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3"
Jul 17 17:11:17.871240 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:17.871025    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx_odh-ai-gateway-infra(88af17e0-8eb1-4fed-9777-cde652a1be05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05"
Jul 17 17:11:18.876750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:18.876721    2561 scope.go:117] "RemoveContainer" containerID="b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3"
Jul 17 17:11:18.877140 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:18.876895    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx_odh-ai-gateway-infra(88af17e0-8eb1-4fed-9777-cde652a1be05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05"
Jul 17 17:11:19.698426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:19.698394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:11:19.781071 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:19.781015    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:11:21.508854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:21.508821    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:11:21.509274 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:21.509080    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:11:22.946123 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:22.946086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:11:23.508306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:23.508275    2561 scope.go:117] "RemoveContainer" containerID="f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec"
Jul 17 17:11:23.900420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:23.900381    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerStarted","Data":"54990f1ce0069853d4287c8bb6d1465873486e73bd040185b13405db08b40dd0"}
Jul 17 17:11:23.916600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:23.916397    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" podStartSLOduration=19.916379973 podStartE2EDuration="19.916379973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:11:23.915693357 +0000 UTC m=+2717.900780897" watchObservedRunningTime="2026-07-17 17:11:23.916379973 +0000 UTC m=+2717.901467511"
Jul 17 17:11:24.286845 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.286810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:11:24.906315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.906277    2561 generic.go:356] "Generic (PLEG): container finished" podID="236f6844-9324-4f11-a938-fedd006303f6" containerID="54990f1ce0069853d4287c8bb6d1465873486e73bd040185b13405db08b40dd0" exitCode=7
Jul 17 17:11:24.906470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.906351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerDied","Data":"54990f1ce0069853d4287c8bb6d1465873486e73bd040185b13405db08b40dd0"}
Jul 17 17:11:24.906470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.906410    2561 scope.go:117] "RemoveContainer" containerID="f1d3281ec18ea789cc3ab7001d002c24718bf9e7e29a796f28faf69b5086d0ec"
Jul 17 17:11:24.906753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.906737    2561 scope.go:117] "RemoveContainer" containerID="54990f1ce0069853d4287c8bb6d1465873486e73bd040185b13405db08b40dd0"
Jul 17 17:11:24.906996 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:24.906978    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-4kllh_odh-ai-gateway-infra(236f6844-9324-4f11-a938-fedd006303f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" podUID="236f6844-9324-4f11-a938-fedd006303f6"
Jul 17 17:11:24.933134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:24.933105    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:26.047858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.047834    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:26.094959 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.094933    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca\") pod \"236f6844-9324-4f11-a938-fedd006303f6\" (UID: \"236f6844-9324-4f11-a938-fedd006303f6\") "
Jul 17 17:11:26.095296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.095275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "236f6844-9324-4f11-a938-fedd006303f6" (UID: "236f6844-9324-4f11-a938-fedd006303f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:11:26.196350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.196273    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/236f6844-9324-4f11-a938-fedd006303f6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:11:26.918700 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.918674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"
Jul 17 17:11:26.918896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.918671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh" event={"ID":"236f6844-9324-4f11-a938-fedd006303f6","Type":"ContainerDied","Data":"59996070a3ba626f8bc4ac94e4529bbf10bb2e73cfe0226354e2955d7f085ce1"}
Jul 17 17:11:26.918896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.918807    2561 scope.go:117] "RemoveContainer" containerID="54990f1ce0069853d4287c8bb6d1465873486e73bd040185b13405db08b40dd0"
Jul 17 17:11:26.935024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.934994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:26.937746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:26.937720    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-4kllh"]
Jul 17 17:11:28.513758 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:28.513728    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236f6844-9324-4f11-a938-fedd006303f6" path="/var/lib/kubelet/pods/236f6844-9324-4f11-a938-fedd006303f6/volumes"
Jul 17 17:11:31.865374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:31.865334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:11:33.509026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:33.508998    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:11:33.509430 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:33.509108    2561 scope.go:117] "RemoveContainer" containerID="b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3"
Jul 17 17:11:33.509430 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:33.509189    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:11:33.952900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:33.952866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerStarted","Data":"7edc424a541814a6f60c03460e259b086610f34377284a19a6a324dc7c1111fa"}
Jul 17 17:11:33.968608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:33.968566    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" podStartSLOduration=18.968551957 podStartE2EDuration="18.968551957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:11:33.967517645 +0000 UTC m=+2727.952605194" watchObservedRunningTime="2026-07-17 17:11:33.968551957 +0000 UTC m=+2727.953639495"
Jul 17 17:11:34.958562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:34.958531    2561 generic.go:356] "Generic (PLEG): container finished" podID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerID="7edc424a541814a6f60c03460e259b086610f34377284a19a6a324dc7c1111fa" exitCode=7
Jul 17 17:11:34.958985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:34.958603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerDied","Data":"7edc424a541814a6f60c03460e259b086610f34377284a19a6a324dc7c1111fa"}
Jul 17 17:11:34.958985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:34.958664    2561 scope.go:117] "RemoveContainer" containerID="b4f72345be894454debb67dfed22f1b8aa1d21d8fc48d9cacaa5bd57977fc8e3"
Jul 17 17:11:34.958985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:34.958914    2561 scope.go:117] "RemoveContainer" containerID="7edc424a541814a6f60c03460e259b086610f34377284a19a6a324dc7c1111fa"
Jul 17 17:11:34.959202 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:34.959182    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx_odh-ai-gateway-infra(88af17e0-8eb1-4fed-9777-cde652a1be05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05"
Jul 17 17:11:34.986497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:34.986465    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:36.092260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.092238    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:36.190092 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.190056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca\") pod \"88af17e0-8eb1-4fed-9777-cde652a1be05\" (UID: \"88af17e0-8eb1-4fed-9777-cde652a1be05\") "
Jul 17 17:11:36.190364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.190344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88af17e0-8eb1-4fed-9777-cde652a1be05" (UID: "88af17e0-8eb1-4fed-9777-cde652a1be05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:11:36.190424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.190412    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88af17e0-8eb1-4fed-9777-cde652a1be05-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:11:36.969462 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.969434    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"
Jul 17 17:11:36.969642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.969434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx" event={"ID":"88af17e0-8eb1-4fed-9777-cde652a1be05","Type":"ContainerDied","Data":"a2005b184d07f98078964b2b4e8321e8ed7cb1e3832779796aaa65ef6488d494"}
Jul 17 17:11:36.969642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.969577    2561 scope.go:117] "RemoveContainer" containerID="7edc424a541814a6f60c03460e259b086610f34377284a19a6a324dc7c1111fa"
Jul 17 17:11:36.986674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.986652    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:36.992572 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:36.992551    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-jnhkx"]
Jul 17 17:11:38.514128 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:38.514097    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" path="/var/lib/kubelet/pods/88af17e0-8eb1-4fed-9777-cde652a1be05/volumes"
Jul 17 17:11:46.510461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:46.510433    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:11:46.510850 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:46.510612    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:11:58.509287 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:11:58.509251    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:11:58.509702 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:11:58.509495    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:12:05.024612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024575    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024938    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024959    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024977    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.024992    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025001    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025010    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025015    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025024    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025029    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025103    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025113    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025121    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:05.025293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.025128    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:05.028004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.027987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:05.030149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.030127    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:12:05.035485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.035163    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:05.038329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.038302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6d6nj\" (UID: \"ffbc7724-e26d-473a-a656-c176138a479a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:05.139107 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.139065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6d6nj\" (UID: \"ffbc7724-e26d-473a-a656-c176138a479a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:05.139728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.139706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6d6nj\" (UID: \"ffbc7724-e26d-473a-a656-c176138a479a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:05.338756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.338672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:05.472037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:05.472012    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:05.473539 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:12:05.473513    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbc7724_e26d_473a_a656_c176138a479a.slice/crio-66ad0db85b6a86b58586ffc0c547cc429ee022a011eaf35f0cc8802d8097c7fa WatchSource:0}: Error finding container 66ad0db85b6a86b58586ffc0c547cc429ee022a011eaf35f0cc8802d8097c7fa: Status 404 returned error can't find the container with id 66ad0db85b6a86b58586ffc0c547cc429ee022a011eaf35f0cc8802d8097c7fa
Jul 17 17:12:06.101114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:06.101076    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffbc7724-e26d-473a-a656-c176138a479a" containerID="a8bb3e874154a41e267beb5b845b0aeecb53fbcea4e57810aef8d827ad8a71b2" exitCode=7
Jul 17 17:12:06.101505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:06.101159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" event={"ID":"ffbc7724-e26d-473a-a656-c176138a479a","Type":"ContainerDied","Data":"a8bb3e874154a41e267beb5b845b0aeecb53fbcea4e57810aef8d827ad8a71b2"}
Jul 17 17:12:06.101505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:06.101215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" event={"ID":"ffbc7724-e26d-473a-a656-c176138a479a","Type":"ContainerStarted","Data":"66ad0db85b6a86b58586ffc0c547cc429ee022a011eaf35f0cc8802d8097c7fa"}
Jul 17 17:12:06.101505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:06.101330    2561 scope.go:117] "RemoveContainer" containerID="a8bb3e874154a41e267beb5b845b0aeecb53fbcea4e57810aef8d827ad8a71b2"
Jul 17 17:12:07.106747 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:07.106718    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffbc7724-e26d-473a-a656-c176138a479a" containerID="ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b" exitCode=7
Jul 17 17:12:07.107240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:07.106803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" event={"ID":"ffbc7724-e26d-473a-a656-c176138a479a","Type":"ContainerDied","Data":"ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b"}
Jul 17 17:12:07.107240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:07.106865    2561 scope.go:117] "RemoveContainer" containerID="a8bb3e874154a41e267beb5b845b0aeecb53fbcea4e57810aef8d827ad8a71b2"
Jul 17 17:12:07.107240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:07.107106    2561 scope.go:117] "RemoveContainer" containerID="ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b"
Jul 17 17:12:07.107455 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:07.107323    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6d6nj_odh-ai-gateway-infra(ffbc7724-e26d-473a-a656-c176138a479a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" podUID="ffbc7724-e26d-473a-a656-c176138a479a"
Jul 17 17:12:08.113412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:08.113382    2561 scope.go:117] "RemoveContainer" containerID="ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b"
Jul 17 17:12:08.113825 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:08.113535    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6d6nj_odh-ai-gateway-infra(ffbc7724-e26d-473a-a656-c176138a479a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" podUID="ffbc7724-e26d-473a-a656-c176138a479a"
Jul 17 17:12:09.508986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:09.508944    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:12:09.509401 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:09.509125    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:12:15.191951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.191920    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:15.192358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.192286    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:15.192358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.192299    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:15.192358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.192350    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="236f6844-9324-4f11-a938-fedd006303f6" containerName="revoke-keys"
Jul 17 17:12:15.192461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.192363    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88af17e0-8eb1-4fed-9777-cde652a1be05" containerName="revoke-keys"
Jul 17 17:12:15.195217 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.195202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:15.203021 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.202994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:15.216898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.216865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm\" (UID: \"2e4cc9e6-2484-4709-9759-7183b663ac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:15.318030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.317994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm\" (UID: \"2e4cc9e6-2484-4709-9759-7183b663ac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:15.318649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.318629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm\" (UID: \"2e4cc9e6-2484-4709-9759-7183b663ac03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:15.506901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.506819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:15.641657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:15.641622    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:15.643876 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:12:15.643846    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4cc9e6_2484_4709_9759_7183b663ac03.slice/crio-8ead6a342ac84d03e84a3cd0f01b1031e3c108c5fb729aeb7fd4160a50f68fc0 WatchSource:0}: Error finding container 8ead6a342ac84d03e84a3cd0f01b1031e3c108c5fb729aeb7fd4160a50f68fc0: Status 404 returned error can't find the container with id 8ead6a342ac84d03e84a3cd0f01b1031e3c108c5fb729aeb7fd4160a50f68fc0
Jul 17 17:12:16.150084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:16.149989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerStarted","Data":"2b86b665bb846b73c4dd40a46494f672667f439be2984482e1dc8896e110650a"}
Jul 17 17:12:16.150084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:16.150061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerStarted","Data":"8ead6a342ac84d03e84a3cd0f01b1031e3c108c5fb729aeb7fd4160a50f68fc0"}
Jul 17 17:12:16.166834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:16.166784    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" podStartSLOduration=1.166769004 podStartE2EDuration="1.166769004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:12:16.164258634 +0000 UTC m=+2770.149346171" watchObservedRunningTime="2026-07-17 17:12:16.166769004 +0000 UTC m=+2770.151856542"
Jul 17 17:12:17.156419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:17.156389    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerID="2b86b665bb846b73c4dd40a46494f672667f439be2984482e1dc8896e110650a" exitCode=7
Jul 17 17:12:17.156815 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:17.156477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerDied","Data":"2b86b665bb846b73c4dd40a46494f672667f439be2984482e1dc8896e110650a"}
Jul 17 17:12:17.156815 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:17.156656    2561 scope.go:117] "RemoveContainer" containerID="2b86b665bb846b73c4dd40a46494f672667f439be2984482e1dc8896e110650a"
Jul 17 17:12:18.163166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:18.163130    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerID="8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1" exitCode=7
Jul 17 17:12:18.163584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:18.163206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerDied","Data":"8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1"}
Jul 17 17:12:18.163584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:18.163258    2561 scope.go:117] "RemoveContainer" containerID="2b86b665bb846b73c4dd40a46494f672667f439be2984482e1dc8896e110650a"
Jul 17 17:12:18.163584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:18.163434    2561 scope.go:117] "RemoveContainer" containerID="8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1"
Jul 17 17:12:18.163750 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:18.163655    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm_odh-ai-gateway-infra(2e4cc9e6-2484-4709-9759-7183b663ac03)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03"
Jul 17 17:12:19.169288 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:19.169258    2561 scope.go:117] "RemoveContainer" containerID="8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1"
Jul 17 17:12:19.169661 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:19.169430    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm_odh-ai-gateway-infra(2e4cc9e6-2484-4709-9759-7183b663ac03)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03"
Jul 17 17:12:20.508574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:20.508548    2561 scope.go:117] "RemoveContainer" containerID="ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b"
Jul 17 17:12:21.181979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:21.181945    2561 generic.go:356] "Generic (PLEG): container finished" podID="ffbc7724-e26d-473a-a656-c176138a479a" containerID="b83d7becb5e1f0f1120fcf2bd4c19a80af0684ad8bb8f9588a98496efc996a94" exitCode=7
Jul 17 17:12:21.182156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:21.182009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" event={"ID":"ffbc7724-e26d-473a-a656-c176138a479a","Type":"ContainerDied","Data":"b83d7becb5e1f0f1120fcf2bd4c19a80af0684ad8bb8f9588a98496efc996a94"}
Jul 17 17:12:21.182156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:21.182095    2561 scope.go:117] "RemoveContainer" containerID="ccf1d6ec2c99c7d23a41e2fe956fa83b7b8539ed4176e64a7f74e8886e8d218b"
Jul 17 17:12:21.182385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:21.182365    2561 scope.go:117] "RemoveContainer" containerID="b83d7becb5e1f0f1120fcf2bd4c19a80af0684ad8bb8f9588a98496efc996a94"
Jul 17 17:12:21.182614 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:21.182584    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6d6nj_odh-ai-gateway-infra(ffbc7724-e26d-473a-a656-c176138a479a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" podUID="ffbc7724-e26d-473a-a656-c176138a479a"
Jul 17 17:12:22.207882 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.207850    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:22.340996 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.340973    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:22.373842 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.373812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca\") pod \"ffbc7724-e26d-473a-a656-c176138a479a\" (UID: \"ffbc7724-e26d-473a-a656-c176138a479a\") "
Jul 17 17:12:22.374139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.374116    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffbc7724-e26d-473a-a656-c176138a479a" (UID: "ffbc7724-e26d-473a-a656-c176138a479a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:12:22.475303 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.475243    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbc7724-e26d-473a-a656-c176138a479a-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:12:22.514468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.514446    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:12:22.514610 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:22.514592    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:12:22.957814 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.957777    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:22.958144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958155    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958161    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958176    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958183    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958241    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.958394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.958248    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:22.961169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.961153    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:22.968354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:22.968294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:23.080827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.080800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr\" (UID: \"38eda223-f3f6-484d-80ee-4d1cfd37169a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:23.182165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.182136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr\" (UID: \"38eda223-f3f6-484d-80ee-4d1cfd37169a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:23.182747 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.182721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr\" (UID: \"38eda223-f3f6-484d-80ee-4d1cfd37169a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:23.193035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.193009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"
Jul 17 17:12:23.193155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.193066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj" event={"ID":"ffbc7724-e26d-473a-a656-c176138a479a","Type":"ContainerDied","Data":"66ad0db85b6a86b58586ffc0c547cc429ee022a011eaf35f0cc8802d8097c7fa"}
Jul 17 17:12:23.193155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.193129    2561 scope.go:117] "RemoveContainer" containerID="b83d7becb5e1f0f1120fcf2bd4c19a80af0684ad8bb8f9588a98496efc996a94"
Jul 17 17:12:23.211986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.211929    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:23.214023 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.214005    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6d6nj"]
Jul 17 17:12:23.272488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.272471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:23.399697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:23.399674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:23.402277 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:12:23.402248    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eda223_f3f6_484d_80ee_4d1cfd37169a.slice/crio-377893d7612acc5ba934430dc3753987f43d723e0617cde06c223b2ad1a28f36 WatchSource:0}: Error finding container 377893d7612acc5ba934430dc3753987f43d723e0617cde06c223b2ad1a28f36: Status 404 returned error can't find the container with id 377893d7612acc5ba934430dc3753987f43d723e0617cde06c223b2ad1a28f36
Jul 17 17:12:24.198619 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:24.198592    2561 generic.go:356] "Generic (PLEG): container finished" podID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerID="0ec44ff88b7791ff6f722c2fad9ee70c42559c600bbe4981a4f004b319e817b9" exitCode=7
Jul 17 17:12:24.198781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:24.198678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" event={"ID":"38eda223-f3f6-484d-80ee-4d1cfd37169a","Type":"ContainerDied","Data":"0ec44ff88b7791ff6f722c2fad9ee70c42559c600bbe4981a4f004b319e817b9"}
Jul 17 17:12:24.198781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:24.198732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" event={"ID":"38eda223-f3f6-484d-80ee-4d1cfd37169a","Type":"ContainerStarted","Data":"377893d7612acc5ba934430dc3753987f43d723e0617cde06c223b2ad1a28f36"}
Jul 17 17:12:24.198867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:24.198836    2561 scope.go:117] "RemoveContainer" containerID="0ec44ff88b7791ff6f722c2fad9ee70c42559c600bbe4981a4f004b319e817b9"
Jul 17 17:12:24.514146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:24.514117    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbc7724-e26d-473a-a656-c176138a479a" path="/var/lib/kubelet/pods/ffbc7724-e26d-473a-a656-c176138a479a/volumes"
Jul 17 17:12:25.206654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:25.206576    2561 generic.go:356] "Generic (PLEG): container finished" podID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerID="d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0" exitCode=7
Jul 17 17:12:25.206654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:25.206637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" event={"ID":"38eda223-f3f6-484d-80ee-4d1cfd37169a","Type":"ContainerDied","Data":"d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0"}
Jul 17 17:12:25.206849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:25.206685    2561 scope.go:117] "RemoveContainer" containerID="0ec44ff88b7791ff6f722c2fad9ee70c42559c600bbe4981a4f004b319e817b9"
Jul 17 17:12:25.206907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:25.206894    2561 scope.go:117] "RemoveContainer" containerID="d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0"
Jul 17 17:12:25.207125 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:25.207107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr_odh-ai-gateway-infra(38eda223-f3f6-484d-80ee-4d1cfd37169a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a"
Jul 17 17:12:26.212979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:26.212942    2561 scope.go:117] "RemoveContainer" containerID="d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0"
Jul 17 17:12:26.213389 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:26.213130    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr_odh-ai-gateway-infra(38eda223-f3f6-484d-80ee-4d1cfd37169a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a"
Jul 17 17:12:31.508319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:31.508236    2561 scope.go:117] "RemoveContainer" containerID="8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1"
Jul 17 17:12:32.241743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:32.241708    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerID="71d7afff2c502c5bc0ca1368682c415307e1409bbccd4ed196390864bc263540" exitCode=7
Jul 17 17:12:32.241743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:32.241748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerDied","Data":"71d7afff2c502c5bc0ca1368682c415307e1409bbccd4ed196390864bc263540"}
Jul 17 17:12:32.241941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:32.241827    2561 scope.go:117] "RemoveContainer" containerID="8ef472b4914d2f1774cd7a83acbccab7d8b6868f0c48c19db320240253d2c5f1"
Jul 17 17:12:32.242132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:32.242111    2561 scope.go:117] "RemoveContainer" containerID="71d7afff2c502c5bc0ca1368682c415307e1409bbccd4ed196390864bc263540"
Jul 17 17:12:32.242349 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:32.242326    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm_odh-ai-gateway-infra(2e4cc9e6-2484-4709-9759-7183b663ac03)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03"
Jul 17 17:12:33.266992 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:33.266955    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:33.401520 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:33.401492    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:33.472586 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:33.472557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca\") pod \"2e4cc9e6-2484-4709-9759-7183b663ac03\" (UID: \"2e4cc9e6-2484-4709-9759-7183b663ac03\") "
Jul 17 17:12:33.472892 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:33.472868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e4cc9e6-2484-4709-9759-7183b663ac03" (UID: "2e4cc9e6-2484-4709-9759-7183b663ac03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:12:33.573758 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:33.573731    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e4cc9e6-2484-4709-9759-7183b663ac03-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:12:34.252803 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.252776    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"
Jul 17 17:12:34.252947 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.252826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm" event={"ID":"2e4cc9e6-2484-4709-9759-7183b663ac03","Type":"ContainerDied","Data":"8ead6a342ac84d03e84a3cd0f01b1031e3c108c5fb729aeb7fd4160a50f68fc0"}
Jul 17 17:12:34.252947 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.252881    2561 scope.go:117] "RemoveContainer" containerID="71d7afff2c502c5bc0ca1368682c415307e1409bbccd4ed196390864bc263540"
Jul 17 17:12:34.274575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.274554    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:34.283593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.283572    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ssrkm"]
Jul 17 17:12:34.513958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.513885    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" path="/var/lib/kubelet/pods/2e4cc9e6-2484-4709-9759-7183b663ac03/volumes"
Jul 17 17:12:34.514506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:34.514492    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:12:34.514660 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:34.514642    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:12:35.102892 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:35.102859    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:12:35.203293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:35.203262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:12:39.508189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:39.508153    2561 scope.go:117] "RemoveContainer" containerID="d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0"
Jul 17 17:12:40.283315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:40.283285    2561 generic.go:356] "Generic (PLEG): container finished" podID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerID="8291bd629afe9f904e057667a1eaf0d9ada9c1e51e2240dcf0665e6a64a158bf" exitCode=7
Jul 17 17:12:40.283473 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:40.283362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" event={"ID":"38eda223-f3f6-484d-80ee-4d1cfd37169a","Type":"ContainerDied","Data":"8291bd629afe9f904e057667a1eaf0d9ada9c1e51e2240dcf0665e6a64a158bf"}
Jul 17 17:12:40.283473 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:40.283441    2561 scope.go:117] "RemoveContainer" containerID="d6fdad2399caa82e36f070c2a69eebb517509258a2fa071e449c608edfcfc9a0"
Jul 17 17:12:40.283711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:40.283686    2561 scope.go:117] "RemoveContainer" containerID="8291bd629afe9f904e057667a1eaf0d9ada9c1e51e2240dcf0665e6a64a158bf"
Jul 17 17:12:40.283924 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:40.283902    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr_odh-ai-gateway-infra(38eda223-f3f6-484d-80ee-4d1cfd37169a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a"
Jul 17 17:12:41.309622 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:41.309587    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:41.441881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:41.441861    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:41.533319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:41.533293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca\") pod \"38eda223-f3f6-484d-80ee-4d1cfd37169a\" (UID: \"38eda223-f3f6-484d-80ee-4d1cfd37169a\") "
Jul 17 17:12:41.533621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:41.533599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38eda223-f3f6-484d-80ee-4d1cfd37169a" (UID: "38eda223-f3f6-484d-80ee-4d1cfd37169a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:12:41.634658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:41.634604    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38eda223-f3f6-484d-80ee-4d1cfd37169a-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:12:42.296247 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.296215    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"
Jul 17 17:12:42.296423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.296250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr" event={"ID":"38eda223-f3f6-484d-80ee-4d1cfd37169a","Type":"ContainerDied","Data":"377893d7612acc5ba934430dc3753987f43d723e0617cde06c223b2ad1a28f36"}
Jul 17 17:12:42.296423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.296300    2561 scope.go:117] "RemoveContainer" containerID="8291bd629afe9f904e057667a1eaf0d9ada9c1e51e2240dcf0665e6a64a158bf"
Jul 17 17:12:42.317714 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.317688    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:42.326672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.326652    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-plghr"]
Jul 17 17:12:42.513996 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:42.513959    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" path="/var/lib/kubelet/pods/38eda223-f3f6-484d-80ee-4d1cfd37169a/volumes"
Jul 17 17:12:48.509235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:48.509201    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:12:48.509614 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:48.509377    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:12:55.064464 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064431    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064755    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064765    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064779    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064784    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064796    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064802    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064811    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064816    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064823    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.064830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064828    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.065194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064880    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.065194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064888    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.065194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064894    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:12:55.065194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064900    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:12:55.065194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.064906    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbc7724-e26d-473a-a656-c176138a479a" containerName="revoke-keys"
Jul 17 17:12:55.067841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.067824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:12:55.070568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.070551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:12:55.079642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.079612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:12:55.249345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.249319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-sbcpn\" (UID: \"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:12:55.350294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.350218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-sbcpn\" (UID: \"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:12:55.350811 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.350791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-sbcpn\" (UID: \"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:12:55.378073 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.378032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:12:55.506355 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:55.506328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:12:55.507903 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:12:55.507876    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c3e97e_3b90_4b14_bfc2_0e59adfc25cb.slice/crio-4b738ac8bea4488531bc7b12a93282acad534ba4f74f685fbe02e5382af263c1 WatchSource:0}: Error finding container 4b738ac8bea4488531bc7b12a93282acad534ba4f74f685fbe02e5382af263c1: Status 404 returned error can't find the container with id 4b738ac8bea4488531bc7b12a93282acad534ba4f74f685fbe02e5382af263c1
Jul 17 17:12:56.364281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:56.364249    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerID="5e39d2ca800b5256d3d99f3528638b4b5717dbe2afbee2b628bb8dd76f99eb7e" exitCode=7
Jul 17 17:12:56.364649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:56.364327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" event={"ID":"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb","Type":"ContainerDied","Data":"5e39d2ca800b5256d3d99f3528638b4b5717dbe2afbee2b628bb8dd76f99eb7e"}
Jul 17 17:12:56.364649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:56.364379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" event={"ID":"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb","Type":"ContainerStarted","Data":"4b738ac8bea4488531bc7b12a93282acad534ba4f74f685fbe02e5382af263c1"}
Jul 17 17:12:56.364649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:56.364508    2561 scope.go:117] "RemoveContainer" containerID="5e39d2ca800b5256d3d99f3528638b4b5717dbe2afbee2b628bb8dd76f99eb7e"
Jul 17 17:12:57.370902 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:57.370866    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerID="349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b" exitCode=7
Jul 17 17:12:57.371380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:57.370909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" event={"ID":"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb","Type":"ContainerDied","Data":"349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b"}
Jul 17 17:12:57.371380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:57.371001    2561 scope.go:117] "RemoveContainer" containerID="5e39d2ca800b5256d3d99f3528638b4b5717dbe2afbee2b628bb8dd76f99eb7e"
Jul 17 17:12:57.371380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:57.371188    2561 scope.go:117] "RemoveContainer" containerID="349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b"
Jul 17 17:12:57.371495 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:57.371394    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-sbcpn_odh-ai-gateway-infra(d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb"
Jul 17 17:12:58.377177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:12:58.377153    2561 scope.go:117] "RemoveContainer" containerID="349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b"
Jul 17 17:12:58.377553 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:12:58.377329    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-sbcpn_odh-ai-gateway-infra(d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb"
Jul 17 17:13:03.509177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:03.509151    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:13:03.509567 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:03.509308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:05.233005 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.232973    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:05.233385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.233319    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:13:05.233385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.233331    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:13:05.233455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.233406    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38eda223-f3f6-484d-80ee-4d1cfd37169a" containerName="revoke-keys"
Jul 17 17:13:05.233455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.233419    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc9e6-2484-4709-9759-7183b663ac03" containerName="revoke-keys"
Jul 17 17:13:05.237325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.237308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:05.245482 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.245444    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:05.331884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.331852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq\" (UID: \"50e15dbc-3eea-4805-85d2-5aefb0394c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:05.432267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.432240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq\" (UID: \"50e15dbc-3eea-4805-85d2-5aefb0394c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:05.432829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.432811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq\" (UID: \"50e15dbc-3eea-4805-85d2-5aefb0394c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:05.549072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.549030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:05.677002 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:05.676978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:05.679420 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:13:05.679390    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e15dbc_3eea_4805_85d2_5aefb0394c87.slice/crio-8c2f1e0dbe0c69d6fe4bee9db8f117d61548dbcde7f606554517f20bb95dfa1e WatchSource:0}: Error finding container 8c2f1e0dbe0c69d6fe4bee9db8f117d61548dbcde7f606554517f20bb95dfa1e: Status 404 returned error can't find the container with id 8c2f1e0dbe0c69d6fe4bee9db8f117d61548dbcde7f606554517f20bb95dfa1e
Jul 17 17:13:06.416935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:06.416906    2561 generic.go:356] "Generic (PLEG): container finished" podID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerID="a26b8e17bba59f7d116cc6c591ea256eb2f1d87c60c1d1d28e184c2f1098d778" exitCode=7
Jul 17 17:13:06.417277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:06.416990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" event={"ID":"50e15dbc-3eea-4805-85d2-5aefb0394c87","Type":"ContainerDied","Data":"a26b8e17bba59f7d116cc6c591ea256eb2f1d87c60c1d1d28e184c2f1098d778"}
Jul 17 17:13:06.417277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:06.417061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" event={"ID":"50e15dbc-3eea-4805-85d2-5aefb0394c87","Type":"ContainerStarted","Data":"8c2f1e0dbe0c69d6fe4bee9db8f117d61548dbcde7f606554517f20bb95dfa1e"}
Jul 17 17:13:06.417277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:06.417162    2561 scope.go:117] "RemoveContainer" containerID="a26b8e17bba59f7d116cc6c591ea256eb2f1d87c60c1d1d28e184c2f1098d778"
Jul 17 17:13:07.422910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:07.422869    2561 generic.go:356] "Generic (PLEG): container finished" podID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerID="9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23" exitCode=7
Jul 17 17:13:07.423331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:07.422945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" event={"ID":"50e15dbc-3eea-4805-85d2-5aefb0394c87","Type":"ContainerDied","Data":"9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23"}
Jul 17 17:13:07.423331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:07.423001    2561 scope.go:117] "RemoveContainer" containerID="a26b8e17bba59f7d116cc6c591ea256eb2f1d87c60c1d1d28e184c2f1098d778"
Jul 17 17:13:07.423331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:07.423224    2561 scope.go:117] "RemoveContainer" containerID="9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23"
Jul 17 17:13:07.423451 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:07.423412    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq_odh-ai-gateway-infra(50e15dbc-3eea-4805-85d2-5aefb0394c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87"
Jul 17 17:13:08.429515 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:08.429488    2561 scope.go:117] "RemoveContainer" containerID="9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23"
Jul 17 17:13:08.429895 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:08.429674    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq_odh-ai-gateway-infra(50e15dbc-3eea-4805-85d2-5aefb0394c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87"
Jul 17 17:13:13.003126 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.003088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:13.007777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.007760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:13.014759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.014658    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:13.099920 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.099886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl\" (UID: \"7475563f-3a9d-4a11-b3df-d87c1fd5b623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:13.201212 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.201175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl\" (UID: \"7475563f-3a9d-4a11-b3df-d87c1fd5b623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:13.201801 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.201777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl\" (UID: \"7475563f-3a9d-4a11-b3df-d87c1fd5b623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:13.319630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.319595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:13.448261 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.448236    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:13.449828 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:13:13.449800    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7475563f_3a9d_4a11_b3df_d87c1fd5b623.slice/crio-8be6e3bf70520e4c6c82af8d9c48dc72fed473da3a42750a3ba9e9d6af884c0a WatchSource:0}: Error finding container 8be6e3bf70520e4c6c82af8d9c48dc72fed473da3a42750a3ba9e9d6af884c0a: Status 404 returned error can't find the container with id 8be6e3bf70520e4c6c82af8d9c48dc72fed473da3a42750a3ba9e9d6af884c0a
Jul 17 17:13:13.508308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:13.508283    2561 scope.go:117] "RemoveContainer" containerID="349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b"
Jul 17 17:13:14.461706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.461668    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerID="b94cb93d6b1108760320c96069b4b0878ab00395f6032048845695c412020949" exitCode=7
Jul 17 17:13:14.462190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.461738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" event={"ID":"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb","Type":"ContainerDied","Data":"b94cb93d6b1108760320c96069b4b0878ab00395f6032048845695c412020949"}
Jul 17 17:13:14.462190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.461793    2561 scope.go:117] "RemoveContainer" containerID="349d6b5882caa30aff14594979bdd200002505fa6c31a37de175f9965ca8df4b"
Jul 17 17:13:14.462190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.462151    2561 scope.go:117] "RemoveContainer" containerID="b94cb93d6b1108760320c96069b4b0878ab00395f6032048845695c412020949"
Jul 17 17:13:14.462396 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:14.462377    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-sbcpn_odh-ai-gateway-infra(d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb"
Jul 17 17:13:14.463566 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.463542    2561 generic.go:356] "Generic (PLEG): container finished" podID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerID="7625ed6066240137079a851bb0345401d0fe69640cc4e447c9f506ed27a73780" exitCode=7
Jul 17 17:13:14.463673 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.463606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" event={"ID":"7475563f-3a9d-4a11-b3df-d87c1fd5b623","Type":"ContainerDied","Data":"7625ed6066240137079a851bb0345401d0fe69640cc4e447c9f506ed27a73780"}
Jul 17 17:13:14.463673 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.463647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" event={"ID":"7475563f-3a9d-4a11-b3df-d87c1fd5b623","Type":"ContainerStarted","Data":"8be6e3bf70520e4c6c82af8d9c48dc72fed473da3a42750a3ba9e9d6af884c0a"}
Jul 17 17:13:14.463764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:14.463751    2561 scope.go:117] "RemoveContainer" containerID="7625ed6066240137079a851bb0345401d0fe69640cc4e447c9f506ed27a73780"
Jul 17 17:13:15.470440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.470350    2561 generic.go:356] "Generic (PLEG): container finished" podID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerID="a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f" exitCode=7
Jul 17 17:13:15.470874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.470444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" event={"ID":"7475563f-3a9d-4a11-b3df-d87c1fd5b623","Type":"ContainerDied","Data":"a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f"}
Jul 17 17:13:15.470874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.470521    2561 scope.go:117] "RemoveContainer" containerID="7625ed6066240137079a851bb0345401d0fe69640cc4e447c9f506ed27a73780"
Jul 17 17:13:15.470874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.470642    2561 scope.go:117] "RemoveContainer" containerID="a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f"
Jul 17 17:13:15.471000 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:15.470873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl_odh-ai-gateway-infra(7475563f-3a9d-4a11-b3df-d87c1fd5b623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623"
Jul 17 17:13:15.490726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.490699    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:13:15.620571 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.620545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:13:15.723229 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.723151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca\") pod \"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb\" (UID: \"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb\") "
Jul 17 17:13:15.723500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.723477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" (UID: "d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:13:15.824466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:15.824436    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:13:16.478103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.478071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"
Jul 17 17:13:16.478599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.478092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn" event={"ID":"d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb","Type":"ContainerDied","Data":"4b738ac8bea4488531bc7b12a93282acad534ba4f74f685fbe02e5382af263c1"}
Jul 17 17:13:16.478599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.478159    2561 scope.go:117] "RemoveContainer" containerID="b94cb93d6b1108760320c96069b4b0878ab00395f6032048845695c412020949"
Jul 17 17:13:16.479896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.479879    2561 scope.go:117] "RemoveContainer" containerID="a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f"
Jul 17 17:13:16.480186 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:16.480162    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl_odh-ai-gateway-infra(7475563f-3a9d-4a11-b3df-d87c1fd5b623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623"
Jul 17 17:13:16.513637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.513613    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:13:16.513937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.513925    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:13:16.514208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.514196    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:13:16.518186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:16.518167    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-sbcpn"]
Jul 17 17:13:17.490482 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.490451    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8" exitCode=1
Jul 17 17:13:17.490904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.490500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"}
Jul 17 17:13:17.490904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.490568    2561 scope.go:117] "RemoveContainer" containerID="9fcacc3d9aa3e0fde8e3d92e614b8c9acbd88cb485e9434497df22e01eb81e9c"
Jul 17 17:13:17.490904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.490899    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:13:17.491141 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:17.491114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:17.507765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.507742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:13:17.507869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:17.507777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:13:18.497109 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:18.497084    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:13:18.497507 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:18.497240    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:18.514299 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:18.514273    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" path="/var/lib/kubelet/pods/d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb/volumes"
Jul 17 17:13:22.509148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:22.509118    2561 scope.go:117] "RemoveContainer" containerID="9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23"
Jul 17 17:13:23.519880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:23.519795    2561 generic.go:356] "Generic (PLEG): container finished" podID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerID="892548eb63eb97da12dfd630fc8a2e80f35ce0341fb442c7e1f38b51e41c91e0" exitCode=7
Jul 17 17:13:23.519880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:23.519868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" event={"ID":"50e15dbc-3eea-4805-85d2-5aefb0394c87","Type":"ContainerDied","Data":"892548eb63eb97da12dfd630fc8a2e80f35ce0341fb442c7e1f38b51e41c91e0"}
Jul 17 17:13:23.520351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:23.519923    2561 scope.go:117] "RemoveContainer" containerID="9897284c2f1386d07655e8273e8d40da997b7854b40009c71c1d6af0aa16ef23"
Jul 17 17:13:23.520351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:23.520191    2561 scope.go:117] "RemoveContainer" containerID="892548eb63eb97da12dfd630fc8a2e80f35ce0341fb442c7e1f38b51e41c91e0"
Jul 17 17:13:23.520425 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:23.520401    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq_odh-ai-gateway-infra(50e15dbc-3eea-4805-85d2-5aefb0394c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87"
Jul 17 17:13:24.559561 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:24.559532    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:24.687090 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:24.687066    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:24.804564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:24.804524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca\") pod \"50e15dbc-3eea-4805-85d2-5aefb0394c87\" (UID: \"50e15dbc-3eea-4805-85d2-5aefb0394c87\") "
Jul 17 17:13:24.804876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:24.804854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50e15dbc-3eea-4805-85d2-5aefb0394c87" (UID: "50e15dbc-3eea-4805-85d2-5aefb0394c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:13:24.905900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:24.905827    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e15dbc-3eea-4805-85d2-5aefb0394c87-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:13:25.531727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:25.531701    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"
Jul 17 17:13:25.531912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:25.531745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq" event={"ID":"50e15dbc-3eea-4805-85d2-5aefb0394c87","Type":"ContainerDied","Data":"8c2f1e0dbe0c69d6fe4bee9db8f117d61548dbcde7f606554517f20bb95dfa1e"}
Jul 17 17:13:25.531912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:25.531805    2561 scope.go:117] "RemoveContainer" containerID="892548eb63eb97da12dfd630fc8a2e80f35ce0341fb442c7e1f38b51e41c91e0"
Jul 17 17:13:25.555607 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:25.555581    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:25.560233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:25.560204    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nd8pq"]
Jul 17 17:13:26.513974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:26.513942    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" path="/var/lib/kubelet/pods/50e15dbc-3eea-4805-85d2-5aefb0394c87/volumes"
Jul 17 17:13:30.508724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:30.508686    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:13:30.509201 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:30.508868    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:31.509185 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:31.509147    2561 scope.go:117] "RemoveContainer" containerID="a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f"
Jul 17 17:13:32.566748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:32.566717    2561 generic.go:356] "Generic (PLEG): container finished" podID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerID="de344890d16cadba6815ac9096d2fad34893a153805f7936c01aa1da22d52493" exitCode=7
Jul 17 17:13:32.567211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:32.566796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" event={"ID":"7475563f-3a9d-4a11-b3df-d87c1fd5b623","Type":"ContainerDied","Data":"de344890d16cadba6815ac9096d2fad34893a153805f7936c01aa1da22d52493"}
Jul 17 17:13:32.567211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:32.566855    2561 scope.go:117] "RemoveContainer" containerID="a03a2dc7371aabdc1b4d3165cac2bfa3f89226a2520f060c2b7f67a7a0c1580f"
Jul 17 17:13:32.567313 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:32.567212    2561 scope.go:117] "RemoveContainer" containerID="de344890d16cadba6815ac9096d2fad34893a153805f7936c01aa1da22d52493"
Jul 17 17:13:32.567443 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:32.567422    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl_odh-ai-gateway-infra(7475563f-3a9d-4a11-b3df-d87c1fd5b623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623"
Jul 17 17:13:33.607861 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:33.607830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:33.737484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:33.737457    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:33.886122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:33.886024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca\") pod \"7475563f-3a9d-4a11-b3df-d87c1fd5b623\" (UID: \"7475563f-3a9d-4a11-b3df-d87c1fd5b623\") "
Jul 17 17:13:33.886380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:33.886355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7475563f-3a9d-4a11-b3df-d87c1fd5b623" (UID: "7475563f-3a9d-4a11-b3df-d87c1fd5b623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:13:33.986891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:33.986845    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7475563f-3a9d-4a11-b3df-d87c1fd5b623-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:13:34.578427 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:34.578391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl" event={"ID":"7475563f-3a9d-4a11-b3df-d87c1fd5b623","Type":"ContainerDied","Data":"8be6e3bf70520e4c6c82af8d9c48dc72fed473da3a42750a3ba9e9d6af884c0a"}
Jul 17 17:13:34.578427 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:34.578424    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"
Jul 17 17:13:34.578671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:34.578462    2561 scope.go:117] "RemoveContainer" containerID="de344890d16cadba6815ac9096d2fad34893a153805f7936c01aa1da22d52493"
Jul 17 17:13:34.606212 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:34.606190    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:34.615365 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:34.615340    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-snzwl"]
Jul 17 17:13:36.513584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:36.513548    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" path="/var/lib/kubelet/pods/7475563f-3a9d-4a11-b3df-d87c1fd5b623/volumes"
Jul 17 17:13:45.508920 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:45.508885    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:13:45.509421 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:45.509103    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:55.116667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.116634    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.116972    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.116985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117001    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117006    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117015    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117021    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117033    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117039    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117071    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117082    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117090    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117100    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117105    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117163    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117170    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117176    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117182    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117188    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117195    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117201    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.117560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.117207    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c3e97e-3b90-4b14-bfc2-0e59adfc25cb" containerName="revoke-keys"
Jul 17 17:13:55.120037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.120021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:13:55.122320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.122299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:13:55.127306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.127196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:13:55.165667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.165633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9mthq\" (UID: \"659d919b-3436-4612-a251-9aeee6501c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:13:55.266889 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.266857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9mthq\" (UID: \"659d919b-3436-4612-a251-9aeee6501c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:13:55.267476 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.267457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-9mthq\" (UID: \"659d919b-3436-4612-a251-9aeee6501c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:13:55.431160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.431075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:13:55.561022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.560998    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:13:55.563815 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:13:55.563786    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659d919b_3436_4612_a251_9aeee6501c12.slice/crio-a2e1462633ab14b6187048a728c50f2fc1fa8b274e7e5a82a6d37deb8aec5442 WatchSource:0}: Error finding container a2e1462633ab14b6187048a728c50f2fc1fa8b274e7e5a82a6d37deb8aec5442: Status 404 returned error can't find the container with id a2e1462633ab14b6187048a728c50f2fc1fa8b274e7e5a82a6d37deb8aec5442
Jul 17 17:13:55.675760 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:55.675728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" event={"ID":"659d919b-3436-4612-a251-9aeee6501c12","Type":"ContainerStarted","Data":"a2e1462633ab14b6187048a728c50f2fc1fa8b274e7e5a82a6d37deb8aec5442"}
Jul 17 17:13:56.681579 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:56.681544    2561 generic.go:356] "Generic (PLEG): container finished" podID="659d919b-3436-4612-a251-9aeee6501c12" containerID="ff3e4bfee2df6ddcaa599b4999823fcac4b860c295b74e70de23655c203bc183" exitCode=7
Jul 17 17:13:56.681968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:56.681629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" event={"ID":"659d919b-3436-4612-a251-9aeee6501c12","Type":"ContainerDied","Data":"ff3e4bfee2df6ddcaa599b4999823fcac4b860c295b74e70de23655c203bc183"}
Jul 17 17:13:56.681968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:56.681789    2561 scope.go:117] "RemoveContainer" containerID="ff3e4bfee2df6ddcaa599b4999823fcac4b860c295b74e70de23655c203bc183"
Jul 17 17:13:57.688726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:57.688632    2561 generic.go:356] "Generic (PLEG): container finished" podID="659d919b-3436-4612-a251-9aeee6501c12" containerID="a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf" exitCode=7
Jul 17 17:13:57.688726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:57.688708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" event={"ID":"659d919b-3436-4612-a251-9aeee6501c12","Type":"ContainerDied","Data":"a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf"}
Jul 17 17:13:57.689192 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:57.688801    2561 scope.go:117] "RemoveContainer" containerID="ff3e4bfee2df6ddcaa599b4999823fcac4b860c295b74e70de23655c203bc183"
Jul 17 17:13:57.689192 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:57.688989    2561 scope.go:117] "RemoveContainer" containerID="a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf"
Jul 17 17:13:57.689298 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:57.689237    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9mthq_odh-ai-gateway-infra(659d919b-3436-4612-a251-9aeee6501c12)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" podUID="659d919b-3436-4612-a251-9aeee6501c12"
Jul 17 17:13:58.508679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:58.508592    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:13:58.508837 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:58.508784    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:13:58.694611 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:13:58.694583    2561 scope.go:117] "RemoveContainer" containerID="a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf"
Jul 17 17:13:58.694972 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:13:58.694768    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9mthq_odh-ai-gateway-infra(659d919b-3436-4612-a251-9aeee6501c12)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" podUID="659d919b-3436-4612-a251-9aeee6501c12"
Jul 17 17:14:05.288614 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.288570    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:05.289022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.288916    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:14:05.289022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.288938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:14:05.289022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.288957    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:14:05.289022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.288963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7475563f-3a9d-4a11-b3df-d87c1fd5b623" containerName="revoke-keys"
Jul 17 17:14:05.289022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.289024    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e15dbc-3eea-4805-85d2-5aefb0394c87" containerName="revoke-keys"
Jul 17 17:14:05.293038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.293022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:05.301879 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.301854    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:05.355824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.355795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr\" (UID: \"ca090593-2ce9-4e61-b444-5c5c2d2012f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:05.456499 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.456472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr\" (UID: \"ca090593-2ce9-4e61-b444-5c5c2d2012f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:05.457131 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.457111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr\" (UID: \"ca090593-2ce9-4e61-b444-5c5c2d2012f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:05.603833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.603758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:05.941695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:05.941616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:05.945079 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:14:05.945037    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca090593_2ce9_4e61_b444_5c5c2d2012f9.slice/crio-af32afb85b4900c2a2fb437fc732dd109399b7d8cf103c10dfcee30059ea4736 WatchSource:0}: Error finding container af32afb85b4900c2a2fb437fc732dd109399b7d8cf103c10dfcee30059ea4736: Status 404 returned error can't find the container with id af32afb85b4900c2a2fb437fc732dd109399b7d8cf103c10dfcee30059ea4736
Jul 17 17:14:06.732684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:06.732653    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerID="d24962902b8486d1129a7489355ed3c720e57631f4a6e84ea400ecb503e8af8c" exitCode=7
Jul 17 17:14:06.733140 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:06.732691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerDied","Data":"d24962902b8486d1129a7489355ed3c720e57631f4a6e84ea400ecb503e8af8c"}
Jul 17 17:14:06.733140 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:06.732731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerStarted","Data":"af32afb85b4900c2a2fb437fc732dd109399b7d8cf103c10dfcee30059ea4736"}
Jul 17 17:14:06.733140 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:06.732895    2561 scope.go:117] "RemoveContainer" containerID="d24962902b8486d1129a7489355ed3c720e57631f4a6e84ea400ecb503e8af8c"
Jul 17 17:14:07.739493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:07.739393    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerID="b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699" exitCode=7
Jul 17 17:14:07.739915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:07.739477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerDied","Data":"b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699"}
Jul 17 17:14:07.739915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:07.739554    2561 scope.go:117] "RemoveContainer" containerID="d24962902b8486d1129a7489355ed3c720e57631f4a6e84ea400ecb503e8af8c"
Jul 17 17:14:07.739915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:07.739743    2561 scope.go:117] "RemoveContainer" containerID="b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699"
Jul 17 17:14:07.740032 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:07.739974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr_odh-ai-gateway-infra(ca090593-2ce9-4e61-b444-5c5c2d2012f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9"
Jul 17 17:14:08.745825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:08.745799    2561 scope.go:117] "RemoveContainer" containerID="b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699"
Jul 17 17:14:08.746235 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:08.745966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr_odh-ai-gateway-infra(ca090593-2ce9-4e61-b444-5c5c2d2012f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9"
Jul 17 17:14:12.508525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:12.508480    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:14:12.509001 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:12.508734    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:14:13.054833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.054797    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:13.059565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.059548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:13.066643 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.066612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:13.122408 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.122377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn\" (UID: \"ed72b572-271c-48c0-87fd-776dea1d228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:13.223785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.223751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn\" (UID: \"ed72b572-271c-48c0-87fd-776dea1d228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:13.224454 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.224435    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn\" (UID: \"ed72b572-271c-48c0-87fd-776dea1d228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:13.370956 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.370883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:13.498180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.498155    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:13.499652 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:14:13.499619    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded72b572_271c_48c0_87fd_776dea1d228b.slice/crio-e94bcf787df959d5dd7bd6f904ee7e87090bb7cee89f5a7eb793fd159d519b3f WatchSource:0}: Error finding container e94bcf787df959d5dd7bd6f904ee7e87090bb7cee89f5a7eb793fd159d519b3f: Status 404 returned error can't find the container with id e94bcf787df959d5dd7bd6f904ee7e87090bb7cee89f5a7eb793fd159d519b3f
Jul 17 17:14:13.508077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.508057    2561 scope.go:117] "RemoveContainer" containerID="a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf"
Jul 17 17:14:13.770066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:13.770013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerStarted","Data":"e94bcf787df959d5dd7bd6f904ee7e87090bb7cee89f5a7eb793fd159d519b3f"}
Jul 17 17:14:14.776116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.776075    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed72b572-271c-48c0-87fd-776dea1d228b" containerID="62bf5983b55d4de51aeb4cd59163406fc4d85f7429eb065e87292a82c6754d79" exitCode=7
Jul 17 17:14:14.776512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.776165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerDied","Data":"62bf5983b55d4de51aeb4cd59163406fc4d85f7429eb065e87292a82c6754d79"}
Jul 17 17:14:14.776512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.776328    2561 scope.go:117] "RemoveContainer" containerID="62bf5983b55d4de51aeb4cd59163406fc4d85f7429eb065e87292a82c6754d79"
Jul 17 17:14:14.777998 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.777979    2561 generic.go:356] "Generic (PLEG): container finished" podID="659d919b-3436-4612-a251-9aeee6501c12" containerID="7b543a6bba444fb5de2a8f483dfc78567f11c845c7d12096c719960dffb83b0f" exitCode=7
Jul 17 17:14:14.778108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.778065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" event={"ID":"659d919b-3436-4612-a251-9aeee6501c12","Type":"ContainerDied","Data":"7b543a6bba444fb5de2a8f483dfc78567f11c845c7d12096c719960dffb83b0f"}
Jul 17 17:14:14.778150 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.778117    2561 scope.go:117] "RemoveContainer" containerID="a4ad378fef75642edb5879d09631bd9274bbae6d1ac605c5878bbdd17591afdf"
Jul 17 17:14:14.778389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:14.778363    2561 scope.go:117] "RemoveContainer" containerID="7b543a6bba444fb5de2a8f483dfc78567f11c845c7d12096c719960dffb83b0f"
Jul 17 17:14:14.778570 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:14.778547    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-9mthq_odh-ai-gateway-infra(659d919b-3436-4612-a251-9aeee6501c12)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" podUID="659d919b-3436-4612-a251-9aeee6501c12"
Jul 17 17:14:15.784745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.784710    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed72b572-271c-48c0-87fd-776dea1d228b" containerID="2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce" exitCode=7
Jul 17 17:14:15.785205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.784783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerDied","Data":"2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce"}
Jul 17 17:14:15.785205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.784848    2561 scope.go:117] "RemoveContainer" containerID="62bf5983b55d4de51aeb4cd59163406fc4d85f7429eb065e87292a82c6754d79"
Jul 17 17:14:15.785205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.785087    2561 scope.go:117] "RemoveContainer" containerID="2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce"
Jul 17 17:14:15.785400 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:15.785308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn_odh-ai-gateway-infra(ed72b572-271c-48c0-87fd-776dea1d228b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" podUID="ed72b572-271c-48c0-87fd-776dea1d228b"
Jul 17 17:14:15.819695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.819671    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:14:15.950877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:15.950847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:14:16.050772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.050695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca\") pod \"659d919b-3436-4612-a251-9aeee6501c12\" (UID: \"659d919b-3436-4612-a251-9aeee6501c12\") "
Jul 17 17:14:16.051057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.051024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "659d919b-3436-4612-a251-9aeee6501c12" (UID: "659d919b-3436-4612-a251-9aeee6501c12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:14:16.151301 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.151271    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d919b-3436-4612-a251-9aeee6501c12-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:14:16.792266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.792233    2561 scope.go:117] "RemoveContainer" containerID="2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce"
Jul 17 17:14:16.792720 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:16.792470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn_odh-ai-gateway-infra(ed72b572-271c-48c0-87fd-776dea1d228b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" podUID="ed72b572-271c-48c0-87fd-776dea1d228b"
Jul 17 17:14:16.793148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.793132    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"
Jul 17 17:14:16.793198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.793144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq" event={"ID":"659d919b-3436-4612-a251-9aeee6501c12","Type":"ContainerDied","Data":"a2e1462633ab14b6187048a728c50f2fc1fa8b274e7e5a82a6d37deb8aec5442"}
Jul 17 17:14:16.793198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.793194    2561 scope.go:117] "RemoveContainer" containerID="7b543a6bba444fb5de2a8f483dfc78567f11c845c7d12096c719960dffb83b0f"
Jul 17 17:14:16.819274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.819244    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:14:16.821543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:16.821520    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-9mthq"]
Jul 17 17:14:18.514208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:18.514177    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659d919b-3436-4612-a251-9aeee6501c12" path="/var/lib/kubelet/pods/659d919b-3436-4612-a251-9aeee6501c12/volumes"
Jul 17 17:14:23.508266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:23.508238    2561 scope.go:117] "RemoveContainer" containerID="b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699"
Jul 17 17:14:23.828065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:23.828025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerStarted","Data":"1ecda03127102a728e919945b96f3a66d329969716dec1f6e3d205b1dcd57136"}
Jul 17 17:14:23.845038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:23.844989    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" podStartSLOduration=18.844975455 podStartE2EDuration="18.844975455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:14:23.843437787 +0000 UTC m=+2897.828525326" watchObservedRunningTime="2026-07-17 17:14:23.844975455 +0000 UTC m=+2897.830062993"
Jul 17 17:14:24.834472 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:24.834440    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerID="1ecda03127102a728e919945b96f3a66d329969716dec1f6e3d205b1dcd57136" exitCode=7
Jul 17 17:14:24.834894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:24.834504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerDied","Data":"1ecda03127102a728e919945b96f3a66d329969716dec1f6e3d205b1dcd57136"}
Jul 17 17:14:24.834894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:24.834560    2561 scope.go:117] "RemoveContainer" containerID="b83a4ba429b6bf1a1aa8bcf0ca2bb6ad988d283a60d833f688f22cea546eb699"
Jul 17 17:14:24.834894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:24.834865    2561 scope.go:117] "RemoveContainer" containerID="1ecda03127102a728e919945b96f3a66d329969716dec1f6e3d205b1dcd57136"
Jul 17 17:14:24.835087 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:24.835067    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr_odh-ai-gateway-infra(ca090593-2ce9-4e61-b444-5c5c2d2012f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9"
Jul 17 17:14:24.859722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:24.859693    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:25.970426 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:25.970402    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:26.134070 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.133971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca\") pod \"ca090593-2ce9-4e61-b444-5c5c2d2012f9\" (UID: \"ca090593-2ce9-4e61-b444-5c5c2d2012f9\") "
Jul 17 17:14:26.134350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.134328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca090593-2ce9-4e61-b444-5c5c2d2012f9" (UID: "ca090593-2ce9-4e61-b444-5c5c2d2012f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:14:26.235121 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.235088    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca090593-2ce9-4e61-b444-5c5c2d2012f9-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:14:26.845926 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.845896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr" event={"ID":"ca090593-2ce9-4e61-b444-5c5c2d2012f9","Type":"ContainerDied","Data":"af32afb85b4900c2a2fb437fc732dd109399b7d8cf103c10dfcee30059ea4736"}
Jul 17 17:14:26.846112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.845953    2561 scope.go:117] "RemoveContainer" containerID="1ecda03127102a728e919945b96f3a66d329969716dec1f6e3d205b1dcd57136"
Jul 17 17:14:26.846112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.845908    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"
Jul 17 17:14:26.864499 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.864474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:26.872069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:26.872035    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rtpbr"]
Jul 17 17:14:27.508838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:27.508804    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:14:27.509322 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:27.509071    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:14:28.513797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:28.513763    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" path="/var/lib/kubelet/pods/ca090593-2ce9-4e61-b444-5c5c2d2012f9/volumes"
Jul 17 17:14:32.508382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:32.508354    2561 scope.go:117] "RemoveContainer" containerID="2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce"
Jul 17 17:14:32.874654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:32.874625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerStarted","Data":"80f065a30012780396aa24a99634c7a0537ccbfe4116332faa7765bc700beb84"}
Jul 17 17:14:32.891075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:32.891016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" podStartSLOduration=19.891002075 podStartE2EDuration="19.891002075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:14:32.889637084 +0000 UTC m=+2906.874724623" watchObservedRunningTime="2026-07-17 17:14:32.891002075 +0000 UTC m=+2906.876089611"
Jul 17 17:14:33.881233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:33.881203    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed72b572-271c-48c0-87fd-776dea1d228b" containerID="80f065a30012780396aa24a99634c7a0537ccbfe4116332faa7765bc700beb84" exitCode=7
Jul 17 17:14:33.881233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:33.881237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerDied","Data":"80f065a30012780396aa24a99634c7a0537ccbfe4116332faa7765bc700beb84"}
Jul 17 17:14:33.881706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:33.881292    2561 scope.go:117] "RemoveContainer" containerID="2df7048e770aa4d0344252d74e853bbda0a029b2ec132caa61b5c8ae6234b3ce"
Jul 17 17:14:33.881706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:33.881551    2561 scope.go:117] "RemoveContainer" containerID="80f065a30012780396aa24a99634c7a0537ccbfe4116332faa7765bc700beb84"
Jul 17 17:14:33.881794 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:33.881770    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn_odh-ai-gateway-infra(ed72b572-271c-48c0-87fd-776dea1d228b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" podUID="ed72b572-271c-48c0-87fd-776dea1d228b"
Jul 17 17:14:33.903629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:33.903605    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:35.028125 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.028101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:35.216218 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.216142    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca\") pod \"ed72b572-271c-48c0-87fd-776dea1d228b\" (UID: \"ed72b572-271c-48c0-87fd-776dea1d228b\") "
Jul 17 17:14:35.216470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.216449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed72b572-271c-48c0-87fd-776dea1d228b" (UID: "ed72b572-271c-48c0-87fd-776dea1d228b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:14:35.316951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.316913    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed72b572-271c-48c0-87fd-776dea1d228b-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:14:35.899776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.899748    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"
Jul 17 17:14:35.899958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.899791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn" event={"ID":"ed72b572-271c-48c0-87fd-776dea1d228b","Type":"ContainerDied","Data":"e94bcf787df959d5dd7bd6f904ee7e87090bb7cee89f5a7eb793fd159d519b3f"}
Jul 17 17:14:35.899958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.899849    2561 scope.go:117] "RemoveContainer" containerID="80f065a30012780396aa24a99634c7a0537ccbfe4116332faa7765bc700beb84"
Jul 17 17:14:35.921487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.921458    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:35.928522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:35.928499    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-7w6hn"]
Jul 17 17:14:36.513868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:36.513835    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" path="/var/lib/kubelet/pods/ed72b572-271c-48c0-87fd-776dea1d228b/volumes"
Jul 17 17:14:41.508348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:41.508313    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:14:41.508784 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:41.508543    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:14:52.508370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:52.508343    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:14:52.508744 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:52.508535    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:14:55.160894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.160861    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:14:55.161453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161433    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161456    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161469    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161477    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161492    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161502    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161523    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161531    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161541    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161549    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161560    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161568    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161584    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161593    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161604    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161612    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161705    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161719    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161728    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161737    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161757    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.161931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.161765    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:14:55.165032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.165014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:14:55.167557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.167537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:14:55.171106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.171002    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:14:55.272013 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.271978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-db2g4\" (UID: \"87d26ead-7864-44e8-8688-6d8ac9878313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:14:55.372745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.372709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-db2g4\" (UID: \"87d26ead-7864-44e8-8688-6d8ac9878313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:14:55.373363 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.373342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-db2g4\" (UID: \"87d26ead-7864-44e8-8688-6d8ac9878313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:14:55.476588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.476514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:14:55.628635 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.628611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:14:55.630684 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:14:55.630661    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d26ead_7864_44e8_8688_6d8ac9878313.slice/crio-e4b7a8f47a9d6b11f8c64353ea1d684e96e2b804d4804d7d37cb4f352d090912 WatchSource:0}: Error finding container e4b7a8f47a9d6b11f8c64353ea1d684e96e2b804d4804d7d37cb4f352d090912: Status 404 returned error can't find the container with id e4b7a8f47a9d6b11f8c64353ea1d684e96e2b804d4804d7d37cb4f352d090912
Jul 17 17:14:55.989838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.989806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerStarted","Data":"88566aff31adaaacafeb897d841d4eb53bbe882d961dcbcbb88d495a901bfc75"}
Jul 17 17:14:55.989996 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:55.989861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerStarted","Data":"e4b7a8f47a9d6b11f8c64353ea1d684e96e2b804d4804d7d37cb4f352d090912"}
Jul 17 17:14:56.009710 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:56.009666    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" podStartSLOduration=1.00965072 podStartE2EDuration="1.00965072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:14:56.007737834 +0000 UTC m=+2929.992825370" watchObservedRunningTime="2026-07-17 17:14:56.00965072 +0000 UTC m=+2929.994738303"
Jul 17 17:14:56.994904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:56.994869    2561 generic.go:356] "Generic (PLEG): container finished" podID="87d26ead-7864-44e8-8688-6d8ac9878313" containerID="88566aff31adaaacafeb897d841d4eb53bbe882d961dcbcbb88d495a901bfc75" exitCode=7
Jul 17 17:14:56.995407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:56.994960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerDied","Data":"88566aff31adaaacafeb897d841d4eb53bbe882d961dcbcbb88d495a901bfc75"}
Jul 17 17:14:56.995407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:56.995197    2561 scope.go:117] "RemoveContainer" containerID="88566aff31adaaacafeb897d841d4eb53bbe882d961dcbcbb88d495a901bfc75"
Jul 17 17:14:58.001302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:58.001212    2561 generic.go:356] "Generic (PLEG): container finished" podID="87d26ead-7864-44e8-8688-6d8ac9878313" containerID="2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6" exitCode=7
Jul 17 17:14:58.001302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:58.001273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerDied","Data":"2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6"}
Jul 17 17:14:58.001735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:58.001330    2561 scope.go:117] "RemoveContainer" containerID="88566aff31adaaacafeb897d841d4eb53bbe882d961dcbcbb88d495a901bfc75"
Jul 17 17:14:58.001735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:58.001504    2561 scope.go:117] "RemoveContainer" containerID="2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6"
Jul 17 17:14:58.001735 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:58.001710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-db2g4_odh-ai-gateway-infra(87d26ead-7864-44e8-8688-6d8ac9878313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" podUID="87d26ead-7864-44e8-8688-6d8ac9878313"
Jul 17 17:14:59.007177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:14:59.007151    2561 scope.go:117] "RemoveContainer" containerID="2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6"
Jul 17 17:14:59.007557 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:14:59.007312    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-db2g4_odh-ai-gateway-infra(87d26ead-7864-44e8-8688-6d8ac9878313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" podUID="87d26ead-7864-44e8-8688-6d8ac9878313"
Jul 17 17:15:00.147733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.147697    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:00.148253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.148234    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:15:00.148334 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.148255    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca090593-2ce9-4e61-b444-5c5c2d2012f9" containerName="revoke-keys"
Jul 17 17:15:00.148390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.148372    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="659d919b-3436-4612-a251-9aeee6501c12" containerName="revoke-keys"
Jul 17 17:15:00.148443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.148390    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed72b572-271c-48c0-87fd-776dea1d228b" containerName="revoke-keys"
Jul 17 17:15:00.152412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.152392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:00.152535 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.152499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"
Jul 17 17:15:00.155445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.155428    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:00.155551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.155525    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"
Jul 17 17:15:00.165753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.165728    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"
Jul 17 17:15:00.166085 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.165918    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"]
Jul 17 17:15:00.169279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.169255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"
Jul 17 17:15:00.170475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.170453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:00.170590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.170491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:00.170590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.170514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:00.170590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.170534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"]
Jul 17 17:15:00.170733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.170607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"
Jul 17 17:15:00.178857 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.178838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"
Jul 17 17:15:00.209756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.209736    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"
Jul 17 17:15:00.218155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.218119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"
Jul 17 17:15:00.344989 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.344567    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:00.347562 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:00.347522    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc585b017_4c4f_4f45_b3ef_ece9613ec940.slice/crio-7d58744f13622cf9c792104c6ee4cf765a8a35027aef916b0e812c28f72ea818 WatchSource:0}: Error finding container 7d58744f13622cf9c792104c6ee4cf765a8a35027aef916b0e812c28f72ea818: Status 404 returned error can't find the container with id 7d58744f13622cf9c792104c6ee4cf765a8a35027aef916b0e812c28f72ea818
Jul 17 17:15:00.376558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.376534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:00.387330 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:00.387304    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d44964_1484_4263_9974_2d8796b3dfe7.slice/crio-54bcbb052e7062fb63e670d5940ff9fe889bff160b9f77d00d2fa49dd3c45bd2 WatchSource:0}: Error finding container 54bcbb052e7062fb63e670d5940ff9fe889bff160b9f77d00d2fa49dd3c45bd2: Status 404 returned error can't find the container with id 54bcbb052e7062fb63e670d5940ff9fe889bff160b9f77d00d2fa49dd3c45bd2
Jul 17 17:15:00.601119 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.601021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"]
Jul 17 17:15:00.603170 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:00.603149    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b42c42_f817_4067_98b8_0a05402b316a.slice/crio-d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084 WatchSource:0}: Error finding container d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084: Status 404 returned error can't find the container with id d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084
Jul 17 17:15:00.603531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:00.603511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:00.607437 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:00.607417    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a32b4ab_b593_44eb_91aa_2a64a130470b.slice/crio-9ddd711224eafd29624a3a0a1e9489e29a38e56cf7449d26b2dd44e695bceeca WatchSource:0}: Error finding container 9ddd711224eafd29624a3a0a1e9489e29a38e56cf7449d26b2dd44e695bceeca: Status 404 returned error can't find the container with id 9ddd711224eafd29624a3a0a1e9489e29a38e56cf7449d26b2dd44e695bceeca
Jul 17 17:15:01.019056 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.019003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerStarted","Data":"d65ca41e06780c710ff115deee43dd3d4bbf0ca5c7a50f0fe08258dd5dd6b393"}
Jul 17 17:15:01.019175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.019084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerStarted","Data":"9ddd711224eafd29624a3a0a1e9489e29a38e56cf7449d26b2dd44e695bceeca"}
Jul 17 17:15:01.020507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.020476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7" event={"ID":"c9b42c42-f817-4067-98b8-0a05402b316a","Type":"ContainerStarted","Data":"2abbf6ea4e52399a809fc783409cab85090654f11a0d67f818e7cf25bbadb518"}
Jul 17 17:15:01.020626 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.020526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7" event={"ID":"c9b42c42-f817-4067-98b8-0a05402b316a","Type":"ContainerStarted","Data":"d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084"}
Jul 17 17:15:01.022261 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.022237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerStarted","Data":"3942395333a4c35cc22e9fa42151b5a63dd8351a5e13652f2a78f80b0189a5a7"}
Jul 17 17:15:01.022430 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.022282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerStarted","Data":"7d58744f13622cf9c792104c6ee4cf765a8a35027aef916b0e812c28f72ea818"}
Jul 17 17:15:01.023629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.023602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerStarted","Data":"2f059375cfa21ef7f6dd0bc91ddb7731eb98a85c5240e6be4ec79676104f965d"}
Jul 17 17:15:01.023701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.023659    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerStarted","Data":"54bcbb052e7062fb63e670d5940ff9fe889bff160b9f77d00d2fa49dd3c45bd2"}
Jul 17 17:15:01.037436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.037397    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" podStartSLOduration=1.037384737 podStartE2EDuration="1.037384737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:01.035523804 +0000 UTC m=+2935.020611341" watchObservedRunningTime="2026-07-17 17:15:01.037384737 +0000 UTC m=+2935.022472274"
Jul 17 17:15:01.050785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.050742    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7" podStartSLOduration=1.05072904 podStartE2EDuration="1.05072904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:01.050583964 +0000 UTC m=+2935.035671511" watchObservedRunningTime="2026-07-17 17:15:01.05072904 +0000 UTC m=+2935.035816573"
Jul 17 17:15:01.065785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.065750    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" podStartSLOduration=1.065738681 podStartE2EDuration="1.065738681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:01.064851884 +0000 UTC m=+2935.049939423" watchObservedRunningTime="2026-07-17 17:15:01.065738681 +0000 UTC m=+2935.050826218"
Jul 17 17:15:01.080834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:01.080798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" podStartSLOduration=1.080786829 podStartE2EDuration="1.080786829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:01.080298784 +0000 UTC m=+2935.065386333" watchObservedRunningTime="2026-07-17 17:15:01.080786829 +0000 UTC m=+2935.065874366"
Jul 17 17:15:02.032364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.032328    2561 generic.go:356] "Generic (PLEG): container finished" podID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerID="3942395333a4c35cc22e9fa42151b5a63dd8351a5e13652f2a78f80b0189a5a7" exitCode=7
Jul 17 17:15:02.032827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.032411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerDied","Data":"3942395333a4c35cc22e9fa42151b5a63dd8351a5e13652f2a78f80b0189a5a7"}
Jul 17 17:15:02.032827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.032552    2561 scope.go:117] "RemoveContainer" containerID="3942395333a4c35cc22e9fa42151b5a63dd8351a5e13652f2a78f80b0189a5a7"
Jul 17 17:15:02.033923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.033900    2561 generic.go:356] "Generic (PLEG): container finished" podID="65d44964-1484-4263-9974-2d8796b3dfe7" containerID="2f059375cfa21ef7f6dd0bc91ddb7731eb98a85c5240e6be4ec79676104f965d" exitCode=7
Jul 17 17:15:02.034064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.033931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerDied","Data":"2f059375cfa21ef7f6dd0bc91ddb7731eb98a85c5240e6be4ec79676104f965d"}
Jul 17 17:15:02.034145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.034097    2561 scope.go:117] "RemoveContainer" containerID="2f059375cfa21ef7f6dd0bc91ddb7731eb98a85c5240e6be4ec79676104f965d"
Jul 17 17:15:02.035487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.035468    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerID="d65ca41e06780c710ff115deee43dd3d4bbf0ca5c7a50f0fe08258dd5dd6b393" exitCode=7
Jul 17 17:15:02.035568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.035530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerDied","Data":"d65ca41e06780c710ff115deee43dd3d4bbf0ca5c7a50f0fe08258dd5dd6b393"}
Jul 17 17:15:02.035655 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.035641    2561 scope.go:117] "RemoveContainer" containerID="d65ca41e06780c710ff115deee43dd3d4bbf0ca5c7a50f0fe08258dd5dd6b393"
Jul 17 17:15:02.037300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.037283    2561 generic.go:356] "Generic (PLEG): container finished" podID="c9b42c42-f817-4067-98b8-0a05402b316a" containerID="2abbf6ea4e52399a809fc783409cab85090654f11a0d67f818e7cf25bbadb518" exitCode=0
Jul 17 17:15:02.037384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:02.037315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7" event={"ID":"c9b42c42-f817-4067-98b8-0a05402b316a","Type":"ContainerDied","Data":"2abbf6ea4e52399a809fc783409cab85090654f11a0d67f818e7cf25bbadb518"}
Jul 17 17:15:03.043826 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.043799    2561 generic.go:356] "Generic (PLEG): container finished" podID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerID="c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e" exitCode=7
Jul 17 17:15:03.044253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.043857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerDied","Data":"c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e"}
Jul 17 17:15:03.044253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.043917    2561 scope.go:117] "RemoveContainer" containerID="3942395333a4c35cc22e9fa42151b5a63dd8351a5e13652f2a78f80b0189a5a7"
Jul 17 17:15:03.044253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.044061    2561 scope.go:117] "RemoveContainer" containerID="c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e"
Jul 17 17:15:03.044366 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:03.044268    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp_odh-ai-gateway-infra(c585b017-4c4f-4f45-b3ef-ece9613ec940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940"
Jul 17 17:15:03.045804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.045783    2561 generic.go:356] "Generic (PLEG): container finished" podID="65d44964-1484-4263-9974-2d8796b3dfe7" containerID="b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc" exitCode=7
Jul 17 17:15:03.045912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.045868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerDied","Data":"b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc"}
Jul 17 17:15:03.046025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.046010    2561 scope.go:117] "RemoveContainer" containerID="b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc"
Jul 17 17:15:03.046252 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:03.046233    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx_odh-ai-gateway-infra(65d44964-1484-4263-9974-2d8796b3dfe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" podUID="65d44964-1484-4263-9974-2d8796b3dfe7"
Jul 17 17:15:03.047701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.047682    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerID="a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a" exitCode=7
Jul 17 17:15:03.047805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.047712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerDied","Data":"a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a"}
Jul 17 17:15:03.047913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.047899    2561 scope.go:117] "RemoveContainer" containerID="a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a"
Jul 17 17:15:03.048148 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:03.048107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg_odh-ai-gateway-infra(4a32b4ab-b593-44eb-91aa-2a64a130470b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b"
Jul 17 17:15:03.055267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.055247    2561 scope.go:117] "RemoveContainer" containerID="2f059375cfa21ef7f6dd0bc91ddb7731eb98a85c5240e6be4ec79676104f965d"
Jul 17 17:15:03.067194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.067175    2561 scope.go:117] "RemoveContainer" containerID="d65ca41e06780c710ff115deee43dd3d4bbf0ca5c7a50f0fe08258dd5dd6b393"
Jul 17 17:15:03.180327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:03.180306    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"
Jul 17 17:15:04.056211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.056185    2561 scope.go:117] "RemoveContainer" containerID="a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a"
Jul 17 17:15:04.056748 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:04.056419    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg_odh-ai-gateway-infra(4a32b4ab-b593-44eb-91aa-2a64a130470b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b"
Jul 17 17:15:04.057299 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.057274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7"
Jul 17 17:15:04.057417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.057302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738475-w8wz7" event={"ID":"c9b42c42-f817-4067-98b8-0a05402b316a","Type":"ContainerDied","Data":"d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084"}
Jul 17 17:15:04.057417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.057345    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c9fd3e1ecdcdeb2d207ef92758fc7f4c0a98a1dc05178fb9d46ff838ed8084"
Jul 17 17:15:04.058975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.058956    2561 scope.go:117] "RemoveContainer" containerID="c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e"
Jul 17 17:15:04.059172 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:04.059151    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp_odh-ai-gateway-infra(c585b017-4c4f-4f45-b3ef-ece9613ec940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940"
Jul 17 17:15:04.060512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.060488    2561 scope.go:117] "RemoveContainer" containerID="b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc"
Jul 17 17:15:04.060637 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:04.060621    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx_odh-ai-gateway-infra(65d44964-1484-4263-9974-2d8796b3dfe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" podUID="65d44964-1484-4263-9974-2d8796b3dfe7"
Jul 17 17:15:04.509291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:04.509199    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:15:04.509461 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:04.509406    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:15:05.335313 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.335283    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:05.335690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.335618    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b42c42-f817-4067-98b8-0a05402b316a" containerName="cleanup"
Jul 17 17:15:05.335690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.335630    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b42c42-f817-4067-98b8-0a05402b316a" containerName="cleanup"
Jul 17 17:15:05.335763 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.335702    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b42c42-f817-4067-98b8-0a05402b316a" containerName="cleanup"
Jul 17 17:15:05.338243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.338228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:05.346381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.346357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:05.457508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.457480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth\" (UID: \"031e7c5f-d0af-4fcb-8854-f2b362753c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:05.558088 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.558038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth\" (UID: \"031e7c5f-d0af-4fcb-8854-f2b362753c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:05.558643 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.558621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth\" (UID: \"031e7c5f-d0af-4fcb-8854-f2b362753c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:05.649015 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.648942    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:05.775281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:05.775259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:05.777115 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:05.777087    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031e7c5f_d0af_4fcb_8854_f2b362753c70.slice/crio-e10226cfe5057eeb5c35df559fa377223f5a1ada2f7b7eb125f5e44173c34278 WatchSource:0}: Error finding container e10226cfe5057eeb5c35df559fa377223f5a1ada2f7b7eb125f5e44173c34278: Status 404 returned error can't find the container with id e10226cfe5057eeb5c35df559fa377223f5a1ada2f7b7eb125f5e44173c34278
Jul 17 17:15:06.071955 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:06.071924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerStarted","Data":"cc0c993ed51fee47fa1cbba5663e0a424053a08b16ffa37e877433828326954d"}
Jul 17 17:15:06.072072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:06.071979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerStarted","Data":"e10226cfe5057eeb5c35df559fa377223f5a1ada2f7b7eb125f5e44173c34278"}
Jul 17 17:15:06.091221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:06.091178    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" podStartSLOduration=1.091164325 podStartE2EDuration="1.091164325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:06.089559193 +0000 UTC m=+2940.074646732" watchObservedRunningTime="2026-07-17 17:15:06.091164325 +0000 UTC m=+2940.076251939"
Jul 17 17:15:07.077646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:07.077613    2561 generic.go:356] "Generic (PLEG): container finished" podID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerID="cc0c993ed51fee47fa1cbba5663e0a424053a08b16ffa37e877433828326954d" exitCode=7
Jul 17 17:15:07.078130 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:07.077662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerDied","Data":"cc0c993ed51fee47fa1cbba5663e0a424053a08b16ffa37e877433828326954d"}
Jul 17 17:15:07.078130 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:07.077880    2561 scope.go:117] "RemoveContainer" containerID="cc0c993ed51fee47fa1cbba5663e0a424053a08b16ffa37e877433828326954d"
Jul 17 17:15:08.084353 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:08.084323    2561 generic.go:356] "Generic (PLEG): container finished" podID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerID="ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd" exitCode=7
Jul 17 17:15:08.084705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:08.084402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerDied","Data":"ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd"}
Jul 17 17:15:08.084705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:08.084477    2561 scope.go:117] "RemoveContainer" containerID="cc0c993ed51fee47fa1cbba5663e0a424053a08b16ffa37e877433828326954d"
Jul 17 17:15:08.084705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:08.084618    2561 scope.go:117] "RemoveContainer" containerID="ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd"
Jul 17 17:15:08.084841 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:08.084794    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth_odh-ai-gateway-infra(031e7c5f-d0af-4fcb-8854-f2b362753c70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70"
Jul 17 17:15:09.092306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:09.092275    2561 scope.go:117] "RemoveContainer" containerID="ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd"
Jul 17 17:15:09.092699 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:09.092442    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth_odh-ai-gateway-infra(031e7c5f-d0af-4fcb-8854-f2b362753c70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70"
Jul 17 17:15:13.096462 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.096418    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:13.101521 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.101492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:13.113187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.113158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:13.216883 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.216852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx\" (UID: \"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:13.317863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.317828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx\" (UID: \"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:13.318436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.318416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx\" (UID: \"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:13.417003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.416944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:13.543506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:13.543484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:13.545389 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:13.545359    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93c5fdc_aa6e_467a_adfc_bf4a012e7ae5.slice/crio-8f065aeb5df6fd55f980204206b2b29de70c336225dd517dba8d842f9641fc90 WatchSource:0}: Error finding container 8f065aeb5df6fd55f980204206b2b29de70c336225dd517dba8d842f9641fc90: Status 404 returned error can't find the container with id 8f065aeb5df6fd55f980204206b2b29de70c336225dd517dba8d842f9641fc90
Jul 17 17:15:14.128142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:14.128115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerStarted","Data":"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"}
Jul 17 17:15:14.128440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:14.128171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerStarted","Data":"8f065aeb5df6fd55f980204206b2b29de70c336225dd517dba8d842f9641fc90"}
Jul 17 17:15:14.128440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:14.128360    2561 scope.go:117] "RemoveContainer" containerID="50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"
Jul 17 17:15:14.508791 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:14.508770    2561 scope.go:117] "RemoveContainer" containerID="a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a"
Jul 17 17:15:14.508946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:14.508932    2561 scope.go:117] "RemoveContainer" containerID="2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6"
Jul 17 17:15:15.134850 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.134825    2561 generic.go:356] "Generic (PLEG): container finished" podID="87d26ead-7864-44e8-8688-6d8ac9878313" containerID="b9c7f2a09003aeebfe5a800a0e62d2d607eeb1a7e031ea1bef903d885d3586a1" exitCode=7
Jul 17 17:15:15.135205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.134899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerDied","Data":"b9c7f2a09003aeebfe5a800a0e62d2d607eeb1a7e031ea1bef903d885d3586a1"}
Jul 17 17:15:15.135205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.134967    2561 scope.go:117] "RemoveContainer" containerID="2f2ac70b4c5a1ddb2e5961db2a14608a5b8c2e7d2d2f85fc5f60c0ced5f5c9a6"
Jul 17 17:15:15.135294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.135209    2561 scope.go:117] "RemoveContainer" containerID="b9c7f2a09003aeebfe5a800a0e62d2d607eeb1a7e031ea1bef903d885d3586a1"
Jul 17 17:15:15.135438 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:15.135419    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-db2g4_odh-ai-gateway-infra(87d26ead-7864-44e8-8688-6d8ac9878313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" podUID="87d26ead-7864-44e8-8688-6d8ac9878313"
Jul 17 17:15:15.136719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.136699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerStarted","Data":"1c343a176ebe6dc843474423597b9910588398eb9e2fed804cc933c03db46ef0"}
Jul 17 17:15:15.138340 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.138321    2561 generic.go:356] "Generic (PLEG): container finished" podID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerID="50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244" exitCode=7
Jul 17 17:15:15.138340 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.138339    2561 generic.go:356] "Generic (PLEG): container finished" podID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerID="a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52" exitCode=7
Jul 17 17:15:15.138502 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.138377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerDied","Data":"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"}
Jul 17 17:15:15.138502 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.138420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerDied","Data":"a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52"}
Jul 17 17:15:15.138616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.138594    2561 scope.go:117] "RemoveContainer" containerID="a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52"
Jul 17 17:15:15.138789 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:15.138770    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx_odh-ai-gateway-infra(d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5"
Jul 17 17:15:15.164095 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.164075    2561 scope.go:117] "RemoveContainer" containerID="50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"
Jul 17 17:15:15.175288 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.175267    2561 scope.go:117] "RemoveContainer" containerID="50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"
Jul 17 17:15:15.175527 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:15.175509    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244\": container with ID starting with 50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244 not found: ID does not exist" containerID="50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"
Jul 17 17:15:15.175575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:15.175534    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244"} err="failed to get container status \"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244\": rpc error: code = NotFound desc = could not find container \"50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244\": container with ID starting with 50ffdd412217c3e090d0aa592483d5e79ea5f699bdbefeeb4cc15cea897f8244 not found: ID does not exist"
Jul 17 17:15:16.146920 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.146889    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerID="1c343a176ebe6dc843474423597b9910588398eb9e2fed804cc933c03db46ef0" exitCode=7
Jul 17 17:15:16.147345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.146959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerDied","Data":"1c343a176ebe6dc843474423597b9910588398eb9e2fed804cc933c03db46ef0"}
Jul 17 17:15:16.147345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.147013    2561 scope.go:117] "RemoveContainer" containerID="a9d62796d013a67c5b4c5675226ab1b65ad29f8a3782dca04f714d120a41d84a"
Jul 17 17:15:16.147345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.147253    2561 scope.go:117] "RemoveContainer" containerID="1c343a176ebe6dc843474423597b9910588398eb9e2fed804cc933c03db46ef0"
Jul 17 17:15:16.147511 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:16.147455    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg_odh-ai-gateway-infra(4a32b4ab-b593-44eb-91aa-2a64a130470b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b"
Jul 17 17:15:16.148667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.148650    2561 scope.go:117] "RemoveContainer" containerID="a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52"
Jul 17 17:15:16.148794 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:16.148778    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx_odh-ai-gateway-infra(d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5"
Jul 17 17:15:16.160657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.160637    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:15:16.193810 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.193783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:16.290629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.290608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:15:16.345378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.345356    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca\") pod \"87d26ead-7864-44e8-8688-6d8ac9878313\" (UID: \"87d26ead-7864-44e8-8688-6d8ac9878313\") "
Jul 17 17:15:16.345707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.345684    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87d26ead-7864-44e8-8688-6d8ac9878313" (UID: "87d26ead-7864-44e8-8688-6d8ac9878313"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:15:16.446413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.446345    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d26ead-7864-44e8-8688-6d8ac9878313-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:15:16.512280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.512253    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:15:16.512422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:16.512309    2561 scope.go:117] "RemoveContainer" containerID="b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc"
Jul 17 17:15:16.512513 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:16.512493    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:15:17.154529 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.154507    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"
Jul 17 17:15:17.154969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.154505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4" event={"ID":"87d26ead-7864-44e8-8688-6d8ac9878313","Type":"ContainerDied","Data":"e4b7a8f47a9d6b11f8c64353ea1d684e96e2b804d4804d7d37cb4f352d090912"}
Jul 17 17:15:17.154969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.154668    2561 scope.go:117] "RemoveContainer" containerID="b9c7f2a09003aeebfe5a800a0e62d2d607eeb1a7e031ea1bef903d885d3586a1"
Jul 17 17:15:17.156533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.156492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerStarted","Data":"e0168b63c952d36c43b6becd02274b38c88d5d4ca1af910e2738054a40346cbc"}
Jul 17 17:15:17.187100 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.187029    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:15:17.190366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.190343    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-db2g4"]
Jul 17 17:15:17.289205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:17.289182    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"
Jul 17 17:15:18.165517 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.165488    2561 generic.go:356] "Generic (PLEG): container finished" podID="65d44964-1484-4263-9974-2d8796b3dfe7" containerID="e0168b63c952d36c43b6becd02274b38c88d5d4ca1af910e2738054a40346cbc" exitCode=7
Jul 17 17:15:18.165991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.165568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerDied","Data":"e0168b63c952d36c43b6becd02274b38c88d5d4ca1af910e2738054a40346cbc"}
Jul 17 17:15:18.165991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.165638    2561 scope.go:117] "RemoveContainer" containerID="b09a00530cfd411585f3c2e10c7e4416b769b4ce6db940df0152b18e1f37aebc"
Jul 17 17:15:18.165991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.165850    2561 scope.go:117] "RemoveContainer" containerID="e0168b63c952d36c43b6becd02274b38c88d5d4ca1af910e2738054a40346cbc"
Jul 17 17:15:18.166197 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:18.166066    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx_odh-ai-gateway-infra(65d44964-1484-4263-9974-2d8796b3dfe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" podUID="65d44964-1484-4263-9974-2d8796b3dfe7"
Jul 17 17:15:18.166877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.166784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg" event={"ID":"4a32b4ab-b593-44eb-91aa-2a64a130470b","Type":"ContainerDied","Data":"9ddd711224eafd29624a3a0a1e9489e29a38e56cf7449d26b2dd44e695bceeca"}
Jul 17 17:15:18.166877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.166805    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"
Jul 17 17:15:18.176590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.176574    2561 scope.go:117] "RemoveContainer" containerID="1c343a176ebe6dc843474423597b9910588398eb9e2fed804cc933c03db46ef0"
Jul 17 17:15:18.188477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.188453    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:18.192701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.192680    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:18.197883 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.197862    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4cb0f3bd-29738475-jzwhg"]
Jul 17 17:15:18.513888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.513817    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" path="/var/lib/kubelet/pods/4a32b4ab-b593-44eb-91aa-2a64a130470b/volumes"
Jul 17 17:15:18.514148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:18.514134    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" path="/var/lib/kubelet/pods/87d26ead-7864-44e8-8688-6d8ac9878313/volumes"
Jul 17 17:15:19.303621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:19.303602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"
Jul 17 17:15:19.508537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:19.508470    2561 scope.go:117] "RemoveContainer" containerID="c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e"
Jul 17 17:15:20.180460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.180440    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"
Jul 17 17:15:20.180570 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.180485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx" event={"ID":"65d44964-1484-4263-9974-2d8796b3dfe7","Type":"ContainerDied","Data":"54bcbb052e7062fb63e670d5940ff9fe889bff160b9f77d00d2fa49dd3c45bd2"}
Jul 17 17:15:20.180570 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.180545    2561 scope.go:117] "RemoveContainer" containerID="e0168b63c952d36c43b6becd02274b38c88d5d4ca1af910e2738054a40346cbc"
Jul 17 17:15:20.182555 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.182535    2561 generic.go:356] "Generic (PLEG): container finished" podID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerID="8a68b443148c3e1ab8ceee6854c114b7d17b2a202cb11cc87108ad83b54377be" exitCode=7
Jul 17 17:15:20.182646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.182576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerDied","Data":"8a68b443148c3e1ab8ceee6854c114b7d17b2a202cb11cc87108ad83b54377be"}
Jul 17 17:15:20.182896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.182881    2561 scope.go:117] "RemoveContainer" containerID="8a68b443148c3e1ab8ceee6854c114b7d17b2a202cb11cc87108ad83b54377be"
Jul 17 17:15:20.183138 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:20.183113    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp_odh-ai-gateway-infra(c585b017-4c4f-4f45-b3ef-ece9613ec940)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940"
Jul 17 17:15:20.189721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.189704    2561 scope.go:117] "RemoveContainer" containerID="c93053874e58123544c10e845e1956f3a752ef8e6c93244464a9d93afaa64a8e"
Jul 17 17:15:20.215371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.215351    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:20.222089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.219558    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-8c607776-29738475-jrpbx"]
Jul 17 17:15:20.513436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:20.513368    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" path="/var/lib/kubelet/pods/65d44964-1484-4263-9974-2d8796b3dfe7/volumes"
Jul 17 17:15:21.214024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:21.213995    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:21.354351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:21.354328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"
Jul 17 17:15:22.196347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.196320    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"
Jul 17 17:15:22.196739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.196355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp" event={"ID":"c585b017-4c4f-4f45-b3ef-ece9613ec940","Type":"ContainerDied","Data":"7d58744f13622cf9c792104c6ee4cf765a8a35027aef916b0e812c28f72ea818"}
Jul 17 17:15:22.196739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.196416    2561 scope.go:117] "RemoveContainer" containerID="8a68b443148c3e1ab8ceee6854c114b7d17b2a202cb11cc87108ad83b54377be"
Jul 17 17:15:22.219832 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.219809    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:22.228066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.228031    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738475-k28lp"]
Jul 17 17:15:22.508790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.508727    2561 scope.go:117] "RemoveContainer" containerID="ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd"
Jul 17 17:15:22.514125 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:22.514104    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" path="/var/lib/kubelet/pods/c585b017-4c4f-4f45-b3ef-ece9613ec940/volumes"
Jul 17 17:15:23.203220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:23.203137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerStarted","Data":"0ffb608ffb9b6ff9a77899d69e7e13695b11fc5d8fedfd12cb6dbbdb2b7d1abe"}
Jul 17 17:15:23.254932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:23.254896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:23.427500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:23.427473    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:24.213238 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.213205    2561 generic.go:356] "Generic (PLEG): container finished" podID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerID="0ffb608ffb9b6ff9a77899d69e7e13695b11fc5d8fedfd12cb6dbbdb2b7d1abe" exitCode=7
Jul 17 17:15:24.213644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.213272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerDied","Data":"0ffb608ffb9b6ff9a77899d69e7e13695b11fc5d8fedfd12cb6dbbdb2b7d1abe"}
Jul 17 17:15:24.213644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.213331    2561 scope.go:117] "RemoveContainer" containerID="ff2874f5b5bbc01e2a19843788c97a692ee2ccc7d72ac5c23502a24b297267cd"
Jul 17 17:15:24.213644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.213577    2561 scope.go:117] "RemoveContainer" containerID="0ffb608ffb9b6ff9a77899d69e7e13695b11fc5d8fedfd12cb6dbbdb2b7d1abe"
Jul 17 17:15:24.213835 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:24.213762    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth_odh-ai-gateway-infra(031e7c5f-d0af-4fcb-8854-f2b362753c70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70"
Jul 17 17:15:24.259779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.259756    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:24.576566 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.576532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:24.646117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646094    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"]
Jul 17 17:15:24.646876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646852    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646886    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646898    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646906    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646924    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646931    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646955    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646974    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646982    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.646993    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647000    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647014    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647021    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647066    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647075    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647094    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647101    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647109    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647118    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647254    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647268    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647311    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647332    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647339    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:24.647588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.647366    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:24.651761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.651732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.655408 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.655385    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0d4fd87f-serving-cert\""
Jul 17 17:15:24.659455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.658421    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"]
Jul 17 17:15:24.712549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.712524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eed73c2-e0b6-436d-9572-ecbcf429039a-openshift-service-ca\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.712698 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.712584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eed73c2-e0b6-436d-9572-ecbcf429039a-maas-api-tls\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.712698 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.712629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/9eed73c2-e0b6-436d-9572-ecbcf429039a-kube-api-access-6d5kv\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.724381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.724353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:24.813962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.813936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eed73c2-e0b6-436d-9572-ecbcf429039a-openshift-service-ca\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.814116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.813979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eed73c2-e0b6-436d-9572-ecbcf429039a-maas-api-tls\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.814116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.813997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/9eed73c2-e0b6-436d-9572-ecbcf429039a-kube-api-access-6d5kv\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.814549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.814528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eed73c2-e0b6-436d-9572-ecbcf429039a-openshift-service-ca\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.816321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.816298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eed73c2-e0b6-436d-9572-ecbcf429039a-maas-api-tls\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.823209 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.823185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5kv\" (UniqueName: \"kubernetes.io/projected/9eed73c2-e0b6-436d-9572-ecbcf429039a-kube-api-access-6d5kv\") pod \"maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn\" (UID: \"9eed73c2-e0b6-436d-9572-ecbcf429039a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:24.845343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.845290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:24.968575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:24.968558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:25.101343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.101318    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"]
Jul 17 17:15:25.102687 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:25.102663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eed73c2_e0b6_436d_9572_ecbcf429039a.slice/crio-f898c3c7bcef977cc0e84ef1a153b1036defc6dae9ea16336facafe4f64033cb WatchSource:0}: Error finding container f898c3c7bcef977cc0e84ef1a153b1036defc6dae9ea16336facafe4f64033cb: Status 404 returned error can't find the container with id f898c3c7bcef977cc0e84ef1a153b1036defc6dae9ea16336facafe4f64033cb
Jul 17 17:15:25.220451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.220425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerStarted","Data":"f898c3c7bcef977cc0e84ef1a153b1036defc6dae9ea16336facafe4f64033cb"}
Jul 17 17:15:25.313180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.313153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:25.334542 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.334522    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:25.407871 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.407804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:25.420318 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.420292    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca\") pod \"031e7c5f-d0af-4fcb-8854-f2b362753c70\" (UID: \"031e7c5f-d0af-4fcb-8854-f2b362753c70\") "
Jul 17 17:15:25.420617 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.420596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "031e7c5f-d0af-4fcb-8854-f2b362753c70" (UID: "031e7c5f-d0af-4fcb-8854-f2b362753c70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:15:25.521720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:25.521692    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e7c5f-d0af-4fcb-8854-f2b362753c70-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:15:26.227398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.227308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerStarted","Data":"cb0d873a4ffcf7548cea90ff48dd527de06128ba8994c204d53b32c19b407c8c"}
Jul 17 17:15:26.227398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.227365    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:26.228590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.228566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth" event={"ID":"031e7c5f-d0af-4fcb-8854-f2b362753c70","Type":"ContainerDied","Data":"e10226cfe5057eeb5c35df559fa377223f5a1ada2f7b7eb125f5e44173c34278"}
Jul 17 17:15:26.228721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.228571    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"
Jul 17 17:15:26.228721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.228611    2561 scope.go:117] "RemoveContainer" containerID="0ffb608ffb9b6ff9a77899d69e7e13695b11fc5d8fedfd12cb6dbbdb2b7d1abe"
Jul 17 17:15:26.248249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.248194    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podStartSLOduration=1.430477614 podStartE2EDuration="2.248174945s" totalImagesPullingTime="817.697331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:15:26.245459245 +0000 UTC m=+2960.230546775" watchObservedRunningTime="2026-07-17 17:15:26.248174945 +0000 UTC m=+2960.233262479"
Jul 17 17:15:26.265969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.265949    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:26.271401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.271380    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-mskth"]
Jul 17 17:15:26.514356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:26.514294    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" path="/var/lib/kubelet/pods/031e7c5f-d0af-4fcb-8854-f2b362753c70/volumes"
Jul 17 17:15:27.064221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:27.064192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:15:27.234783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:27.234749    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="cb0d873a4ffcf7548cea90ff48dd527de06128ba8994c204d53b32c19b407c8c" exitCode=1
Jul 17 17:15:27.235280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:27.234790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"cb0d873a4ffcf7548cea90ff48dd527de06128ba8994c204d53b32c19b407c8c"}
Jul 17 17:15:27.235280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:27.235097    2561 scope.go:117] "RemoveContainer" containerID="cb0d873a4ffcf7548cea90ff48dd527de06128ba8994c204d53b32c19b407c8c"
Jul 17 17:15:27.509143 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:27.509080    2561 scope.go:117] "RemoveContainer" containerID="a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52"
Jul 17 17:15:28.240886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.240858    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95" exitCode=1
Jul 17 17:15:28.241292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.240971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"}
Jul 17 17:15:28.241292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.241068    2561 scope.go:117] "RemoveContainer" containerID="cb0d873a4ffcf7548cea90ff48dd527de06128ba8994c204d53b32c19b407c8c"
Jul 17 17:15:28.241415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.241293    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:28.241520 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:28.241499    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:28.242932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.242912    2561 generic.go:356] "Generic (PLEG): container finished" podID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerID="8bcb70711b16b1993eededec486f0604f9ce3d08c87e271bc03c76bdce2ab440" exitCode=7
Jul 17 17:15:28.243066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.242984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerDied","Data":"8bcb70711b16b1993eededec486f0604f9ce3d08c87e271bc03c76bdce2ab440"}
Jul 17 17:15:28.243271 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.243256    2561 scope.go:117] "RemoveContainer" containerID="8bcb70711b16b1993eededec486f0604f9ce3d08c87e271bc03c76bdce2ab440"
Jul 17 17:15:28.243446 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:28.243428    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx_odh-ai-gateway-infra(d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5"
Jul 17 17:15:28.251409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.251395    2561 scope.go:117] "RemoveContainer" containerID="a4652df19339bb4fba20e03a28c3e9962d7f924b855e0128aab099649a4e2b52"
Jul 17 17:15:28.508735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:28.508714    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:15:28.508985 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:28.508945    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:15:29.250486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.250455    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:29.250841 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:29.250608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:29.290567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.290543    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:29.432310 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.432285    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:29.551818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.551791    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca\") pod \"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5\" (UID: \"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5\") "
Jul 17 17:15:29.552139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.552120    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" (UID: "d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:15:29.652604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:29.652576    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:15:30.257254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.257225    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"
Jul 17 17:15:30.257633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.257258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx" event={"ID":"d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5","Type":"ContainerDied","Data":"8f065aeb5df6fd55f980204206b2b29de70c336225dd517dba8d842f9641fc90"}
Jul 17 17:15:30.257633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.257311    2561 scope.go:117] "RemoveContainer" containerID="8bcb70711b16b1993eededec486f0604f9ce3d08c87e271bc03c76bdce2ab440"
Jul 17 17:15:30.279603 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.279581    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:30.285113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.285092    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qnhzx"]
Jul 17 17:15:30.513442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:30.513379    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" path="/var/lib/kubelet/pods/d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5/volumes"
Jul 17 17:15:31.227850 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:31.227821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:31.228161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:31.228148    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:31.228318 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:31.228300    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:34.969215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:34.969179    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:34.969602 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:34.969512    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:34.969702 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:34.969682    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:39.508389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:39.508361    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:15:39.508820 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:39.508564    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:15:44.856913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:44.856879    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:44.957237 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:44.957199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:48.508457 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:48.508422    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:48.720605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:48.720574    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:15:49.343718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:49.343686    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31" exitCode=1
Jul 17 17:15:49.343886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:49.343728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"}
Jul 17 17:15:49.343886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:49.343779    2561 scope.go:117] "RemoveContainer" containerID="473c6f3a1d59c46fb52bb1f5c3ec6ccf4c1e217f719966e1c88369b201d3ab95"
Jul 17 17:15:49.344134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:49.344118    2561 scope.go:117] "RemoveContainer" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"
Jul 17 17:15:49.344309 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:49.344290    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:51.228194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:51.228157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:51.228636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:51.228515    2561 scope.go:117] "RemoveContainer" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"
Jul 17 17:15:51.228714 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:51.228695    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:54.509118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:54.509086    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:15:54.509495 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:54.509271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:15:54.969670 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:54.969637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:15:54.969994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:54.969980    2561 scope.go:117] "RemoveContainer" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"
Jul 17 17:15:54.970186 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:54.970168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:15:55.221598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221517    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:15:55.221871 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221858    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221872    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221886    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221891    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221899    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221905    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:55.221914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221912    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221918    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221927    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221932    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221962    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.221967    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222014    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a32b4ab-b593-44eb-91aa-2a64a130470b" containerName="cleanup"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222022    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222029    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222034    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222040    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d44964-1484-4263-9974-2d8796b3dfe7" containerName="cleanup"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222061    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c585b017-4c4f-4f45-b3ef-ece9613ec940" containerName="cleanup"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222068    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d26ead-7864-44e8-8688-6d8ac9878313" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222077    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.222154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.222086    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="031e7c5f-d0af-4fcb-8854-f2b362753c70" containerName="revoke-keys"
Jul 17 17:15:55.226341 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.226325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:15:55.228929 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.228895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:15:55.233831 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.233801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:15:55.266100 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.266067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-pjhqj\" (UID: \"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:15:55.366577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.366547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-pjhqj\" (UID: \"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:15:55.367138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.367121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-pjhqj\" (UID: \"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:15:55.537676 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.537648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:15:55.667252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:55.667230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:15:55.670015 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:15:55.669990    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37717ba3_8d6e_4897_8e9e_a0cfd209d9c5.slice/crio-12ade349c99534cf8f11bcb9a5c825c7a258c405e20f22cd657f513ae3051133 WatchSource:0}: Error finding container 12ade349c99534cf8f11bcb9a5c825c7a258c405e20f22cd657f513ae3051133: Status 404 returned error can't find the container with id 12ade349c99534cf8f11bcb9a5c825c7a258c405e20f22cd657f513ae3051133
Jul 17 17:15:56.379376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:56.379343    2561 generic.go:356] "Generic (PLEG): container finished" podID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerID="32c694209155c3344e8d091d3b349552b3fe05adfc7b24d82ccd094e19ca9675" exitCode=7
Jul 17 17:15:56.379549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:56.379383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" event={"ID":"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5","Type":"ContainerDied","Data":"32c694209155c3344e8d091d3b349552b3fe05adfc7b24d82ccd094e19ca9675"}
Jul 17 17:15:56.379549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:56.379427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" event={"ID":"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5","Type":"ContainerStarted","Data":"12ade349c99534cf8f11bcb9a5c825c7a258c405e20f22cd657f513ae3051133"}
Jul 17 17:15:56.379632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:56.379619    2561 scope.go:117] "RemoveContainer" containerID="32c694209155c3344e8d091d3b349552b3fe05adfc7b24d82ccd094e19ca9675"
Jul 17 17:15:57.385245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:57.385208    2561 generic.go:356] "Generic (PLEG): container finished" podID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerID="0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df" exitCode=7
Jul 17 17:15:57.385663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:57.385294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" event={"ID":"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5","Type":"ContainerDied","Data":"0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df"}
Jul 17 17:15:57.385663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:57.385388    2561 scope.go:117] "RemoveContainer" containerID="32c694209155c3344e8d091d3b349552b3fe05adfc7b24d82ccd094e19ca9675"
Jul 17 17:15:57.385663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:57.385540    2561 scope.go:117] "RemoveContainer" containerID="0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df"
Jul 17 17:15:57.385821 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:57.385747    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-pjhqj_odh-ai-gateway-infra(37717ba3-8d6e-4897-8e9e-a0cfd209d9c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5"
Jul 17 17:15:58.391369 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:15:58.391345    2561 scope.go:117] "RemoveContainer" containerID="0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df"
Jul 17 17:15:58.391746 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:15:58.391535    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-pjhqj_odh-ai-gateway-infra(37717ba3-8d6e-4897-8e9e-a0cfd209d9c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5"
Jul 17 17:16:03.152164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.152128    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:03.152539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.152480    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:16:03.152539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.152490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:16:03.152610 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.152561    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93c5fdc-aa6e-467a-adfc-bf4a012e7ae5" containerName="revoke-keys"
Jul 17 17:16:03.156770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.156754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:03.166018 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.165980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:03.234174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.234131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f\" (UID: \"cb3e37bc-539c-47af-af2e-e39619d1ea11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:03.335189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.335155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f\" (UID: \"cb3e37bc-539c-47af-af2e-e39619d1ea11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:03.335721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.335704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f\" (UID: \"cb3e37bc-539c-47af-af2e-e39619d1ea11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:03.475787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.475719    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:03.603487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:03.603460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:03.605229 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:16:03.605202    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3e37bc_539c_47af_af2e_e39619d1ea11.slice/crio-58db6135aa1fa090592ab8f27c68819d172a6a8c43ea119918f8bd29648b3a12 WatchSource:0}: Error finding container 58db6135aa1fa090592ab8f27c68819d172a6a8c43ea119918f8bd29648b3a12: Status 404 returned error can't find the container with id 58db6135aa1fa090592ab8f27c68819d172a6a8c43ea119918f8bd29648b3a12
Jul 17 17:16:04.421400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:04.421366    2561 generic.go:356] "Generic (PLEG): container finished" podID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerID="94637b822c572fef6e17bf0ae82c7b504b43117b1b8f08f36d2a8db739ea7b0b" exitCode=7
Jul 17 17:16:04.421792 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:04.421417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" event={"ID":"cb3e37bc-539c-47af-af2e-e39619d1ea11","Type":"ContainerDied","Data":"94637b822c572fef6e17bf0ae82c7b504b43117b1b8f08f36d2a8db739ea7b0b"}
Jul 17 17:16:04.421792 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:04.421464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" event={"ID":"cb3e37bc-539c-47af-af2e-e39619d1ea11","Type":"ContainerStarted","Data":"58db6135aa1fa090592ab8f27c68819d172a6a8c43ea119918f8bd29648b3a12"}
Jul 17 17:16:04.421792 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:04.421674    2561 scope.go:117] "RemoveContainer" containerID="94637b822c572fef6e17bf0ae82c7b504b43117b1b8f08f36d2a8db739ea7b0b"
Jul 17 17:16:05.381630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.381593    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:05.385082 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.385040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:05.393503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.393479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:05.427214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.427178    2561 generic.go:356] "Generic (PLEG): container finished" podID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerID="3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e" exitCode=7
Jul 17 17:16:05.427559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.427260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" event={"ID":"cb3e37bc-539c-47af-af2e-e39619d1ea11","Type":"ContainerDied","Data":"3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e"}
Jul 17 17:16:05.427559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.427316    2561 scope.go:117] "RemoveContainer" containerID="94637b822c572fef6e17bf0ae82c7b504b43117b1b8f08f36d2a8db739ea7b0b"
Jul 17 17:16:05.427559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.427488    2561 scope.go:117] "RemoveContainer" containerID="3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e"
Jul 17 17:16:05.427787 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:05.427762    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f_odh-ai-gateway-infra(cb3e37bc-539c-47af-af2e-e39619d1ea11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11"
Jul 17 17:16:05.453335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.453308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4\" (UID: \"977e018e-fe79-4caa-8473-9ec693dc3e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:05.509132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.509106    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:16:05.509336 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:05.509315    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:16:05.553936 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.553910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4\" (UID: \"977e018e-fe79-4caa-8473-9ec693dc3e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:05.554518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.554497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4\" (UID: \"977e018e-fe79-4caa-8473-9ec693dc3e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:05.695522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.695450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:05.823556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:05.823530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:05.825591 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:16:05.825566    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977e018e_fe79_4caa_8473_9ec693dc3e1e.slice/crio-cfce93e861affe343d7edf28ef40c30375f09ac56c93d802aececb9648d23cbb WatchSource:0}: Error finding container cfce93e861affe343d7edf28ef40c30375f09ac56c93d802aececb9648d23cbb: Status 404 returned error can't find the container with id cfce93e861affe343d7edf28ef40c30375f09ac56c93d802aececb9648d23cbb
Jul 17 17:16:06.433390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.433368    2561 scope.go:117] "RemoveContainer" containerID="3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e"
Jul 17 17:16:06.433715 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:06.433563    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f_odh-ai-gateway-infra(cb3e37bc-539c-47af-af2e-e39619d1ea11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11"
Jul 17 17:16:06.434630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.434607    2561 generic.go:356] "Generic (PLEG): container finished" podID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerID="07167e3907d5ca9825c8c74c760bd550338d0f31c7c1ba43a77e928b2bdbb557" exitCode=7
Jul 17 17:16:06.434702 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.434638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" event={"ID":"977e018e-fe79-4caa-8473-9ec693dc3e1e","Type":"ContainerDied","Data":"07167e3907d5ca9825c8c74c760bd550338d0f31c7c1ba43a77e928b2bdbb557"}
Jul 17 17:16:06.434702 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.434693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" event={"ID":"977e018e-fe79-4caa-8473-9ec693dc3e1e","Type":"ContainerStarted","Data":"cfce93e861affe343d7edf28ef40c30375f09ac56c93d802aececb9648d23cbb"}
Jul 17 17:16:06.434838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.434825    2561 scope.go:117] "RemoveContainer" containerID="07167e3907d5ca9825c8c74c760bd550338d0f31c7c1ba43a77e928b2bdbb557"
Jul 17 17:16:06.589059 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.589012    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:16:06.599103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:06.599083    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:16:07.440691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:07.440610    2561 generic.go:356] "Generic (PLEG): container finished" podID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerID="da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be" exitCode=7
Jul 17 17:16:07.441144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:07.440695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" event={"ID":"977e018e-fe79-4caa-8473-9ec693dc3e1e","Type":"ContainerDied","Data":"da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be"}
Jul 17 17:16:07.441144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:07.440747    2561 scope.go:117] "RemoveContainer" containerID="07167e3907d5ca9825c8c74c760bd550338d0f31c7c1ba43a77e928b2bdbb557"
Jul 17 17:16:07.441144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:07.440887    2561 scope.go:117] "RemoveContainer" containerID="da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be"
Jul 17 17:16:07.441144 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:07.441122    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4_odh-ai-gateway-infra(977e018e-fe79-4caa-8473-9ec693dc3e1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e"
Jul 17 17:16:08.446690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:08.446657    2561 scope.go:117] "RemoveContainer" containerID="da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be"
Jul 17 17:16:08.447098 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:08.446837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4_odh-ai-gateway-infra(977e018e-fe79-4caa-8473-9ec693dc3e1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e"
Jul 17 17:16:08.508324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:08.508297    2561 scope.go:117] "RemoveContainer" containerID="0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df"
Jul 17 17:16:09.452462 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:09.452376    2561 generic.go:356] "Generic (PLEG): container finished" podID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerID="2973932a066ed973cb7a2818ac845caa8b2400cfaae7fea98f3dc956949cb851" exitCode=7
Jul 17 17:16:09.452825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:09.452472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" event={"ID":"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5","Type":"ContainerDied","Data":"2973932a066ed973cb7a2818ac845caa8b2400cfaae7fea98f3dc956949cb851"}
Jul 17 17:16:09.452825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:09.452576    2561 scope.go:117] "RemoveContainer" containerID="0b884dd0ae1c470bee59e0c19918c999bda3784fd14d8755c11fd7500c5128df"
Jul 17 17:16:09.452988 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:09.452971    2561 scope.go:117] "RemoveContainer" containerID="2973932a066ed973cb7a2818ac845caa8b2400cfaae7fea98f3dc956949cb851"
Jul 17 17:16:09.453210 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:09.453192    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-pjhqj_odh-ai-gateway-infra(37717ba3-8d6e-4897-8e9e-a0cfd209d9c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5"
Jul 17 17:16:09.508217 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:09.508197    2561 scope.go:117] "RemoveContainer" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"
Jul 17 17:16:10.459571 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.459493    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa" exitCode=1
Jul 17 17:16:10.459571 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.459563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"}
Jul 17 17:16:10.460093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.459626    2561 scope.go:117] "RemoveContainer" containerID="609c60a2cc0b8aa94a7385bb240582036dde680e5f5c28280b0c1fe399c1fe31"
Jul 17 17:16:10.460093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.459978    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:10.460229 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:10.460191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:10.483231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.483207    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:16:10.614681 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.614657    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:16:10.704705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.704677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca\") pod \"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5\" (UID: \"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5\") "
Jul 17 17:16:10.705127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.705106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" (UID: "37717ba3-8d6e-4897-8e9e-a0cfd209d9c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:16:10.806213 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:10.806186    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:16:11.227773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.227683    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:16:11.467320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.467289    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"
Jul 17 17:16:11.467764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.467321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj" event={"ID":"37717ba3-8d6e-4897-8e9e-a0cfd209d9c5","Type":"ContainerDied","Data":"12ade349c99534cf8f11bcb9a5c825c7a258c405e20f22cd657f513ae3051133"}
Jul 17 17:16:11.467764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.467388    2561 scope.go:117] "RemoveContainer" containerID="2973932a066ed973cb7a2818ac845caa8b2400cfaae7fea98f3dc956949cb851"
Jul 17 17:16:11.469273 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.469251    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:11.469508 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:11.469484    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:11.505208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.505119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:16:11.506937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:11.506917    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-pjhqj"]
Jul 17 17:16:12.513234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:12.513198    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" path="/var/lib/kubelet/pods/37717ba3-8d6e-4897-8e9e-a0cfd209d9c5/volumes"
Jul 17 17:16:14.969174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:14.969140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:16:14.969638 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:14.969585    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:14.969810 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:14.969786    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:18.509198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:18.509171    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:16:18.509611 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:18.509359    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:16:20.532860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:20.532831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:16:20.630240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:20.630206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:16:21.508523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:21.508489    2561 scope.go:117] "RemoveContainer" containerID="3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e"
Jul 17 17:16:22.523008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:22.522928    2561 generic.go:356] "Generic (PLEG): container finished" podID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerID="5281a0f111272498b0ba5169687180f64ccce24ec1c16ff8e887f539cf51effe" exitCode=7
Jul 17 17:16:22.523414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:22.523000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" event={"ID":"cb3e37bc-539c-47af-af2e-e39619d1ea11","Type":"ContainerDied","Data":"5281a0f111272498b0ba5169687180f64ccce24ec1c16ff8e887f539cf51effe"}
Jul 17 17:16:22.523414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:22.523071    2561 scope.go:117] "RemoveContainer" containerID="3bd48e5227a4fecf8bbbc6ca7f03170872388362257cb40e2b668fd9edc13d6e"
Jul 17 17:16:22.523414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:22.523319    2561 scope.go:117] "RemoveContainer" containerID="5281a0f111272498b0ba5169687180f64ccce24ec1c16ff8e887f539cf51effe"
Jul 17 17:16:22.523583 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:22.523523    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f_odh-ai-gateway-infra(cb3e37bc-539c-47af-af2e-e39619d1ea11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11"
Jul 17 17:16:23.509007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.508977    2561 scope.go:117] "RemoveContainer" containerID="da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be"
Jul 17 17:16:23.550488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.550459    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:23.683445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.683416    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:23.828227 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.828205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca\") pod \"cb3e37bc-539c-47af-af2e-e39619d1ea11\" (UID: \"cb3e37bc-539c-47af-af2e-e39619d1ea11\") "
Jul 17 17:16:23.828541 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.828516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb3e37bc-539c-47af-af2e-e39619d1ea11" (UID: "cb3e37bc-539c-47af-af2e-e39619d1ea11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:16:23.929912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:23.929871    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e37bc-539c-47af-af2e-e39619d1ea11-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:16:24.536435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.536406    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"
Jul 17 17:16:24.536435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.536427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f" event={"ID":"cb3e37bc-539c-47af-af2e-e39619d1ea11","Type":"ContainerDied","Data":"58db6135aa1fa090592ab8f27c68819d172a6a8c43ea119918f8bd29648b3a12"}
Jul 17 17:16:24.536638 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.536491    2561 scope.go:117] "RemoveContainer" containerID="5281a0f111272498b0ba5169687180f64ccce24ec1c16ff8e887f539cf51effe"
Jul 17 17:16:24.538369 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.538342    2561 generic.go:356] "Generic (PLEG): container finished" podID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerID="3ff9bd3061e2ee9598d96bbe4e465d28002a65f24772b8457f5a3583fdee21c3" exitCode=7
Jul 17 17:16:24.538500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.538384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" event={"ID":"977e018e-fe79-4caa-8473-9ec693dc3e1e","Type":"ContainerDied","Data":"3ff9bd3061e2ee9598d96bbe4e465d28002a65f24772b8457f5a3583fdee21c3"}
Jul 17 17:16:24.538690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.538675    2561 scope.go:117] "RemoveContainer" containerID="3ff9bd3061e2ee9598d96bbe4e465d28002a65f24772b8457f5a3583fdee21c3"
Jul 17 17:16:24.538893 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:24.538873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4_odh-ai-gateway-infra(977e018e-fe79-4caa-8473-9ec693dc3e1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e"
Jul 17 17:16:24.545004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.544986    2561 scope.go:117] "RemoveContainer" containerID="da4aafb3d77eb18054b4d7abb74f18e3865dea7185426b07d57031d2bef719be"
Jul 17 17:16:24.566550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.566525    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:24.569915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:24.569895    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-m4c2f"]
Jul 17 17:16:25.568078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:25.567111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:25.701893 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:25.701860    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:25.845975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:25.845893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca\") pod \"977e018e-fe79-4caa-8473-9ec693dc3e1e\" (UID: \"977e018e-fe79-4caa-8473-9ec693dc3e1e\") "
Jul 17 17:16:25.846288 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:25.846266    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "977e018e-fe79-4caa-8473-9ec693dc3e1e" (UID: "977e018e-fe79-4caa-8473-9ec693dc3e1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:16:25.946956 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:25.946926    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977e018e-fe79-4caa-8473-9ec693dc3e1e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:16:26.510934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.510909    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:26.511160 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:26.511139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:26.514080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.514035    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" path="/var/lib/kubelet/pods/cb3e37bc-539c-47af-af2e-e39619d1ea11/volumes"
Jul 17 17:16:26.551658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.551632    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"
Jul 17 17:16:26.551799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.551632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4" event={"ID":"977e018e-fe79-4caa-8473-9ec693dc3e1e","Type":"ContainerDied","Data":"cfce93e861affe343d7edf28ef40c30375f09ac56c93d802aececb9648d23cbb"}
Jul 17 17:16:26.551799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.551769    2561 scope.go:117] "RemoveContainer" containerID="3ff9bd3061e2ee9598d96bbe4e465d28002a65f24772b8457f5a3583fdee21c3"
Jul 17 17:16:26.580278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.580244    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:26.584760 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:26.584732    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-462q4"]
Jul 17 17:16:27.277104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:27.277072    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:16:28.513527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:28.513496    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" path="/var/lib/kubelet/pods/977e018e-fe79-4caa-8473-9ec693dc3e1e/volumes"
Jul 17 17:16:28.677138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:28.677107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:16:29.509116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:29.509084    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:16:29.509307 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:29.509272    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:16:32.134733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:32.134697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:16:38.508910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:38.508879    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:38.509365 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:38.509107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:42.508754 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:42.508715    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:16:42.509263 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:42.508946    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:16:45.267769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.267736    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268094    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268106    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268114    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268120    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268128    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268135    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268151    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268157    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268162    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268167    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268175    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268180    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268195    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268200    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268247    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268254    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268260    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268267    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268272    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268278    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268283    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:16:45.268562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.268289    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:16:45.272561 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.272545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:16:45.274890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.274865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:16:45.277794 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.277772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:16:45.318697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.318662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mr976\" (UID: \"14fa5519-1474-449c-9dbb-c65fdd618554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:16:45.419370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.419333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mr976\" (UID: \"14fa5519-1474-449c-9dbb-c65fdd618554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:16:45.419952 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.419931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-mr976\" (UID: \"14fa5519-1474-449c-9dbb-c65fdd618554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:16:45.584127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.584093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:16:45.724970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:45.724941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:16:45.727176 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:16:45.727148    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fa5519_1474_449c_9dbb_c65fdd618554.slice/crio-aa857e4e5954135210dcf05a97068e9b0f6dfc29d6ff5de9f2b5f19fad75d652 WatchSource:0}: Error finding container aa857e4e5954135210dcf05a97068e9b0f6dfc29d6ff5de9f2b5f19fad75d652: Status 404 returned error can't find the container with id aa857e4e5954135210dcf05a97068e9b0f6dfc29d6ff5de9f2b5f19fad75d652
Jul 17 17:16:46.644634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:46.644548    2561 generic.go:356] "Generic (PLEG): container finished" podID="14fa5519-1474-449c-9dbb-c65fdd618554" containerID="a5f870941ce6bc7df9416475b6b0b272434653a589f8bd12b5d5817aebf1f865" exitCode=7
Jul 17 17:16:46.645022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:46.644629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" event={"ID":"14fa5519-1474-449c-9dbb-c65fdd618554","Type":"ContainerDied","Data":"a5f870941ce6bc7df9416475b6b0b272434653a589f8bd12b5d5817aebf1f865"}
Jul 17 17:16:46.645022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:46.644677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" event={"ID":"14fa5519-1474-449c-9dbb-c65fdd618554","Type":"ContainerStarted","Data":"aa857e4e5954135210dcf05a97068e9b0f6dfc29d6ff5de9f2b5f19fad75d652"}
Jul 17 17:16:46.645022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:46.644825    2561 scope.go:117] "RemoveContainer" containerID="a5f870941ce6bc7df9416475b6b0b272434653a589f8bd12b5d5817aebf1f865"
Jul 17 17:16:47.651223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:47.651127    2561 generic.go:356] "Generic (PLEG): container finished" podID="14fa5519-1474-449c-9dbb-c65fdd618554" containerID="ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac" exitCode=7
Jul 17 17:16:47.651223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:47.651162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" event={"ID":"14fa5519-1474-449c-9dbb-c65fdd618554","Type":"ContainerDied","Data":"ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac"}
Jul 17 17:16:47.651223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:47.651205    2561 scope.go:117] "RemoveContainer" containerID="a5f870941ce6bc7df9416475b6b0b272434653a589f8bd12b5d5817aebf1f865"
Jul 17 17:16:47.651780 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:47.651502    2561 scope.go:117] "RemoveContainer" containerID="ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac"
Jul 17 17:16:47.651780 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:47.651736    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mr976_odh-ai-gateway-infra(14fa5519-1474-449c-9dbb-c65fdd618554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" podUID="14fa5519-1474-449c-9dbb-c65fdd618554"
Jul 17 17:16:48.657343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:48.657313    2561 scope.go:117] "RemoveContainer" containerID="ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac"
Jul 17 17:16:48.657788 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:48.657494    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mr976_odh-ai-gateway-infra(14fa5519-1474-449c-9dbb-c65fdd618554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" podUID="14fa5519-1474-449c-9dbb-c65fdd618554"
Jul 17 17:16:53.508508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:53.508476    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:54.685200 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:54.685122    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6" exitCode=1
Jul 17 17:16:54.685557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:54.685195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"}
Jul 17 17:16:54.685557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:54.685256    2561 scope.go:117] "RemoveContainer" containerID="7de703bd1a745005a2ff8cd22da88c1d16653de574d15e1899413b3dcdcc15fa"
Jul 17 17:16:54.685656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:54.685629    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:16:54.685848 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:54.685823    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:54.968957 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:54.968870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:16:55.692153 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:55.692125    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:16:55.692529 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:55.692321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:56.227799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:56.227769    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:16:56.696253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:56.696224    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:16:56.696725 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:56.696451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:16:57.508187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:16:57.508156    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:16:57.508373 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:16:57.508338    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:17:01.508392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:01.508317    2561 scope.go:117] "RemoveContainer" containerID="ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac"
Jul 17 17:17:02.725505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:02.725469    2561 generic.go:356] "Generic (PLEG): container finished" podID="14fa5519-1474-449c-9dbb-c65fdd618554" containerID="13b3ddbd0c18b6cbc5dab7a7df19bd09033ce5ea1495d60bcbeff3ed10a00837" exitCode=7
Jul 17 17:17:02.725961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:02.725514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" event={"ID":"14fa5519-1474-449c-9dbb-c65fdd618554","Type":"ContainerDied","Data":"13b3ddbd0c18b6cbc5dab7a7df19bd09033ce5ea1495d60bcbeff3ed10a00837"}
Jul 17 17:17:02.725961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:02.725562    2561 scope.go:117] "RemoveContainer" containerID="ff9e7d81efe628d9fc5d234053669a963893c2f58eedacb3ae568da06ce69fac"
Jul 17 17:17:02.725961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:02.725827    2561 scope.go:117] "RemoveContainer" containerID="13b3ddbd0c18b6cbc5dab7a7df19bd09033ce5ea1495d60bcbeff3ed10a00837"
Jul 17 17:17:02.726114 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:02.726087    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-mr976_odh-ai-gateway-infra(14fa5519-1474-449c-9dbb-c65fdd618554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" podUID="14fa5519-1474-449c-9dbb-c65fdd618554"
Jul 17 17:17:03.196121 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196086    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:03.196452 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196439    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:17:03.196500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196454    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37717ba3-8d6e-4897-8e9e-a0cfd209d9c5" containerName="revoke-keys"
Jul 17 17:17:03.196500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:17:03.196500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196471    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3e37bc-539c-47af-af2e-e39619d1ea11" containerName="revoke-keys"
Jul 17 17:17:03.196599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.196545    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="977e018e-fe79-4caa-8473-9ec693dc3e1e" containerName="revoke-keys"
Jul 17 17:17:03.200728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.200708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:03.211442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.211415    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:03.287173 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.287139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52\" (UID: \"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:03.388033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.387994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52\" (UID: \"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:03.388621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.388599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52\" (UID: \"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:03.510928 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.510845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:03.648031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.648005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:03.650016 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:17:03.649990    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fdb4ea_1ea6_4990_954d_6b8bd73e5bc1.slice/crio-a8c6349fa5d218b97868246b194ee55f6d662b90c8f3c4554d8ce1ade6da27de WatchSource:0}: Error finding container a8c6349fa5d218b97868246b194ee55f6d662b90c8f3c4554d8ce1ade6da27de: Status 404 returned error can't find the container with id a8c6349fa5d218b97868246b194ee55f6d662b90c8f3c4554d8ce1ade6da27de
Jul 17 17:17:03.732422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.732389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" event={"ID":"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1","Type":"ContainerStarted","Data":"a8c6349fa5d218b97868246b194ee55f6d662b90c8f3c4554d8ce1ade6da27de"}
Jul 17 17:17:03.768439 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.768365    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:17:03.894994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.894970    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:17:03.993718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.993686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca\") pod \"14fa5519-1474-449c-9dbb-c65fdd618554\" (UID: \"14fa5519-1474-449c-9dbb-c65fdd618554\") "
Jul 17 17:17:03.994012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:03.993988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14fa5519-1474-449c-9dbb-c65fdd618554" (UID: "14fa5519-1474-449c-9dbb-c65fdd618554"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:17:04.094312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.094277    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fa5519-1474-449c-9dbb-c65fdd618554-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:17:04.738706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.738677    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"
Jul 17 17:17:04.738706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.738686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976" event={"ID":"14fa5519-1474-449c-9dbb-c65fdd618554","Type":"ContainerDied","Data":"aa857e4e5954135210dcf05a97068e9b0f6dfc29d6ff5de9f2b5f19fad75d652"}
Jul 17 17:17:04.739270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.738751    2561 scope.go:117] "RemoveContainer" containerID="13b3ddbd0c18b6cbc5dab7a7df19bd09033ce5ea1495d60bcbeff3ed10a00837"
Jul 17 17:17:04.740210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.740181    2561 generic.go:356] "Generic (PLEG): container finished" podID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerID="3093b458016af43dd231a343ae1ab6ef617fd0320aa8660bda35d1f2e3f9992f" exitCode=7
Jul 17 17:17:04.740336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.740273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" event={"ID":"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1","Type":"ContainerDied","Data":"3093b458016af43dd231a343ae1ab6ef617fd0320aa8660bda35d1f2e3f9992f"}
Jul 17 17:17:04.740458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.740444    2561 scope.go:117] "RemoveContainer" containerID="3093b458016af43dd231a343ae1ab6ef617fd0320aa8660bda35d1f2e3f9992f"
Jul 17 17:17:04.770848 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.770820    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:17:04.773901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:04.773878    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-mr976"]
Jul 17 17:17:05.434117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434079    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:05.434508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434511    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434532    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434540    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434556 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434609    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.434708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.434615    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:05.437565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.437549    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:05.444819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.444681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:05.507639 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.507545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr\" (UID: \"cdfa9455-a09c-499d-9a9f-a630c977ef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:05.608715 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.608676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr\" (UID: \"cdfa9455-a09c-499d-9a9f-a630c977ef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:05.609347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.609329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr\" (UID: \"cdfa9455-a09c-499d-9a9f-a630c977ef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:05.747653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.747617    2561 generic.go:356] "Generic (PLEG): container finished" podID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerID="2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9" exitCode=7
Jul 17 17:17:05.748134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.747704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" event={"ID":"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1","Type":"ContainerDied","Data":"2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9"}
Jul 17 17:17:05.748134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.747759    2561 scope.go:117] "RemoveContainer" containerID="3093b458016af43dd231a343ae1ab6ef617fd0320aa8660bda35d1f2e3f9992f"
Jul 17 17:17:05.748134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.747893    2561 scope.go:117] "RemoveContainer" containerID="2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9"
Jul 17 17:17:05.748270 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:05.748137    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52_odh-ai-gateway-infra(34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1"
Jul 17 17:17:05.748479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:05.748463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:06.089981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.089959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:06.095499 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:17:06.092561    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfa9455_a09c_499d_9a9f_a630c977ef21.slice/crio-bf19460d94b139f4c1f52d578e3b2e81f30c22437fb2cd0d1019be146fc8890f WatchSource:0}: Error finding container bf19460d94b139f4c1f52d578e3b2e81f30c22437fb2cd0d1019be146fc8890f: Status 404 returned error can't find the container with id bf19460d94b139f4c1f52d578e3b2e81f30c22437fb2cd0d1019be146fc8890f
Jul 17 17:17:06.514511 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.514481    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" path="/var/lib/kubelet/pods/14fa5519-1474-449c-9dbb-c65fdd618554/volumes"
Jul 17 17:17:06.754397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.754319    2561 scope.go:117] "RemoveContainer" containerID="2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9"
Jul 17 17:17:06.754850 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:06.754520    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52_odh-ai-gateway-infra(34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1"
Jul 17 17:17:06.755433 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.755410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerStarted","Data":"fae75524a855b5a87b42eeb53af15908de6baeadd9e7c9b80d108303786fddbf"}
Jul 17 17:17:06.755510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.755454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerStarted","Data":"bf19460d94b139f4c1f52d578e3b2e81f30c22437fb2cd0d1019be146fc8890f"}
Jul 17 17:17:06.785076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:06.785009    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" podStartSLOduration=1.784994293 podStartE2EDuration="1.784994293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:17:06.783500285 +0000 UTC m=+3060.768587843" watchObservedRunningTime="2026-07-17 17:17:06.784994293 +0000 UTC m=+3060.770081831"
Jul 17 17:17:07.508545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:07.508515    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:17:07.508724 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:07.508705    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:17:07.762024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:07.761926    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerID="fae75524a855b5a87b42eeb53af15908de6baeadd9e7c9b80d108303786fddbf" exitCode=7
Jul 17 17:17:07.762024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:07.761969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerDied","Data":"fae75524a855b5a87b42eeb53af15908de6baeadd9e7c9b80d108303786fddbf"}
Jul 17 17:17:07.762565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:07.762290    2561 scope.go:117] "RemoveContainer" containerID="fae75524a855b5a87b42eeb53af15908de6baeadd9e7c9b80d108303786fddbf"
Jul 17 17:17:08.769015 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:08.768928    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerID="a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2" exitCode=7
Jul 17 17:17:08.769442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:08.769011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerDied","Data":"a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2"}
Jul 17 17:17:08.769442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:08.769096    2561 scope.go:117] "RemoveContainer" containerID="fae75524a855b5a87b42eeb53af15908de6baeadd9e7c9b80d108303786fddbf"
Jul 17 17:17:08.769442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:08.769346    2561 scope.go:117] "RemoveContainer" containerID="a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2"
Jul 17 17:17:08.769599 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:08.769576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr_odh-ai-gateway-infra(cdfa9455-a09c-499d-9a9f-a630c977ef21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21"
Jul 17 17:17:09.777475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:09.777449    2561 scope.go:117] "RemoveContainer" containerID="a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2"
Jul 17 17:17:09.777891 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:09.777651    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr_odh-ai-gateway-infra(cdfa9455-a09c-499d-9a9f-a630c977ef21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21"
Jul 17 17:17:12.508918 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:12.508872    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:17:12.509411 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:12.509149    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:17:16.987868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:16.987826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:17:17.084667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:17.084631    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:17:19.508574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:19.508541    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:17:19.509074 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:19.508620    2561 scope.go:117] "RemoveContainer" containerID="2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9"
Jul 17 17:17:19.509074 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:19.508724    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:17:20.837254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:20.837221    2561 generic.go:356] "Generic (PLEG): container finished" podID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerID="01a40ac33ade4252c60f1ac8980c08a25dce884bcdd73b789185c63698f4df6a" exitCode=7
Jul 17 17:17:20.837658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:20.837288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" event={"ID":"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1","Type":"ContainerDied","Data":"01a40ac33ade4252c60f1ac8980c08a25dce884bcdd73b789185c63698f4df6a"}
Jul 17 17:17:20.837658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:20.837351    2561 scope.go:117] "RemoveContainer" containerID="2b5103a4dfb2d28801b4959bc71e54a28bc8dada8b10d94ad9da9a09ee1c38b9"
Jul 17 17:17:20.837753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:20.837685    2561 scope.go:117] "RemoveContainer" containerID="01a40ac33ade4252c60f1ac8980c08a25dce884bcdd73b789185c63698f4df6a"
Jul 17 17:17:20.837928 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:20.837908    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52_odh-ai-gateway-infra(34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1"
Jul 17 17:17:21.862035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:21.862003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:21.992985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:21.992963    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:22.156069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.155971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca\") pod \"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1\" (UID: \"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1\") "
Jul 17 17:17:22.156348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.156324    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" (UID: "34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:17:22.257309 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.257276    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:17:22.508980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.508885    2561 scope.go:117] "RemoveContainer" containerID="a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2"
Jul 17 17:17:22.848493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.848468    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"
Jul 17 17:17:22.848493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.848477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52" event={"ID":"34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1","Type":"ContainerDied","Data":"a8c6349fa5d218b97868246b194ee55f6d662b90c8f3c4554d8ce1ade6da27de"}
Jul 17 17:17:22.848747 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.848545    2561 scope.go:117] "RemoveContainer" containerID="01a40ac33ade4252c60f1ac8980c08a25dce884bcdd73b789185c63698f4df6a"
Jul 17 17:17:22.850396 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.850371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerStarted","Data":"03009081a485bfaea9a7430a4f5b9e2863f9a026c73e2be456efcd9fb4006627"}
Jul 17 17:17:22.884163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.884133    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:22.890701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:22.890673    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-2zc52"]
Jul 17 17:17:23.856877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:23.856849    2561 generic.go:356] "Generic (PLEG): container finished" podID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerID="03009081a485bfaea9a7430a4f5b9e2863f9a026c73e2be456efcd9fb4006627" exitCode=7
Jul 17 17:17:23.857071 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:23.856928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerDied","Data":"03009081a485bfaea9a7430a4f5b9e2863f9a026c73e2be456efcd9fb4006627"}
Jul 17 17:17:23.857071 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:23.856981    2561 scope.go:117] "RemoveContainer" containerID="a4919f3f3302ffb5e9474c78cdd002174385fd6d58326e6bb7e09164fd2ca5a2"
Jul 17 17:17:23.857304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:23.857280    2561 scope.go:117] "RemoveContainer" containerID="03009081a485bfaea9a7430a4f5b9e2863f9a026c73e2be456efcd9fb4006627"
Jul 17 17:17:23.857513 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:23.857491    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr_odh-ai-gateway-infra(cdfa9455-a09c-499d-9a9f-a630c977ef21)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21"
Jul 17 17:17:23.878634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:23.878606    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:24.519955 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:24.519920    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" path="/var/lib/kubelet/pods/34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1/volumes"
Jul 17 17:17:24.989058 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:24.989028    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:25.084848 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.084821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca\") pod \"cdfa9455-a09c-499d-9a9f-a630c977ef21\" (UID: \"cdfa9455-a09c-499d-9a9f-a630c977ef21\") "
Jul 17 17:17:25.085230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.085206    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdfa9455-a09c-499d-9a9f-a630c977ef21" (UID: "cdfa9455-a09c-499d-9a9f-a630c977ef21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:17:25.186142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.186062    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdfa9455-a09c-499d-9a9f-a630c977ef21-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:17:25.869348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.869305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr" event={"ID":"cdfa9455-a09c-499d-9a9f-a630c977ef21","Type":"ContainerDied","Data":"bf19460d94b139f4c1f52d578e3b2e81f30c22437fb2cd0d1019be146fc8890f"}
Jul 17 17:17:25.869348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.869329    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"
Jul 17 17:17:25.869953 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.869364    2561 scope.go:117] "RemoveContainer" containerID="03009081a485bfaea9a7430a4f5b9e2863f9a026c73e2be456efcd9fb4006627"
Jul 17 17:17:25.891981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.891954    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:25.896236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:25.896210    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-f6rmr"]
Jul 17 17:17:26.510524 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:26.510501    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:17:26.510704 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:26.510674    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:17:26.513774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:26.513746    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" path="/var/lib/kubelet/pods/cdfa9455-a09c-499d-9a9f-a630c977ef21/volumes"
Jul 17 17:17:33.509232 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:33.509200    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:17:33.509710 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:33.509390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:17:35.307266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307598    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307610    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307631    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307652    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307659    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307664    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307739    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307748    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fa5519-1474-449c-9dbb-c65fdd618554" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307760    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307764    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.307973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.307770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:35.312160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.312142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:35.314765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.314746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:17:35.318373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.318345    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:35.378624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.378592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-xx6j5\" (UID: \"414d57ad-aff0-4945-b917-eed34fed96d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:35.480012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.479968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-xx6j5\" (UID: \"414d57ad-aff0-4945-b917-eed34fed96d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:35.480606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.480585    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-xx6j5\" (UID: \"414d57ad-aff0-4945-b917-eed34fed96d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:35.623821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.623743    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:35.755010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.754983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:35.757194 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:17:35.757168    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414d57ad_aff0_4945_b917_eed34fed96d6.slice/crio-6ed99c53a23d2a06acbb4649691e48b68203eb76fd2d2abb63144ca194eae8da WatchSource:0}: Error finding container 6ed99c53a23d2a06acbb4649691e48b68203eb76fd2d2abb63144ca194eae8da: Status 404 returned error can't find the container with id 6ed99c53a23d2a06acbb4649691e48b68203eb76fd2d2abb63144ca194eae8da
Jul 17 17:17:35.917327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:35.917247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerStarted","Data":"6ed99c53a23d2a06acbb4649691e48b68203eb76fd2d2abb63144ca194eae8da"}
Jul 17 17:17:36.923651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:36.923614    2561 generic.go:356] "Generic (PLEG): container finished" podID="414d57ad-aff0-4945-b917-eed34fed96d6" containerID="b8d9b2f72241a615e371cdff634e7957a1cc49f3e7407af01532409ce0b89ad7" exitCode=7
Jul 17 17:17:36.924067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:36.923708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerDied","Data":"b8d9b2f72241a615e371cdff634e7957a1cc49f3e7407af01532409ce0b89ad7"}
Jul 17 17:17:36.924067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:36.923870    2561 scope.go:117] "RemoveContainer" containerID="b8d9b2f72241a615e371cdff634e7957a1cc49f3e7407af01532409ce0b89ad7"
Jul 17 17:17:37.931833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:37.931795    2561 generic.go:356] "Generic (PLEG): container finished" podID="414d57ad-aff0-4945-b917-eed34fed96d6" containerID="dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05" exitCode=7
Jul 17 17:17:37.932335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:37.931877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerDied","Data":"dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05"}
Jul 17 17:17:37.932335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:37.931932    2561 scope.go:117] "RemoveContainer" containerID="b8d9b2f72241a615e371cdff634e7957a1cc49f3e7407af01532409ce0b89ad7"
Jul 17 17:17:37.932335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:37.932197    2561 scope.go:117] "RemoveContainer" containerID="dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05"
Jul 17 17:17:37.932494 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:37.932395    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-xx6j5_odh-ai-gateway-infra(414d57ad-aff0-4945-b917-eed34fed96d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" podUID="414d57ad-aff0-4945-b917-eed34fed96d6"
Jul 17 17:17:38.937669 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:38.937638    2561 scope.go:117] "RemoveContainer" containerID="dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05"
Jul 17 17:17:38.938097 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:38.937842    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-xx6j5_odh-ai-gateway-infra(414d57ad-aff0-4945-b917-eed34fed96d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" podUID="414d57ad-aff0-4945-b917-eed34fed96d6"
Jul 17 17:17:41.508517 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:41.508487    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:17:41.508934 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:41.508672    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:17:45.787726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.787686    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:17:45.788239 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.788210    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:45.788239 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.788229    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:45.788366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.788286    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:45.788366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.788295    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fdb4ea-1ea6-4990-954d-6b8bd73e5bc1" containerName="revoke-keys"
Jul 17 17:17:45.788463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.788381    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfa9455-a09c-499d-9a9f-a630c977ef21" containerName="revoke-keys"
Jul 17 17:17:45.792651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.792630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:45.794951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.794929    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nv2jn\""
Jul 17 17:17:45.798885 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.798863    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:17:45.873639 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.873607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rl2w\" (UniqueName: \"kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w\") pod \"kuadrant-operator-catalog-pzcd9\" (UID: \"bf41d019-3c51-4ed8-8266-64fcef652ff9\") " pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:45.974828 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.974792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rl2w\" (UniqueName: \"kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w\") pod \"kuadrant-operator-catalog-pzcd9\" (UID: \"bf41d019-3c51-4ed8-8266-64fcef652ff9\") " pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:45.983793 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:45.983760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rl2w\" (UniqueName: \"kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w\") pod \"kuadrant-operator-catalog-pzcd9\" (UID: \"bf41d019-3c51-4ed8-8266-64fcef652ff9\") " pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:46.103090 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:46.102971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:46.234630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:46.234603    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:17:46.236617 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:17:46.236587    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf41d019_3c51_4ed8_8266_64fcef652ff9.slice/crio-179cd3cc2c2a8ab6f0fd35ee69f842fa98c2db31546ef45c83cad92318e2ef11 WatchSource:0}: Error finding container 179cd3cc2c2a8ab6f0fd35ee69f842fa98c2db31546ef45c83cad92318e2ef11: Status 404 returned error can't find the container with id 179cd3cc2c2a8ab6f0fd35ee69f842fa98c2db31546ef45c83cad92318e2ef11
Jul 17 17:17:46.511182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:46.511090    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:17:46.511344 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:46.511322    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:17:46.976777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:46.976742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" event={"ID":"bf41d019-3c51-4ed8-8266-64fcef652ff9","Type":"ContainerStarted","Data":"179cd3cc2c2a8ab6f0fd35ee69f842fa98c2db31546ef45c83cad92318e2ef11"}
Jul 17 17:17:48.987981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:48.987946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" event={"ID":"bf41d019-3c51-4ed8-8266-64fcef652ff9","Type":"ContainerStarted","Data":"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"}
Jul 17 17:17:49.006079 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:49.006000    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" podStartSLOduration=2.165197985 podStartE2EDuration="4.005981279s" totalImagesPullingTime="1.840783294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:17:49.003923569 +0000 UTC m=+3102.989011107" watchObservedRunningTime="2026-07-17 17:17:49.005981279 +0000 UTC m=+3102.991068818"
Jul 17 17:17:49.508224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:49.508187    2561 scope.go:117] "RemoveContainer" containerID="dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05"
Jul 17 17:17:49.994641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:49.994603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerStarted","Data":"624ea6b81229695229c5899c2c10c024b50571b5942dc505b7ec0e3acc5eec8f"}
Jul 17 17:17:50.010599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:50.010523    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" podStartSLOduration=15.010509019 podStartE2EDuration="15.010509019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:17:50.010065505 +0000 UTC m=+3103.995153061" watchObservedRunningTime="2026-07-17 17:17:50.010509019 +0000 UTC m=+3103.995596556"
Jul 17 17:17:51.001122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:51.001089    2561 generic.go:356] "Generic (PLEG): container finished" podID="414d57ad-aff0-4945-b917-eed34fed96d6" containerID="624ea6b81229695229c5899c2c10c024b50571b5942dc505b7ec0e3acc5eec8f" exitCode=7
Jul 17 17:17:51.001578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:51.001161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerDied","Data":"624ea6b81229695229c5899c2c10c024b50571b5942dc505b7ec0e3acc5eec8f"}
Jul 17 17:17:51.001578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:51.001219    2561 scope.go:117] "RemoveContainer" containerID="dbf4acf709d87c77337435dff46ba8920b2e56322fb5154957b46d6e05e5dd05"
Jul 17 17:17:51.001578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:51.001469    2561 scope.go:117] "RemoveContainer" containerID="624ea6b81229695229c5899c2c10c024b50571b5942dc505b7ec0e3acc5eec8f"
Jul 17 17:17:51.001717 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:51.001696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-xx6j5_odh-ai-gateway-infra(414d57ad-aff0-4945-b917-eed34fed96d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" podUID="414d57ad-aff0-4945-b917-eed34fed96d6"
Jul 17 17:17:51.026264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:51.026231    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:52.136030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:52.136002    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:52.236070 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:52.236010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca\") pod \"414d57ad-aff0-4945-b917-eed34fed96d6\" (UID: \"414d57ad-aff0-4945-b917-eed34fed96d6\") "
Jul 17 17:17:52.236391 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:52.236365    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "414d57ad-aff0-4945-b917-eed34fed96d6" (UID: "414d57ad-aff0-4945-b917-eed34fed96d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:17:52.336831 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:52.336800    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/414d57ad-aff0-4945-b917-eed34fed96d6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:17:52.508268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:52.508235    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:17:52.508434 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:52.508410    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:17:53.013124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.013083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5" event={"ID":"414d57ad-aff0-4945-b917-eed34fed96d6","Type":"ContainerDied","Data":"6ed99c53a23d2a06acbb4649691e48b68203eb76fd2d2abb63144ca194eae8da"}
Jul 17 17:17:53.013324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.013160    2561 scope.go:117] "RemoveContainer" containerID="624ea6b81229695229c5899c2c10c024b50571b5942dc505b7ec0e3acc5eec8f"
Jul 17 17:17:53.013324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.013099    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"
Jul 17 17:17:53.035384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.035347    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:53.041950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.041925    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-xx6j5"]
Jul 17 17:17:53.255790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.255755    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256117    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256128    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256138    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256144    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256155    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256161    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256232    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256240    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.256467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.256248    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" containerName="revoke-keys"
Jul 17 17:17:53.260603 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.260586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:17:53.263897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.263837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:17:53.270792 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.270759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:17:53.345560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.345529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97\" (UID: \"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:17:53.446442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.446401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97\" (UID: \"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:17:53.447088 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.447037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97\" (UID: \"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:17:53.571260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.571213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:17:53.720319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:53.720295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:17:53.722809 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:17:53.722780    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8e34bf_ded6_4f9a_9156_26b8cb62a2e1.slice/crio-7e959c85ee80e7486d1f574fdc8be7d9c36e49ea0225918db7d7cb98f452c9b4 WatchSource:0}: Error finding container 7e959c85ee80e7486d1f574fdc8be7d9c36e49ea0225918db7d7cb98f452c9b4: Status 404 returned error can't find the container with id 7e959c85ee80e7486d1f574fdc8be7d9c36e49ea0225918db7d7cb98f452c9b4
Jul 17 17:17:54.020317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:54.020088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerStarted","Data":"7e959c85ee80e7486d1f574fdc8be7d9c36e49ea0225918db7d7cb98f452c9b4"}
Jul 17 17:17:54.514103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:54.514063    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="414d57ad-aff0-4945-b917-eed34fed96d6" path="/var/lib/kubelet/pods/414d57ad-aff0-4945-b917-eed34fed96d6/volumes"
Jul 17 17:17:55.025757 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:55.025725    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerID="067d3ea5f15cd4730a8b11d48bf3ff27db25c65aa6c5cb9cf93b3e30383b4ff3" exitCode=7
Jul 17 17:17:55.025924 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:55.025808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerDied","Data":"067d3ea5f15cd4730a8b11d48bf3ff27db25c65aa6c5cb9cf93b3e30383b4ff3"}
Jul 17 17:17:55.025990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:55.025974    2561 scope.go:117] "RemoveContainer" containerID="067d3ea5f15cd4730a8b11d48bf3ff27db25c65aa6c5cb9cf93b3e30383b4ff3"
Jul 17 17:17:56.032137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.032106    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerID="9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11" exitCode=7
Jul 17 17:17:56.032527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.032196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerDied","Data":"9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11"}
Jul 17 17:17:56.032527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.032254    2561 scope.go:117] "RemoveContainer" containerID="067d3ea5f15cd4730a8b11d48bf3ff27db25c65aa6c5cb9cf93b3e30383b4ff3"
Jul 17 17:17:56.032527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.032380    2561 scope.go:117] "RemoveContainer" containerID="9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11"
Jul 17 17:17:56.032664 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:56.032624    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97_odh-ai-gateway-infra(0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1"
Jul 17 17:17:56.103349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.103309    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:56.103539 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.103372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:56.126099 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:56.126073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:57.039916 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:57.039888    2561 scope.go:117] "RemoveContainer" containerID="9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11"
Jul 17 17:17:57.040417 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:17:57.040131    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97_odh-ai-gateway-infra(0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1"
Jul 17 17:17:57.062337 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:57.062310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:57.098775 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:57.098741    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:17:59.048574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:59.048536    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" podUID="bf41d019-3c51-4ed8-8266-64fcef652ff9" containerName="registry-server" containerID="cri-o://4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7" gracePeriod=2
Jul 17 17:17:59.300362 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:59.300306    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:17:59.401798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:59.401764    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rl2w\" (UniqueName: \"kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w\") pod \"bf41d019-3c51-4ed8-8266-64fcef652ff9\" (UID: \"bf41d019-3c51-4ed8-8266-64fcef652ff9\") "
Jul 17 17:17:59.403836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:59.403812    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w" (OuterVolumeSpecName: "kube-api-access-4rl2w") pod "bf41d019-3c51-4ed8-8266-64fcef652ff9" (UID: "bf41d019-3c51-4ed8-8266-64fcef652ff9"). InnerVolumeSpecName "kube-api-access-4rl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:17:59.502730 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:17:59.502689    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rl2w\" (UniqueName: \"kubernetes.io/projected/bf41d019-3c51-4ed8-8266-64fcef652ff9-kube-api-access-4rl2w\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:18:00.054537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.054501    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf41d019-3c51-4ed8-8266-64fcef652ff9" containerID="4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7" exitCode=0
Jul 17 17:18:00.054985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.054549    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9"
Jul 17 17:18:00.054985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.054587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" event={"ID":"bf41d019-3c51-4ed8-8266-64fcef652ff9","Type":"ContainerDied","Data":"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"}
Jul 17 17:18:00.054985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.054635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pzcd9" event={"ID":"bf41d019-3c51-4ed8-8266-64fcef652ff9","Type":"ContainerDied","Data":"179cd3cc2c2a8ab6f0fd35ee69f842fa98c2db31546ef45c83cad92318e2ef11"}
Jul 17 17:18:00.054985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.054656    2561 scope.go:117] "RemoveContainer" containerID="4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"
Jul 17 17:18:00.063740 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.063723    2561 scope.go:117] "RemoveContainer" containerID="4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"
Jul 17 17:18:00.063994 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:00.063974    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7\": container with ID starting with 4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7 not found: ID does not exist" containerID="4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"
Jul 17 17:18:00.064081 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.064001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7"} err="failed to get container status \"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7\": rpc error: code = NotFound desc = could not find container \"4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7\": container with ID starting with 4f4f49904151c15cb91eccee028c4ca6ca19edba926e7b538e5aec6c6de5cfe7 not found: ID does not exist"
Jul 17 17:18:00.075571 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.075548    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:18:00.079990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.079965    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pzcd9"]
Jul 17 17:18:00.513814 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:00.513733    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf41d019-3c51-4ed8-8266-64fcef652ff9" path="/var/lib/kubelet/pods/bf41d019-3c51-4ed8-8266-64fcef652ff9/volumes"
Jul 17 17:18:01.509164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:01.509138    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:18:01.509561 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:01.509314    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:05.484067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.483205    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:05.484067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.483956    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf41d019-3c51-4ed8-8266-64fcef652ff9" containerName="registry-server"
Jul 17 17:18:05.484067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.483977    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf41d019-3c51-4ed8-8266-64fcef652ff9" containerName="registry-server"
Jul 17 17:18:05.485067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.484996    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf41d019-3c51-4ed8-8266-64fcef652ff9" containerName="registry-server"
Jul 17 17:18:05.489823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.489797    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:05.491880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.491842    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:05.558296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.558253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx\" (UID: \"a4ce32c7-236f-4b9d-bf44-972717b3d733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:05.659621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.659584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx\" (UID: \"a4ce32c7-236f-4b9d-bf44-972717b3d733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:05.660258 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.660240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx\" (UID: \"a4ce32c7-236f-4b9d-bf44-972717b3d733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:05.802010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:05.801960    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:06.135950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:06.135925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:06.138223 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:18:06.138198    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ce32c7_236f_4b9d_bf44_972717b3d733.slice/crio-9979bf4efde122b46d49d3dddff6f5f2dcd209ee446c26e0ebd003de920de880 WatchSource:0}: Error finding container 9979bf4efde122b46d49d3dddff6f5f2dcd209ee446c26e0ebd003de920de880: Status 404 returned error can't find the container with id 9979bf4efde122b46d49d3dddff6f5f2dcd209ee446c26e0ebd003de920de880
Jul 17 17:18:07.089864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:07.089783    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerID="edf12e5e276bba97636d8ee3df6a10a24aa7ddaeb8121719e232db3133a3e728" exitCode=7
Jul 17 17:18:07.090244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:07.089865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" event={"ID":"a4ce32c7-236f-4b9d-bf44-972717b3d733","Type":"ContainerDied","Data":"edf12e5e276bba97636d8ee3df6a10a24aa7ddaeb8121719e232db3133a3e728"}
Jul 17 17:18:07.090244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:07.089923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" event={"ID":"a4ce32c7-236f-4b9d-bf44-972717b3d733","Type":"ContainerStarted","Data":"9979bf4efde122b46d49d3dddff6f5f2dcd209ee446c26e0ebd003de920de880"}
Jul 17 17:18:07.090244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:07.090064    2561 scope.go:117] "RemoveContainer" containerID="edf12e5e276bba97636d8ee3df6a10a24aa7ddaeb8121719e232db3133a3e728"
Jul 17 17:18:07.508308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:07.508283    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:18:07.508457 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:07.508440    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:08.096776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:08.096740    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerID="247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce" exitCode=7
Jul 17 17:18:08.097233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:08.096811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" event={"ID":"a4ce32c7-236f-4b9d-bf44-972717b3d733","Type":"ContainerDied","Data":"247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce"}
Jul 17 17:18:08.097233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:08.096860    2561 scope.go:117] "RemoveContainer" containerID="edf12e5e276bba97636d8ee3df6a10a24aa7ddaeb8121719e232db3133a3e728"
Jul 17 17:18:08.097233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:08.097039    2561 scope.go:117] "RemoveContainer" containerID="247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce"
Jul 17 17:18:08.097370 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:08.097266    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx_odh-ai-gateway-infra(a4ce32c7-236f-4b9d-bf44-972717b3d733)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733"
Jul 17 17:18:09.106246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:09.106220    2561 scope.go:117] "RemoveContainer" containerID="247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce"
Jul 17 17:18:09.106662 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:09.106390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx_odh-ai-gateway-infra(a4ce32c7-236f-4b9d-bf44-972717b3d733)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733"
Jul 17 17:18:12.508552 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:12.508525    2561 scope.go:117] "RemoveContainer" containerID="9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11"
Jul 17 17:18:13.127383 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:13.127358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerStarted","Data":"c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585"}
Jul 17 17:18:13.127691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:13.127675    2561 scope.go:117] "RemoveContainer" containerID="c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585"
Jul 17 17:18:13.127895 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:13.127878    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97_odh-ai-gateway-infra(0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1"
Jul 17 17:18:14.133519 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.133486    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerID="c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585" exitCode=7
Jul 17 17:18:14.133519 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.133527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerDied","Data":"c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585"}
Jul 17 17:18:14.134035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.133586    2561 scope.go:117] "RemoveContainer" containerID="9d333cb26958c5613b8e08c8e227a600ee9d2ce915fad5d3bddd67cfa382ea11"
Jul 17 17:18:14.134035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.133859    2561 scope.go:117] "RemoveContainer" containerID="c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585"
Jul 17 17:18:14.134162 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:14.134094    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97_odh-ai-gateway-infra(0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1"
Jul 17 17:18:14.153796 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.153769    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:18:14.508846 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:14.508783    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:18:15.141062 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.141022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerStarted","Data":"1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"}
Jul 17 17:18:15.141505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.141246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:18:15.320585 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.320565    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:18:15.336368 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.336338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca\") pod \"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1\" (UID: \"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1\") "
Jul 17 17:18:15.336800 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.336762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" (UID: "0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:18:15.437190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:15.437162    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:18:16.150117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.150080    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a" exitCode=1
Jul 17 17:18:16.150559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.150173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"}
Jul 17 17:18:16.150559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.150237    2561 scope.go:117] "RemoveContainer" containerID="53f40966bffa266f5ac578fb214d068cbed9f9dc3c35accbcc478f5a4a13b0f6"
Jul 17 17:18:16.150559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.150470    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:18:16.150695 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:16.150674    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:16.151472 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.151455    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"
Jul 17 17:18:16.151534 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.151487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97" event={"ID":"0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1","Type":"ContainerDied","Data":"7e959c85ee80e7486d1f574fdc8be7d9c36e49ea0225918db7d7cb98f452c9b4"}
Jul 17 17:18:16.164027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.164002    2561 scope.go:117] "RemoveContainer" containerID="c6d6462b09f8bbc887f85b78a27850f4c93f08bf2aa863cac1594e9ebb095585"
Jul 17 17:18:16.180606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.180585    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:18:16.186075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.186039    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-wlh97"]
Jul 17 17:18:16.515315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:16.515237    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" path="/var/lib/kubelet/pods/0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1/volumes"
Jul 17 17:18:17.157863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:17.157828    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:18:17.158310 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:17.158071    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:19.508451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:19.508422    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:18:19.508874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:19.508705    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:18:20.174080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:20.174031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerStarted","Data":"4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"}
Jul 17 17:18:20.174267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:20.174249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:18:20.191527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:20.191478    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podStartSLOduration=659.234914196 podStartE2EDuration="11m4.191464612s" totalImagesPullingTime="4.956550416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:07:16 +0000 UTC" imagePullSessionsCount=8 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:18:20.190812112 +0000 UTC m=+3134.175899657" watchObservedRunningTime="2026-07-17 17:18:20.191464612 +0000 UTC m=+3134.176552149"
Jul 17 17:18:21.179963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:21.179928    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7" exitCode=1
Jul 17 17:18:21.180410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:21.179972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"}
Jul 17 17:18:21.180410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:21.180021    2561 scope.go:117] "RemoveContainer" containerID="aa9173b61ecdf2229ddfad9ea42816df292dd5756829c7ffeca3a30adceb7df8"
Jul 17 17:18:21.180410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:21.180278    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:18:21.180568 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:21.180484    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:22.185949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:22.185923    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:18:22.186357 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:22.186117    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:22.509197 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:22.509129    2561 scope.go:117] "RemoveContainer" containerID="247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce"
Jul 17 17:18:23.192297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:23.192267    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerID="d4e448ee82cf93d27b29a09c7e36f49f48b342195c1947fc1a8035eb786bb37f" exitCode=7
Jul 17 17:18:23.192661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:23.192334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" event={"ID":"a4ce32c7-236f-4b9d-bf44-972717b3d733","Type":"ContainerDied","Data":"d4e448ee82cf93d27b29a09c7e36f49f48b342195c1947fc1a8035eb786bb37f"}
Jul 17 17:18:23.192661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:23.192393    2561 scope.go:117] "RemoveContainer" containerID="247f82a00e8cb8e5fd69e22b32bbf04d460e1f8f5a8bf8b135accdd5830f68ce"
Jul 17 17:18:23.192788 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:23.192712    2561 scope.go:117] "RemoveContainer" containerID="d4e448ee82cf93d27b29a09c7e36f49f48b342195c1947fc1a8035eb786bb37f"
Jul 17 17:18:23.192973 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:23.192953    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx_odh-ai-gateway-infra(a4ce32c7-236f-4b9d-bf44-972717b3d733)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733"
Jul 17 17:18:24.218383 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.218350    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:24.351703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.351678    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:24.422446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.422416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca\") pod \"a4ce32c7-236f-4b9d-bf44-972717b3d733\" (UID: \"a4ce32c7-236f-4b9d-bf44-972717b3d733\") "
Jul 17 17:18:24.422762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.422741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4ce32c7-236f-4b9d-bf44-972717b3d733" (UID: "a4ce32c7-236f-4b9d-bf44-972717b3d733"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:18:24.523357 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.523334    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ce32c7-236f-4b9d-bf44-972717b3d733-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:18:24.968937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.968901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:18:24.969291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:24.969277    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:18:24.969460 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:24.969442    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:25.204158 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.204131    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"
Jul 17 17:18:25.204315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.204184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx" event={"ID":"a4ce32c7-236f-4b9d-bf44-972717b3d733","Type":"ContainerDied","Data":"9979bf4efde122b46d49d3dddff6f5f2dcd209ee446c26e0ebd003de920de880"}
Jul 17 17:18:25.204315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.204237    2561 scope.go:117] "RemoveContainer" containerID="d4e448ee82cf93d27b29a09c7e36f49f48b342195c1947fc1a8035eb786bb37f"
Jul 17 17:18:25.220593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.220539    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:25.224303 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.224282    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-v2dqx"]
Jul 17 17:18:25.351146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351114    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:25.351498 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351481    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351512    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351521    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351531    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351539    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351552    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351560    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.351590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351615    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351717    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351730    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351739    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351747    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351759    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8e34bf-ded6-4f9a-9156-26b8cb62a2e1" containerName="revoke-keys"
Jul 17 17:18:25.352020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.351768    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" containerName="revoke-keys"
Jul 17 17:18:25.356154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.356134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:25.358780 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.358762    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:18:25.367282 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.367261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:25.432578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.432553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-8rt5j\" (UID: \"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:25.533353 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.533316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-8rt5j\" (UID: \"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:25.533913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.533894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-8rt5j\" (UID: \"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:25.666483 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.666454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:25.811903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:25.811878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:25.814068 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:18:25.814017    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b349d2_f586_4bfb_ab4f_d89cb8c9ba7c.slice/crio-a029aa36c0aa2679b7a5a5659e5e3e58532cf89dcce54b3186ef3c66cf99c1f5 WatchSource:0}: Error finding container a029aa36c0aa2679b7a5a5659e5e3e58532cf89dcce54b3186ef3c66cf99c1f5: Status 404 returned error can't find the container with id a029aa36c0aa2679b7a5a5659e5e3e58532cf89dcce54b3186ef3c66cf99c1f5
Jul 17 17:18:26.211314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:26.211235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerStarted","Data":"bd54af421b1dbe223c89da15047573c323234b56c8bf0a2e0183f3161a91c2d6"}
Jul 17 17:18:26.211314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:26.211286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerStarted","Data":"a029aa36c0aa2679b7a5a5659e5e3e58532cf89dcce54b3186ef3c66cf99c1f5"}
Jul 17 17:18:26.226659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:26.226615    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" podStartSLOduration=1.226600868 podStartE2EDuration="1.226600868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:18:26.225559282 +0000 UTC m=+3140.210646821" watchObservedRunningTime="2026-07-17 17:18:26.226600868 +0000 UTC m=+3140.211688405"
Jul 17 17:18:26.514027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:26.513957    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ce32c7-236f-4b9d-bf44-972717b3d733" path="/var/lib/kubelet/pods/a4ce32c7-236f-4b9d-bf44-972717b3d733/volumes"
Jul 17 17:18:27.216387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:27.216353    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerID="bd54af421b1dbe223c89da15047573c323234b56c8bf0a2e0183f3161a91c2d6" exitCode=7
Jul 17 17:18:27.216604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:27.216437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerDied","Data":"bd54af421b1dbe223c89da15047573c323234b56c8bf0a2e0183f3161a91c2d6"}
Jul 17 17:18:27.216604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:27.216578    2561 scope.go:117] "RemoveContainer" containerID="bd54af421b1dbe223c89da15047573c323234b56c8bf0a2e0183f3161a91c2d6"
Jul 17 17:18:27.508227 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:27.508159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:18:27.508559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:27.508538    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:18:27.508698 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:27.508680    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:28.223861 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:28.223782    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerID="ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb" exitCode=7
Jul 17 17:18:28.224010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:28.223859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerDied","Data":"ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb"}
Jul 17 17:18:28.224010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:28.223914    2561 scope.go:117] "RemoveContainer" containerID="bd54af421b1dbe223c89da15047573c323234b56c8bf0a2e0183f3161a91c2d6"
Jul 17 17:18:28.224160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:28.224111    2561 scope.go:117] "RemoveContainer" containerID="ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb"
Jul 17 17:18:28.224336 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:28.224317    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-8rt5j_odh-ai-gateway-infra(d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c"
Jul 17 17:18:29.230228 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:29.230201    2561 scope.go:117] "RemoveContainer" containerID="ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb"
Jul 17 17:18:29.230614 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:29.230369    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-8rt5j_odh-ai-gateway-infra(d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c"
Jul 17 17:18:37.508691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:37.508662    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:18:37.509178 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:37.508833    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:40.509066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:40.509017    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:18:40.509474 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:40.509262    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:43.508304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:43.508276    2561 scope.go:117] "RemoveContainer" containerID="ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb"
Jul 17 17:18:44.301458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:44.301426    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerID="0105cdc360fe39ab1901711023cf4c96d9e5e1bf42cf5e51bd29ae9602677b2f" exitCode=7
Jul 17 17:18:44.301592 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:44.301496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerDied","Data":"0105cdc360fe39ab1901711023cf4c96d9e5e1bf42cf5e51bd29ae9602677b2f"}
Jul 17 17:18:44.301592 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:44.301557    2561 scope.go:117] "RemoveContainer" containerID="ee1dcf82bbd382fe90bd56b11700725bab6c20115688b03018152fc2b089a1bb"
Jul 17 17:18:44.301871 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:44.301851    2561 scope.go:117] "RemoveContainer" containerID="0105cdc360fe39ab1901711023cf4c96d9e5e1bf42cf5e51bd29ae9602677b2f"
Jul 17 17:18:44.302118 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:44.302097    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-8rt5j_odh-ai-gateway-infra(d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c"
Jul 17 17:18:45.329608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:45.329578    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:45.458733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:45.458713    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:45.608777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:45.608705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca\") pod \"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c\" (UID: \"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c\") "
Jul 17 17:18:45.609029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:45.609009    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" (UID: "d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:18:45.710265 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:45.710242    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:18:46.313458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.313426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j" event={"ID":"d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c","Type":"ContainerDied","Data":"a029aa36c0aa2679b7a5a5659e5e3e58532cf89dcce54b3186ef3c66cf99c1f5"}
Jul 17 17:18:46.313458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.313458    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"
Jul 17 17:18:46.313685 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.313486    2561 scope.go:117] "RemoveContainer" containerID="0105cdc360fe39ab1901711023cf4c96d9e5e1bf42cf5e51bd29ae9602677b2f"
Jul 17 17:18:46.342108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.342079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:46.343764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.343746    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-8rt5j"]
Jul 17 17:18:46.514012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:46.513984    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" path="/var/lib/kubelet/pods/d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c/volumes"
Jul 17 17:18:49.508658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:49.508627    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:18:49.509082 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:49.508792    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:18:52.508507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:52.508479    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:18:52.508908 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:52.508649    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:18:53.296756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.296721    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:18:53.297104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.297157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297106    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.297157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297114    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.297157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297119    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.297255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297187    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.297255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.297200    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:53.300148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.300131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:18:53.302590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.302571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:18:53.308679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.308654    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:18:53.368806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.368771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p\" (UID: \"bb43ec73-1400-42bd-b6f8-7693dd4634cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:18:53.470005 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.469982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p\" (UID: \"bb43ec73-1400-42bd-b6f8-7693dd4634cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:18:53.470576 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.470557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p\" (UID: \"bb43ec73-1400-42bd-b6f8-7693dd4634cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:18:53.611774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.611707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:18:53.738324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:53.738297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:18:53.740296 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:18:53.740262    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb43ec73_1400_42bd_b6f8_7693dd4634cb.slice/crio-dffffb179bdac5934eed0e4c2cb3d3b99500561899b0d9b1d0a6b2b030259a81 WatchSource:0}: Error finding container dffffb179bdac5934eed0e4c2cb3d3b99500561899b0d9b1d0a6b2b030259a81: Status 404 returned error can't find the container with id dffffb179bdac5934eed0e4c2cb3d3b99500561899b0d9b1d0a6b2b030259a81
Jul 17 17:18:54.352872 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:54.352846    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerID="fc11990fb1a855d56e370a9c7d0a8645e6d68816c7fc26ff26f20d198dc04621" exitCode=7
Jul 17 17:18:54.353007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:54.352929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" event={"ID":"bb43ec73-1400-42bd-b6f8-7693dd4634cb","Type":"ContainerDied","Data":"fc11990fb1a855d56e370a9c7d0a8645e6d68816c7fc26ff26f20d198dc04621"}
Jul 17 17:18:54.353007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:54.352989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" event={"ID":"bb43ec73-1400-42bd-b6f8-7693dd4634cb","Type":"ContainerStarted","Data":"dffffb179bdac5934eed0e4c2cb3d3b99500561899b0d9b1d0a6b2b030259a81"}
Jul 17 17:18:54.353157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:54.353146    2561 scope.go:117] "RemoveContainer" containerID="fc11990fb1a855d56e370a9c7d0a8645e6d68816c7fc26ff26f20d198dc04621"
Jul 17 17:18:55.358915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.358887    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerID="112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4" exitCode=7
Jul 17 17:18:55.359327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.358975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" event={"ID":"bb43ec73-1400-42bd-b6f8-7693dd4634cb","Type":"ContainerDied","Data":"112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4"}
Jul 17 17:18:55.359327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.359090    2561 scope.go:117] "RemoveContainer" containerID="fc11990fb1a855d56e370a9c7d0a8645e6d68816c7fc26ff26f20d198dc04621"
Jul 17 17:18:55.359327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.359141    2561 scope.go:117] "RemoveContainer" containerID="112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4"
Jul 17 17:18:55.359327 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:55.359314    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p_odh-ai-gateway-infra(bb43ec73-1400-42bd-b6f8-7693dd4634cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb"
Jul 17 17:18:55.521148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.521119    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:18:55.521519 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.521501    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:55.521605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.521521    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:55.521661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.521647    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b349d2-f586-4bfb-ab4f-d89cb8c9ba7c" containerName="revoke-keys"
Jul 17 17:18:55.525724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.525705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:18:55.534940 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.534921    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:18:55.587797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.587764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz\" (UID: \"3504b4cb-f329-45bb-a05a-8b7642f3cd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:18:55.688381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.688324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz\" (UID: \"3504b4cb-f329-45bb-a05a-8b7642f3cd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:18:55.688846 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.688828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz\" (UID: \"3504b4cb-f329-45bb-a05a-8b7642f3cd4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:18:55.836578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.836549    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:18:55.973690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:55.973628    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:18:55.977455 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:18:55.977423    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3504b4cb_f329_45bb_a05a_8b7642f3cd4e.slice/crio-4eab997d23171580bbe67f8193383fb6029e6acf3543c2fc6606f84055f0847a WatchSource:0}: Error finding container 4eab997d23171580bbe67f8193383fb6029e6acf3543c2fc6606f84055f0847a: Status 404 returned error can't find the container with id 4eab997d23171580bbe67f8193383fb6029e6acf3543c2fc6606f84055f0847a
Jul 17 17:18:56.368465 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:56.368444    2561 scope.go:117] "RemoveContainer" containerID="112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4"
Jul 17 17:18:56.368861 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:56.368681    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p_odh-ai-gateway-infra(bb43ec73-1400-42bd-b6f8-7693dd4634cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb"
Jul 17 17:18:56.369624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:56.369601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerStarted","Data":"ea2ec70f806a84ae7d26f99232a67701fcc7b76abb8c80b19aada834e85e35a2"}
Jul 17 17:18:56.369719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:56.369644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerStarted","Data":"4eab997d23171580bbe67f8193383fb6029e6acf3543c2fc6606f84055f0847a"}
Jul 17 17:18:56.406240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:56.406199    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" podStartSLOduration=1.406187683 podStartE2EDuration="1.406187683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:18:56.404753452 +0000 UTC m=+3170.389841021" watchObservedRunningTime="2026-07-17 17:18:56.406187683 +0000 UTC m=+3170.391275221"
Jul 17 17:18:57.375627 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:57.375594    2561 generic.go:356] "Generic (PLEG): container finished" podID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerID="ea2ec70f806a84ae7d26f99232a67701fcc7b76abb8c80b19aada834e85e35a2" exitCode=7
Jul 17 17:18:57.376028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:57.375686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerDied","Data":"ea2ec70f806a84ae7d26f99232a67701fcc7b76abb8c80b19aada834e85e35a2"}
Jul 17 17:18:57.376028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:57.375820    2561 scope.go:117] "RemoveContainer" containerID="ea2ec70f806a84ae7d26f99232a67701fcc7b76abb8c80b19aada834e85e35a2"
Jul 17 17:18:58.382723 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:58.382690    2561 generic.go:356] "Generic (PLEG): container finished" podID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerID="afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff" exitCode=7
Jul 17 17:18:58.383210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:58.382775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerDied","Data":"afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff"}
Jul 17 17:18:58.383210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:58.382838    2561 scope.go:117] "RemoveContainer" containerID="ea2ec70f806a84ae7d26f99232a67701fcc7b76abb8c80b19aada834e85e35a2"
Jul 17 17:18:58.383210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:58.382917    2561 scope.go:117] "RemoveContainer" containerID="afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff"
Jul 17 17:18:58.383210 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:58.383134    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz_odh-ai-gateway-infra(3504b4cb-f329-45bb-a05a-8b7642f3cd4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e"
Jul 17 17:18:59.388658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:18:59.388635    2561 scope.go:117] "RemoveContainer" containerID="afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff"
Jul 17 17:18:59.389030 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:18:59.388787    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz_odh-ai-gateway-infra(3504b4cb-f329-45bb-a05a-8b7642f3cd4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e"
Jul 17 17:19:02.508821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:02.508792    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:19:02.509227 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:02.508952    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:19:05.508400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:05.508366    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:19:05.508746 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:05.508534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:19:08.508677 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:08.508646    2561 scope.go:117] "RemoveContainer" containerID="112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4"
Jul 17 17:19:09.442172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:09.442087    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerID="d77708f55137fe90f2fcaf54d396d20b86b2a0d397b2d40687d9bbfe21b0fac0" exitCode=7
Jul 17 17:19:09.442172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:09.442163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" event={"ID":"bb43ec73-1400-42bd-b6f8-7693dd4634cb","Type":"ContainerDied","Data":"d77708f55137fe90f2fcaf54d396d20b86b2a0d397b2d40687d9bbfe21b0fac0"}
Jul 17 17:19:09.442434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:09.442221    2561 scope.go:117] "RemoveContainer" containerID="112c2cd1f7d2658a65065a7e702d8baf00f52ae8b82a1b405591eaff0efd9fb4"
Jul 17 17:19:09.442503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:09.442490    2561 scope.go:117] "RemoveContainer" containerID="d77708f55137fe90f2fcaf54d396d20b86b2a0d397b2d40687d9bbfe21b0fac0"
Jul 17 17:19:09.442724 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:09.442698    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p_odh-ai-gateway-infra(bb43ec73-1400-42bd-b6f8-7693dd4634cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb"
Jul 17 17:19:10.466972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.466938    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:19:10.508622 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.508583    2561 scope.go:117] "RemoveContainer" containerID="afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff"
Jul 17 17:19:10.602739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.602720    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:19:10.711173 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.711145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca\") pod \"bb43ec73-1400-42bd-b6f8-7693dd4634cb\" (UID: \"bb43ec73-1400-42bd-b6f8-7693dd4634cb\") "
Jul 17 17:19:10.711455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.711434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb43ec73-1400-42bd-b6f8-7693dd4634cb" (UID: "bb43ec73-1400-42bd-b6f8-7693dd4634cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:19:10.811990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:10.811965    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb43ec73-1400-42bd-b6f8-7693dd4634cb-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:19:11.454244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.454162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p" event={"ID":"bb43ec73-1400-42bd-b6f8-7693dd4634cb","Type":"ContainerDied","Data":"dffffb179bdac5934eed0e4c2cb3d3b99500561899b0d9b1d0a6b2b030259a81"}
Jul 17 17:19:11.454244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.454235    2561 scope.go:117] "RemoveContainer" containerID="d77708f55137fe90f2fcaf54d396d20b86b2a0d397b2d40687d9bbfe21b0fac0"
Jul 17 17:19:11.454448 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.454177    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"
Jul 17 17:19:11.456008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.455984    2561 generic.go:356] "Generic (PLEG): container finished" podID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerID="6d92e48762a3b3d68e6892f671b8b150e1b572c7c1eaa42653c3ef92fc809a2f" exitCode=7
Jul 17 17:19:11.456142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.456077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerDied","Data":"6d92e48762a3b3d68e6892f671b8b150e1b572c7c1eaa42653c3ef92fc809a2f"}
Jul 17 17:19:11.456345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.456331    2561 scope.go:117] "RemoveContainer" containerID="6d92e48762a3b3d68e6892f671b8b150e1b572c7c1eaa42653c3ef92fc809a2f"
Jul 17 17:19:11.456538 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:11.456510    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz_odh-ai-gateway-infra(3504b4cb-f329-45bb-a05a-8b7642f3cd4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e"
Jul 17 17:19:11.462540 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.462525    2561 scope.go:117] "RemoveContainer" containerID="afe9aef87a2ad099d617e08e8e09e980577c2c49c4640464caeb01cae07ebbff"
Jul 17 17:19:11.488797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.488773    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:19:11.492903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:11.492875    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-fmh6p"]
Jul 17 17:19:12.484197 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.484160    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:19:12.515263 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.515231    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" path="/var/lib/kubelet/pods/bb43ec73-1400-42bd-b6f8-7693dd4634cb/volumes"
Jul 17 17:19:12.635347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.635319    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:19:12.726878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.726840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca\") pod \"3504b4cb-f329-45bb-a05a-8b7642f3cd4e\" (UID: \"3504b4cb-f329-45bb-a05a-8b7642f3cd4e\") "
Jul 17 17:19:12.727225 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.727201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3504b4cb-f329-45bb-a05a-8b7642f3cd4e" (UID: "3504b4cb-f329-45bb-a05a-8b7642f3cd4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:19:12.828145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:12.828118    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3504b4cb-f329-45bb-a05a-8b7642f3cd4e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:19:13.477211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:13.477184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"
Jul 17 17:19:13.477211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:13.477200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz" event={"ID":"3504b4cb-f329-45bb-a05a-8b7642f3cd4e","Type":"ContainerDied","Data":"4eab997d23171580bbe67f8193383fb6029e6acf3543c2fc6606f84055f0847a"}
Jul 17 17:19:13.477417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:13.477263    2561 scope.go:117] "RemoveContainer" containerID="6d92e48762a3b3d68e6892f671b8b150e1b572c7c1eaa42653c3ef92fc809a2f"
Jul 17 17:19:13.497631 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:13.497601    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:19:13.504673 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:13.504653    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-ts4vz"]
Jul 17 17:19:14.513053 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:14.513003    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" path="/var/lib/kubelet/pods/3504b4cb-f329-45bb-a05a-8b7642f3cd4e/volumes"
Jul 17 17:19:16.513484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:16.513458    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:19:16.513874 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:16.513645    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:19:17.509037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:17.509006    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:19:17.509235 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:17.509194    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:19:25.402964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.402935    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403269    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403280    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403295    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403301    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403310    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403317    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403341    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403347    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403353    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403410    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403417    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403424    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3504b4cb-f329-45bb-a05a-8b7642f3cd4e" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403431    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403438    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.403641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.403443    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb43ec73-1400-42bd-b6f8-7693dd4634cb" containerName="revoke-keys"
Jul 17 17:19:25.407807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.407793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:25.410242 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.410222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:19:25.415089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.414712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:25.530169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.530135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6ngz6\" (UID: \"66924833-7416-472b-abdf-72ab09c01244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:25.631167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.631137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6ngz6\" (UID: \"66924833-7416-472b-abdf-72ab09c01244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:25.631745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.631722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-6ngz6\" (UID: \"66924833-7416-472b-abdf-72ab09c01244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:25.719387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.719323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:25.845669 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:25.845645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:25.847660 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:19:25.847634    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66924833_7416_472b_abdf_72ab09c01244.slice/crio-c6c5dfd84c90c5443983872a762140cee3704fe7589a16af19201880bcbc41a6 WatchSource:0}: Error finding container c6c5dfd84c90c5443983872a762140cee3704fe7589a16af19201880bcbc41a6: Status 404 returned error can't find the container with id c6c5dfd84c90c5443983872a762140cee3704fe7589a16af19201880bcbc41a6
Jul 17 17:19:26.539084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:26.539034    2561 generic.go:356] "Generic (PLEG): container finished" podID="66924833-7416-472b-abdf-72ab09c01244" containerID="3a4a4650e6aa530e268780e346d9973fcc02a76b8e30a4985720a595ec853dc5" exitCode=7
Jul 17 17:19:26.539460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:26.539092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" event={"ID":"66924833-7416-472b-abdf-72ab09c01244","Type":"ContainerDied","Data":"3a4a4650e6aa530e268780e346d9973fcc02a76b8e30a4985720a595ec853dc5"}
Jul 17 17:19:26.539460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:26.539165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" event={"ID":"66924833-7416-472b-abdf-72ab09c01244","Type":"ContainerStarted","Data":"c6c5dfd84c90c5443983872a762140cee3704fe7589a16af19201880bcbc41a6"}
Jul 17 17:19:26.539460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:26.539370    2561 scope.go:117] "RemoveContainer" containerID="3a4a4650e6aa530e268780e346d9973fcc02a76b8e30a4985720a595ec853dc5"
Jul 17 17:19:27.546247 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:27.546212    2561 generic.go:356] "Generic (PLEG): container finished" podID="66924833-7416-472b-abdf-72ab09c01244" containerID="45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da" exitCode=7
Jul 17 17:19:27.546707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:27.546292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" event={"ID":"66924833-7416-472b-abdf-72ab09c01244","Type":"ContainerDied","Data":"45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da"}
Jul 17 17:19:27.546707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:27.546389    2561 scope.go:117] "RemoveContainer" containerID="3a4a4650e6aa530e268780e346d9973fcc02a76b8e30a4985720a595ec853dc5"
Jul 17 17:19:27.546707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:27.546601    2561 scope.go:117] "RemoveContainer" containerID="45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da"
Jul 17 17:19:27.546837 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:27.546816    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6ngz6_odh-ai-gateway-infra(66924833-7416-472b-abdf-72ab09c01244)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" podUID="66924833-7416-472b-abdf-72ab09c01244"
Jul 17 17:19:28.551975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:28.551948    2561 scope.go:117] "RemoveContainer" containerID="45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da"
Jul 17 17:19:28.552423 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:28.552187    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6ngz6_odh-ai-gateway-infra(66924833-7416-472b-abdf-72ab09c01244)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" podUID="66924833-7416-472b-abdf-72ab09c01244"
Jul 17 17:19:29.508838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:29.508806    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:19:29.509027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:29.508906    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:19:29.509027 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:29.509014    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:19:29.509130 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:29.509085    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:19:40.508384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:40.508353    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:19:40.508847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:40.508492    2561 scope.go:117] "RemoveContainer" containerID="45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da"
Jul 17 17:19:40.508847 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:40.508583    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:19:41.610135 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:41.610085    2561 generic.go:356] "Generic (PLEG): container finished" podID="66924833-7416-472b-abdf-72ab09c01244" containerID="844d01a0775670c73784edfd961f08de374eef28e8c4d418971e11ea2d49c6d9" exitCode=7
Jul 17 17:19:41.610530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:41.610157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" event={"ID":"66924833-7416-472b-abdf-72ab09c01244","Type":"ContainerDied","Data":"844d01a0775670c73784edfd961f08de374eef28e8c4d418971e11ea2d49c6d9"}
Jul 17 17:19:41.610530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:41.610242    2561 scope.go:117] "RemoveContainer" containerID="45d9b83cd1663b59fbc122a03d1b05795121a4770ac6efb1c3474a00cbe093da"
Jul 17 17:19:41.610530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:41.610485    2561 scope.go:117] "RemoveContainer" containerID="844d01a0775670c73784edfd961f08de374eef28e8c4d418971e11ea2d49c6d9"
Jul 17 17:19:41.610700 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:41.610681    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-6ngz6_odh-ai-gateway-infra(66924833-7416-472b-abdf-72ab09c01244)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" podUID="66924833-7416-472b-abdf-72ab09c01244"
Jul 17 17:19:42.508294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.508267    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:19:42.508469 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:42.508449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:19:42.644756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.644719    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:42.772319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.772266    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:42.786777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.786756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca\") pod \"66924833-7416-472b-abdf-72ab09c01244\" (UID: \"66924833-7416-472b-abdf-72ab09c01244\") "
Jul 17 17:19:42.787118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.787100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66924833-7416-472b-abdf-72ab09c01244" (UID: "66924833-7416-472b-abdf-72ab09c01244"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:19:42.887294 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:42.887263    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66924833-7416-472b-abdf-72ab09c01244-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:19:43.337175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337137    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:19:43.337598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337577    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337694 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337599    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337694 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337694 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337637    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337694 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337651    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337694 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337759    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337772    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.337946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.337783    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66924833-7416-472b-abdf-72ab09c01244" containerName="revoke-keys"
Jul 17 17:19:43.340797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.340776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:19:43.349661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.349638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:19:43.391338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.391303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4\" (UID: \"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:19:43.492105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.492065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4\" (UID: \"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:19:43.492640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.492622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4\" (UID: \"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:19:43.622371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.622286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6" event={"ID":"66924833-7416-472b-abdf-72ab09c01244","Type":"ContainerDied","Data":"c6c5dfd84c90c5443983872a762140cee3704fe7589a16af19201880bcbc41a6"}
Jul 17 17:19:43.622371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.622303    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"
Jul 17 17:19:43.622371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.622349    2561 scope.go:117] "RemoveContainer" containerID="844d01a0775670c73784edfd961f08de374eef28e8c4d418971e11ea2d49c6d9"
Jul 17 17:19:43.642346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.642321    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:43.648584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.648558    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-6ngz6"]
Jul 17 17:19:43.651698 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.651683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:19:43.780788 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:43.780758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:19:43.781973 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:19:43.781948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2073b6e5_d7b1_4717_9ade_f24f4d5ad18a.slice/crio-c6949a5b11f9f26797fea31dc239ec044804ff7d7368dac9f1ff45674416a613 WatchSource:0}: Error finding container c6949a5b11f9f26797fea31dc239ec044804ff7d7368dac9f1ff45674416a613: Status 404 returned error can't find the container with id c6949a5b11f9f26797fea31dc239ec044804ff7d7368dac9f1ff45674416a613
Jul 17 17:19:44.513599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:44.513559    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66924833-7416-472b-abdf-72ab09c01244" path="/var/lib/kubelet/pods/66924833-7416-472b-abdf-72ab09c01244/volumes"
Jul 17 17:19:44.630214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:44.630166    2561 generic.go:356] "Generic (PLEG): container finished" podID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerID="f15803ed8c302ba1cdb24c2b267cbccdcfbe2fd63e17a67779777d1a205aa7a0" exitCode=7
Jul 17 17:19:44.630382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:44.630226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" event={"ID":"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a","Type":"ContainerDied","Data":"f15803ed8c302ba1cdb24c2b267cbccdcfbe2fd63e17a67779777d1a205aa7a0"}
Jul 17 17:19:44.630382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:44.630273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" event={"ID":"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a","Type":"ContainerStarted","Data":"c6949a5b11f9f26797fea31dc239ec044804ff7d7368dac9f1ff45674416a613"}
Jul 17 17:19:44.630516 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:44.630500    2561 scope.go:117] "RemoveContainer" containerID="f15803ed8c302ba1cdb24c2b267cbccdcfbe2fd63e17a67779777d1a205aa7a0"
Jul 17 17:19:45.571114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.571029    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:19:45.574279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.574262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:19:45.582197 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.581778    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:19:45.611105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.611070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp\" (UID: \"3a43a9d9-1b3d-4391-9622-3f2c191848ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:19:45.636183 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.636154    2561 generic.go:356] "Generic (PLEG): container finished" podID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerID="4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb" exitCode=7
Jul 17 17:19:45.636291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.636228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" event={"ID":"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a","Type":"ContainerDied","Data":"4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb"}
Jul 17 17:19:45.636291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.636288    2561 scope.go:117] "RemoveContainer" containerID="f15803ed8c302ba1cdb24c2b267cbccdcfbe2fd63e17a67779777d1a205aa7a0"
Jul 17 17:19:45.636513 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.636497    2561 scope.go:117] "RemoveContainer" containerID="4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb"
Jul 17 17:19:45.636722 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:45.636700    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4_odh-ai-gateway-infra(2073b6e5-d7b1-4717-9ade-f24f4d5ad18a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a"
Jul 17 17:19:45.712299 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.712272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp\" (UID: \"3a43a9d9-1b3d-4391-9622-3f2c191848ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:19:45.712864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.712843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp\" (UID: \"3a43a9d9-1b3d-4391-9622-3f2c191848ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:19:45.885643 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:45.885554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:19:46.016802 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.016767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:19:46.022312 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:19:46.022282    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a43a9d9_1b3d_4391_9622_3f2c191848ca.slice/crio-b80ca7c86b585ecf10f4bcfca1b0f8f8c11c1f151d77149668df815844a03232 WatchSource:0}: Error finding container b80ca7c86b585ecf10f4bcfca1b0f8f8c11c1f151d77149668df815844a03232: Status 404 returned error can't find the container with id b80ca7c86b585ecf10f4bcfca1b0f8f8c11c1f151d77149668df815844a03232
Jul 17 17:19:46.642356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.642331    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerID="1c76548de1aefd71c646026a2dddc6f79c636e7743dd81907f5cb63777f945aa" exitCode=7
Jul 17 17:19:46.642675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.642407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" event={"ID":"3a43a9d9-1b3d-4391-9622-3f2c191848ca","Type":"ContainerDied","Data":"1c76548de1aefd71c646026a2dddc6f79c636e7743dd81907f5cb63777f945aa"}
Jul 17 17:19:46.642675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.642456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" event={"ID":"3a43a9d9-1b3d-4391-9622-3f2c191848ca","Type":"ContainerStarted","Data":"b80ca7c86b585ecf10f4bcfca1b0f8f8c11c1f151d77149668df815844a03232"}
Jul 17 17:19:46.642675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.642586    2561 scope.go:117] "RemoveContainer" containerID="1c76548de1aefd71c646026a2dddc6f79c636e7743dd81907f5cb63777f945aa"
Jul 17 17:19:46.644230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:46.644215    2561 scope.go:117] "RemoveContainer" containerID="4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb"
Jul 17 17:19:46.644386 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:46.644370    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4_odh-ai-gateway-infra(2073b6e5-d7b1-4717-9ade-f24f4d5ad18a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a"
Jul 17 17:19:47.651366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:47.651332    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerID="188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f" exitCode=7
Jul 17 17:19:47.651818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:47.651414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" event={"ID":"3a43a9d9-1b3d-4391-9622-3f2c191848ca","Type":"ContainerDied","Data":"188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f"}
Jul 17 17:19:47.651818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:47.651479    2561 scope.go:117] "RemoveContainer" containerID="1c76548de1aefd71c646026a2dddc6f79c636e7743dd81907f5cb63777f945aa"
Jul 17 17:19:47.651818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:47.651722    2561 scope.go:117] "RemoveContainer" containerID="188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f"
Jul 17 17:19:47.652017 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:47.651994    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp_odh-ai-gateway-infra(3a43a9d9-1b3d-4391-9622-3f2c191848ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca"
Jul 17 17:19:48.657972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:48.657942    2561 scope.go:117] "RemoveContainer" containerID="188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f"
Jul 17 17:19:48.658407 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:48.658151    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp_odh-ai-gateway-infra(3a43a9d9-1b3d-4391-9622-3f2c191848ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca"
Jul 17 17:19:54.508590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:54.508559    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:19:54.509076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:54.508655    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:19:54.509076 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:54.508794    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:19:54.509076 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:19:54.508798    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:19:57.145851 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:57.145807    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:19:59.509303 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:19:59.509219    2561 scope.go:117] "RemoveContainer" containerID="188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f"
Jul 17 17:20:00.509163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:00.509065    2561 scope.go:117] "RemoveContainer" containerID="4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb"
Jul 17 17:20:00.715938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:00.715905    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerID="7b916b6af42e51cd08bbd7c85cfa4927ca352ec8fe3ae368f5d5939eb5175708" exitCode=7
Jul 17 17:20:00.716401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:00.715960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" event={"ID":"3a43a9d9-1b3d-4391-9622-3f2c191848ca","Type":"ContainerDied","Data":"7b916b6af42e51cd08bbd7c85cfa4927ca352ec8fe3ae368f5d5939eb5175708"}
Jul 17 17:20:00.716401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:00.716057    2561 scope.go:117] "RemoveContainer" containerID="188c55fff6306933670b892f412d13288d4dba0dc1c2817887ab4706215a1a6f"
Jul 17 17:20:00.716401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:00.716393    2561 scope.go:117] "RemoveContainer" containerID="7b916b6af42e51cd08bbd7c85cfa4927ca352ec8fe3ae368f5d5939eb5175708"
Jul 17 17:20:00.716623 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:00.716603    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp_odh-ai-gateway-infra(3a43a9d9-1b3d-4391-9622-3f2c191848ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca"
Jul 17 17:20:01.723333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.723291    2561 generic.go:356] "Generic (PLEG): container finished" podID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerID="8b8c08ad9499fa0f82425090114ac94a8fd9dd140c2c173afa33def0d0fb98e5" exitCode=7
Jul 17 17:20:01.723333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.723336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" event={"ID":"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a","Type":"ContainerDied","Data":"8b8c08ad9499fa0f82425090114ac94a8fd9dd140c2c173afa33def0d0fb98e5"}
Jul 17 17:20:01.723827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.723395    2561 scope.go:117] "RemoveContainer" containerID="4f6fde17a7c6dcb0b1efa24b07ae499fa2b9d992b234c5b3ce1e89861b4b6cbb"
Jul 17 17:20:01.723827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.723732    2561 scope.go:117] "RemoveContainer" containerID="8b8c08ad9499fa0f82425090114ac94a8fd9dd140c2c173afa33def0d0fb98e5"
Jul 17 17:20:01.723966 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:01.723946    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4_odh-ai-gateway-infra(2073b6e5-d7b1-4717-9ade-f24f4d5ad18a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a"
Jul 17 17:20:01.743907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.743878    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:20:01.879544 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.879517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:20:01.969856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.969816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca\") pod \"3a43a9d9-1b3d-4391-9622-3f2c191848ca\" (UID: \"3a43a9d9-1b3d-4391-9622-3f2c191848ca\") "
Jul 17 17:20:01.970216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:01.970194    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a43a9d9-1b3d-4391-9622-3f2c191848ca" (UID: "3a43a9d9-1b3d-4391-9622-3f2c191848ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:20:02.070882 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.070845    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a43a9d9-1b3d-4391-9622-3f2c191848ca-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:20:02.731912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.731880    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"
Jul 17 17:20:02.732366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.731880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp" event={"ID":"3a43a9d9-1b3d-4391-9622-3f2c191848ca","Type":"ContainerDied","Data":"b80ca7c86b585ecf10f4bcfca1b0f8f8c11c1f151d77149668df815844a03232"}
Jul 17 17:20:02.732366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.732025    2561 scope.go:117] "RemoveContainer" containerID="7b916b6af42e51cd08bbd7c85cfa4927ca352ec8fe3ae368f5d5939eb5175708"
Jul 17 17:20:02.750875 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.750845    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:20:02.759377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.759339    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:20:02.763730 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.763703    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hnzjp"]
Jul 17 17:20:02.906782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.906755    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:20:02.979501 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.979461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca\") pod \"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a\" (UID: \"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a\") "
Jul 17 17:20:02.979809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:02.979784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" (UID: "2073b6e5-d7b1-4717-9ade-f24f4d5ad18a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:20:03.081324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.081238    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:20:03.738344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.738311    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"
Jul 17 17:20:03.738344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.738339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4" event={"ID":"2073b6e5-d7b1-4717-9ade-f24f4d5ad18a","Type":"ContainerDied","Data":"c6949a5b11f9f26797fea31dc239ec044804ff7d7368dac9f1ff45674416a613"}
Jul 17 17:20:03.738855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.738403    2561 scope.go:117] "RemoveContainer" containerID="8b8c08ad9499fa0f82425090114ac94a8fd9dd140c2c173afa33def0d0fb98e5"
Jul 17 17:20:03.761272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.761244    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:20:03.767431 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:03.767403    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-h5dn4"]
Jul 17 17:20:04.514740 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:04.514703    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" path="/var/lib/kubelet/pods/2073b6e5-d7b1-4717-9ade-f24f4d5ad18a/volumes"
Jul 17 17:20:04.515080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:04.515063    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" path="/var/lib/kubelet/pods/3a43a9d9-1b3d-4391-9622-3f2c191848ca/volumes"
Jul 17 17:20:07.509234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:07.509199    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:20:07.509630 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:07.509387    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:20:08.509133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:08.509099    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:20:08.509311 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:08.509254    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:20:15.452518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.452478    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:15.453016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.452998    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453020    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453034    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453058    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453075    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453109    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453116    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453133    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453141    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453150    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453157    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453254    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453267    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453278    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2073b6e5-d7b1-4717-9ade-f24f4d5ad18a" containerName="revoke-keys"
Jul 17 17:20:15.453419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.453419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.453304    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:15.457839 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.457819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:15.460243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.460219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:20:15.465509 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.464971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:15.601261 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.601222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-r8tmx\" (UID: \"1a07846a-31a2-41a1-9065-3b0df667d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:15.702306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.702271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-r8tmx\" (UID: \"1a07846a-31a2-41a1-9065-3b0df667d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:15.702881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.702836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-r8tmx\" (UID: \"1a07846a-31a2-41a1-9065-3b0df667d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:15.769434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.769404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:15.904325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:15.904299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:15.905989 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:20:15.905953    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a07846a_31a2_41a1_9065_3b0df667d34d.slice/crio-44cfc4b4977ee1ac30f04829b6590d9cfcbfd14b364819db1afac221e6448dc4 WatchSource:0}: Error finding container 44cfc4b4977ee1ac30f04829b6590d9cfcbfd14b364819db1afac221e6448dc4: Status 404 returned error can't find the container with id 44cfc4b4977ee1ac30f04829b6590d9cfcbfd14b364819db1afac221e6448dc4
Jul 17 17:20:16.805215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:16.805179    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerID="f3453ff2997d223f9063ccc180361c1646b0c1c6faf219738771eaa9302ed1bd" exitCode=7
Jul 17 17:20:16.805605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:16.805226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerDied","Data":"f3453ff2997d223f9063ccc180361c1646b0c1c6faf219738771eaa9302ed1bd"}
Jul 17 17:20:16.805605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:16.805264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerStarted","Data":"44cfc4b4977ee1ac30f04829b6590d9cfcbfd14b364819db1afac221e6448dc4"}
Jul 17 17:20:16.805884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:16.805859    2561 scope.go:117] "RemoveContainer" containerID="f3453ff2997d223f9063ccc180361c1646b0c1c6faf219738771eaa9302ed1bd"
Jul 17 17:20:17.812118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:17.812078    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerID="9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848" exitCode=7
Jul 17 17:20:17.812577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:17.812127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerDied","Data":"9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848"}
Jul 17 17:20:17.812577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:17.812218    2561 scope.go:117] "RemoveContainer" containerID="f3453ff2997d223f9063ccc180361c1646b0c1c6faf219738771eaa9302ed1bd"
Jul 17 17:20:17.812577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:17.812406    2561 scope.go:117] "RemoveContainer" containerID="9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848"
Jul 17 17:20:17.812703 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:17.812624    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-r8tmx_odh-ai-gateway-infra(1a07846a-31a2-41a1-9065-3b0df667d34d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d"
Jul 17 17:20:18.818123 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:18.818098    2561 scope.go:117] "RemoveContainer" containerID="9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848"
Jul 17 17:20:18.818490 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:18.818308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-r8tmx_odh-ai-gateway-infra(1a07846a-31a2-41a1-9065-3b0df667d34d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d"
Jul 17 17:20:22.509138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:22.509107    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:20:22.509618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:22.509179    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:20:22.509618 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:22.509347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:20:22.509618 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:22.509368    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:20:30.888400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:30.888366    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:30.889067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:30.888773    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a43a9d9-1b3d-4391-9622-3f2c191848ca" containerName="revoke-keys"
Jul 17 17:20:30.893031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:30.893013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:30.900223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:30.899914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:30.942279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:30.942243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w\" (UID: \"185a8a6e-67c7-40af-917c-926d98ee8c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:31.043342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.043302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w\" (UID: \"185a8a6e-67c7-40af-917c-926d98ee8c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:31.043974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.043951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w\" (UID: \"185a8a6e-67c7-40af-917c-926d98ee8c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:31.206198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.206113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:31.544402 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.544378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:31.546547 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:20:31.546521    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185a8a6e_67c7_40af_917c_926d98ee8c87.slice/crio-c740f102c463dfc6d44228f5e3c1b4f86762145b7e8df771bf896bc5c2ba0848 WatchSource:0}: Error finding container c740f102c463dfc6d44228f5e3c1b4f86762145b7e8df771bf896bc5c2ba0848: Status 404 returned error can't find the container with id c740f102c463dfc6d44228f5e3c1b4f86762145b7e8df771bf896bc5c2ba0848
Jul 17 17:20:31.879959 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.879932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerStarted","Data":"17cc3110eb5ef65cf3b7b890b6b258688e3a9775857a1140d0e5bfdc4841af98"}
Jul 17 17:20:31.880123 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.879984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerStarted","Data":"c740f102c463dfc6d44228f5e3c1b4f86762145b7e8df771bf896bc5c2ba0848"}
Jul 17 17:20:31.895961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:31.895918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" podStartSLOduration=1.895904673 podStartE2EDuration="1.895904673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:20:31.89395384 +0000 UTC m=+3265.879041389" watchObservedRunningTime="2026-07-17 17:20:31.895904673 +0000 UTC m=+3265.880992257"
Jul 17 17:20:32.508704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.508618    2561 scope.go:117] "RemoveContainer" containerID="9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848"
Jul 17 17:20:32.886057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.886003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerStarted","Data":"6e8e8d008f343b4cf063463765ebc3de7eef7ad91913d854c96c829361f8af13"}
Jul 17 17:20:32.887428 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.887403    2561 generic.go:356] "Generic (PLEG): container finished" podID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerID="17cc3110eb5ef65cf3b7b890b6b258688e3a9775857a1140d0e5bfdc4841af98" exitCode=7
Jul 17 17:20:32.887541 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.887460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerDied","Data":"17cc3110eb5ef65cf3b7b890b6b258688e3a9775857a1140d0e5bfdc4841af98"}
Jul 17 17:20:32.887637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.887627    2561 scope.go:117] "RemoveContainer" containerID="17cc3110eb5ef65cf3b7b890b6b258688e3a9775857a1140d0e5bfdc4841af98"
Jul 17 17:20:32.904241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:32.904202    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" podStartSLOduration=17.904187835 podStartE2EDuration="17.904187835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:20:32.902687387 +0000 UTC m=+3266.887774925" watchObservedRunningTime="2026-07-17 17:20:32.904187835 +0000 UTC m=+3266.889275372"
Jul 17 17:20:33.893854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.893770    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerID="6e8e8d008f343b4cf063463765ebc3de7eef7ad91913d854c96c829361f8af13" exitCode=7
Jul 17 17:20:33.893854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.893842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerDied","Data":"6e8e8d008f343b4cf063463765ebc3de7eef7ad91913d854c96c829361f8af13"}
Jul 17 17:20:33.894086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.893907    2561 scope.go:117] "RemoveContainer" containerID="9002a1c6b313a20755c57885de11ffe1a6d501360e25245b911857f641647848"
Jul 17 17:20:33.894249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.894228    2561 scope.go:117] "RemoveContainer" containerID="6e8e8d008f343b4cf063463765ebc3de7eef7ad91913d854c96c829361f8af13"
Jul 17 17:20:33.894478 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:33.894458    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-r8tmx_odh-ai-gateway-infra(1a07846a-31a2-41a1-9065-3b0df667d34d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d"
Jul 17 17:20:33.895761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.895738    2561 generic.go:356] "Generic (PLEG): container finished" podID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerID="f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8" exitCode=7
Jul 17 17:20:33.895841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.895790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerDied","Data":"f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8"}
Jul 17 17:20:33.895992 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.895980    2561 scope.go:117] "RemoveContainer" containerID="f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8"
Jul 17 17:20:33.896188 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:33.896170    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w_odh-ai-gateway-infra(185a8a6e-67c7-40af-917c-926d98ee8c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87"
Jul 17 17:20:33.904518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.904504    2561 scope.go:117] "RemoveContainer" containerID="17cc3110eb5ef65cf3b7b890b6b258688e3a9775857a1140d0e5bfdc4841af98"
Jul 17 17:20:33.921772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:33.921750    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:34.508385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:34.508355    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:20:34.508561 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:34.508533    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:20:34.901985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:34.901952    2561 scope.go:117] "RemoveContainer" containerID="f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8"
Jul 17 17:20:34.902241 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:34.902215    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w_odh-ai-gateway-infra(185a8a6e-67c7-40af-917c-926d98ee8c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87"
Jul 17 17:20:35.040848 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.040825    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:35.082328 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.082294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca\") pod \"1a07846a-31a2-41a1-9065-3b0df667d34d\" (UID: \"1a07846a-31a2-41a1-9065-3b0df667d34d\") "
Jul 17 17:20:35.082658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.082637    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a07846a-31a2-41a1-9065-3b0df667d34d" (UID: "1a07846a-31a2-41a1-9065-3b0df667d34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:20:35.183614 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.183524    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a07846a-31a2-41a1-9065-3b0df667d34d-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:20:35.509253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.509177    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:20:35.509402 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:35.509321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:20:35.608054 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608018    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:35.608392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608380    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.608436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608394    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.608436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608409    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.608436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608414    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.608533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.608533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.608492    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:35.611386 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.611371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:35.617639 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.617598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:35.687397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.687362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8\" (UID: \"64544642-7248-44b4-8838-3c2312827b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:35.788117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.788076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8\" (UID: \"64544642-7248-44b4-8838-3c2312827b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:35.788720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.788698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8\" (UID: \"64544642-7248-44b4-8838-3c2312827b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:35.908687 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.908658    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"
Jul 17 17:20:35.908687 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.908683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx" event={"ID":"1a07846a-31a2-41a1-9065-3b0df667d34d","Type":"ContainerDied","Data":"44cfc4b4977ee1ac30f04829b6590d9cfcbfd14b364819db1afac221e6448dc4"}
Jul 17 17:20:35.908919 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.908740    2561 scope.go:117] "RemoveContainer" containerID="6e8e8d008f343b4cf063463765ebc3de7eef7ad91913d854c96c829361f8af13"
Jul 17 17:20:35.922970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.922950    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:35.931749 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.931724    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:35.936285 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:35.936162    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-r8tmx"]
Jul 17 17:20:36.058356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.058179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:36.060939 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:20:36.060907    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64544642_7248_44b4_8838_3c2312827b4e.slice/crio-5c2235dea3db24dc4f20388f83f2b5778595b94a0e65e0a9e8c94aa7cba11050 WatchSource:0}: Error finding container 5c2235dea3db24dc4f20388f83f2b5778595b94a0e65e0a9e8c94aa7cba11050: Status 404 returned error can't find the container with id 5c2235dea3db24dc4f20388f83f2b5778595b94a0e65e0a9e8c94aa7cba11050
Jul 17 17:20:36.515665 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.515623    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" path="/var/lib/kubelet/pods/1a07846a-31a2-41a1-9065-3b0df667d34d/volumes"
Jul 17 17:20:36.915457 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.915369    2561 generic.go:356] "Generic (PLEG): container finished" podID="64544642-7248-44b4-8838-3c2312827b4e" containerID="d7ee8ff0f29e53658f27cf51667cad98f90c49bf3ca2fb291032e888e4a6275b" exitCode=7
Jul 17 17:20:36.915636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.915451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerDied","Data":"d7ee8ff0f29e53658f27cf51667cad98f90c49bf3ca2fb291032e888e4a6275b"}
Jul 17 17:20:36.915636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.915506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerStarted","Data":"5c2235dea3db24dc4f20388f83f2b5778595b94a0e65e0a9e8c94aa7cba11050"}
Jul 17 17:20:36.915739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:36.915681    2561 scope.go:117] "RemoveContainer" containerID="d7ee8ff0f29e53658f27cf51667cad98f90c49bf3ca2fb291032e888e4a6275b"
Jul 17 17:20:37.922797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:37.922712    2561 generic.go:356] "Generic (PLEG): container finished" podID="64544642-7248-44b4-8838-3c2312827b4e" containerID="7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f" exitCode=7
Jul 17 17:20:37.923253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:37.922799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerDied","Data":"7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f"}
Jul 17 17:20:37.923253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:37.922871    2561 scope.go:117] "RemoveContainer" containerID="d7ee8ff0f29e53658f27cf51667cad98f90c49bf3ca2fb291032e888e4a6275b"
Jul 17 17:20:37.923253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:37.922975    2561 scope.go:117] "RemoveContainer" containerID="7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f"
Jul 17 17:20:37.923253 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:37.923231    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8_odh-ai-gateway-infra(64544642-7248-44b4-8838-3c2312827b4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" podUID="64544642-7248-44b4-8838-3c2312827b4e"
Jul 17 17:20:38.929688 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:38.929665    2561 scope.go:117] "RemoveContainer" containerID="7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f"
Jul 17 17:20:38.930082 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:38.929845    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8_odh-ai-gateway-infra(64544642-7248-44b4-8838-3c2312827b4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" podUID="64544642-7248-44b4-8838-3c2312827b4e"
Jul 17 17:20:43.380534 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.380496    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:20:43.380949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.380896    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:43.380949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.380909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:43.381029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.380960    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a07846a-31a2-41a1-9065-3b0df667d34d" containerName="revoke-keys"
Jul 17 17:20:43.385274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.385255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:20:43.392690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.392204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:20:43.459727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.459687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n\" (UID: \"e203d77b-75cd-456c-bea5-43143483db56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:20:43.560886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.560843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n\" (UID: \"e203d77b-75cd-456c-bea5-43143483db56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:20:43.561488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.561467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n\" (UID: \"e203d77b-75cd-456c-bea5-43143483db56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:20:43.697276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.697190    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:20:43.844678 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.844632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:20:43.848360 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:20:43.848332    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode203d77b_75cd_456c_bea5_43143483db56.slice/crio-852da637bf92b0737cec14e8b9e9c9a431a439551a0b0f20ae5f2415f041b4cf WatchSource:0}: Error finding container 852da637bf92b0737cec14e8b9e9c9a431a439551a0b0f20ae5f2415f041b4cf: Status 404 returned error can't find the container with id 852da637bf92b0737cec14e8b9e9c9a431a439551a0b0f20ae5f2415f041b4cf
Jul 17 17:20:43.955349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:43.955256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerStarted","Data":"852da637bf92b0737cec14e8b9e9c9a431a439551a0b0f20ae5f2415f041b4cf"}
Jul 17 17:20:44.960782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:44.960746    2561 generic.go:356] "Generic (PLEG): container finished" podID="e203d77b-75cd-456c-bea5-43143483db56" containerID="62a58c8a30f76e23abb292626608282a438bccfddb0b794bc578ba085d0db6d6" exitCode=7
Jul 17 17:20:44.961249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:44.960827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerDied","Data":"62a58c8a30f76e23abb292626608282a438bccfddb0b794bc578ba085d0db6d6"}
Jul 17 17:20:44.961249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:44.961064    2561 scope.go:117] "RemoveContainer" containerID="62a58c8a30f76e23abb292626608282a438bccfddb0b794bc578ba085d0db6d6"
Jul 17 17:20:45.508703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:45.508670    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:20:45.508901 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:45.508856    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:20:45.967665 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:45.967626    2561 generic.go:356] "Generic (PLEG): container finished" podID="e203d77b-75cd-456c-bea5-43143483db56" containerID="f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455" exitCode=7
Jul 17 17:20:45.968170 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:45.967724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerDied","Data":"f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455"}
Jul 17 17:20:45.968170 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:45.967801    2561 scope.go:117] "RemoveContainer" containerID="62a58c8a30f76e23abb292626608282a438bccfddb0b794bc578ba085d0db6d6"
Jul 17 17:20:45.968170 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:45.967924    2561 scope.go:117] "RemoveContainer" containerID="f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455"
Jul 17 17:20:45.968297 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:45.968190    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n_odh-ai-gateway-infra(e203d77b-75cd-456c-bea5-43143483db56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" podUID="e203d77b-75cd-456c-bea5-43143483db56"
Jul 17 17:20:46.974585 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:46.974550    2561 scope.go:117] "RemoveContainer" containerID="f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455"
Jul 17 17:20:46.974957 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:46.974788    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n_odh-ai-gateway-infra(e203d77b-75cd-456c-bea5-43143483db56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" podUID="e203d77b-75cd-456c-bea5-43143483db56"
Jul 17 17:20:49.508370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:49.508339    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:20:49.508762 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:49.508530    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:20:49.508762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:49.508647    2561 scope.go:117] "RemoveContainer" containerID="f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8"
Jul 17 17:20:49.990382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:49.990349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerStarted","Data":"be502f3d5c2e09857bcb3fca2f3ff548581dbbf886e3bd26aea13ec2f1180f5b"}
Jul 17 17:20:50.996695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:50.996660    2561 generic.go:356] "Generic (PLEG): container finished" podID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerID="be502f3d5c2e09857bcb3fca2f3ff548581dbbf886e3bd26aea13ec2f1180f5b" exitCode=7
Jul 17 17:20:50.997211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:50.996732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerDied","Data":"be502f3d5c2e09857bcb3fca2f3ff548581dbbf886e3bd26aea13ec2f1180f5b"}
Jul 17 17:20:50.997211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:50.996807    2561 scope.go:117] "RemoveContainer" containerID="f534b3e5e199059e3d4c242d4c656398e34d60c276a6e050790bd6def079f2f8"
Jul 17 17:20:50.997211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:50.997126    2561 scope.go:117] "RemoveContainer" containerID="be502f3d5c2e09857bcb3fca2f3ff548581dbbf886e3bd26aea13ec2f1180f5b"
Jul 17 17:20:50.997408 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:50.997387    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w_odh-ai-gateway-infra(185a8a6e-67c7-40af-917c-926d98ee8c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87"
Jul 17 17:20:51.022974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:51.022941    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:51.509174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:51.509140    2561 scope.go:117] "RemoveContainer" containerID="7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f"
Jul 17 17:20:52.004984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.004953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerStarted","Data":"2f912b8c256833363bf3cb921511243825e3be6b573788159b9c120fe1afeda0"}
Jul 17 17:20:52.033563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.033508    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" podStartSLOduration=17.033491313 podStartE2EDuration="17.033491313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:20:52.032363695 +0000 UTC m=+3286.017451256" watchObservedRunningTime="2026-07-17 17:20:52.033491313 +0000 UTC m=+3286.018578870"
Jul 17 17:20:52.191395 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.191370    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:52.343150 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.343114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca\") pod \"185a8a6e-67c7-40af-917c-926d98ee8c87\" (UID: \"185a8a6e-67c7-40af-917c-926d98ee8c87\") "
Jul 17 17:20:52.343475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.343454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "185a8a6e-67c7-40af-917c-926d98ee8c87" (UID: "185a8a6e-67c7-40af-917c-926d98ee8c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:20:52.443952 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:52.443910    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a8a6e-67c7-40af-917c-926d98ee8c87-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:20:53.010808 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.010778    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"
Jul 17 17:20:53.010808 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.010805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w" event={"ID":"185a8a6e-67c7-40af-917c-926d98ee8c87","Type":"ContainerDied","Data":"c740f102c463dfc6d44228f5e3c1b4f86762145b7e8df771bf896bc5c2ba0848"}
Jul 17 17:20:53.011348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.010865    2561 scope.go:117] "RemoveContainer" containerID="be502f3d5c2e09857bcb3fca2f3ff548581dbbf886e3bd26aea13ec2f1180f5b"
Jul 17 17:20:53.012727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.012701    2561 generic.go:356] "Generic (PLEG): container finished" podID="64544642-7248-44b4-8838-3c2312827b4e" containerID="2f912b8c256833363bf3cb921511243825e3be6b573788159b9c120fe1afeda0" exitCode=7
Jul 17 17:20:53.012844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.012745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerDied","Data":"2f912b8c256833363bf3cb921511243825e3be6b573788159b9c120fe1afeda0"}
Jul 17 17:20:53.013193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.013160    2561 scope.go:117] "RemoveContainer" containerID="2f912b8c256833363bf3cb921511243825e3be6b573788159b9c120fe1afeda0"
Jul 17 17:20:53.013418 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:53.013394    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8_odh-ai-gateway-infra(64544642-7248-44b4-8838-3c2312827b4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" podUID="64544642-7248-44b4-8838-3c2312827b4e"
Jul 17 17:20:53.019614 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.019597    2561 scope.go:117] "RemoveContainer" containerID="7463aa88f70f8c686d9d6fb33fbc064961ea1346750ae3c67991afb2ccf7149f"
Jul 17 17:20:53.027640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.027617    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:53.033641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.033617    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6454w"]
Jul 17 17:20:53.041746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:53.041725    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:54.149012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:54.148981    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:54.261075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:54.261007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca\") pod \"64544642-7248-44b4-8838-3c2312827b4e\" (UID: \"64544642-7248-44b4-8838-3c2312827b4e\") "
Jul 17 17:20:54.261384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:54.261361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64544642-7248-44b4-8838-3c2312827b4e" (UID: "64544642-7248-44b4-8838-3c2312827b4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:20:54.361967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:54.361936    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64544642-7248-44b4-8838-3c2312827b4e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:20:54.514219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:54.514186    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" path="/var/lib/kubelet/pods/185a8a6e-67c7-40af-917c-926d98ee8c87/volumes"
Jul 17 17:20:55.025882 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:55.025853    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"
Jul 17 17:20:55.026098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:55.025885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8" event={"ID":"64544642-7248-44b4-8838-3c2312827b4e","Type":"ContainerDied","Data":"5c2235dea3db24dc4f20388f83f2b5778595b94a0e65e0a9e8c94aa7cba11050"}
Jul 17 17:20:55.026098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:55.025952    2561 scope.go:117] "RemoveContainer" containerID="2f912b8c256833363bf3cb921511243825e3be6b573788159b9c120fe1afeda0"
Jul 17 17:20:55.044020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:55.043993    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:55.048933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:55.048905    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-nq8l8"]
Jul 17 17:20:56.514767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:56.514731    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64544642-7248-44b4-8838-3c2312827b4e" path="/var/lib/kubelet/pods/64544642-7248-44b4-8838-3c2312827b4e/volumes"
Jul 17 17:20:58.508559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:58.508521    2561 scope.go:117] "RemoveContainer" containerID="f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455"
Jul 17 17:20:59.047161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:59.047121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerStarted","Data":"05a8fe2daa9d0fb7900adba858e3ee20ef44d6b4074833624f065b0273ffa9fe"}
Jul 17 17:20:59.072738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:59.072635    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" podStartSLOduration=16.072615961 podStartE2EDuration="16.072615961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:20:59.070401336 +0000 UTC m=+3293.055488912" watchObservedRunningTime="2026-07-17 17:20:59.072615961 +0000 UTC m=+3293.057703500"
Jul 17 17:20:59.508915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:20:59.508829    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:20:59.509338 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:20:59.509040    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:21:00.053445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:00.053411    2561 generic.go:356] "Generic (PLEG): container finished" podID="e203d77b-75cd-456c-bea5-43143483db56" containerID="05a8fe2daa9d0fb7900adba858e3ee20ef44d6b4074833624f065b0273ffa9fe" exitCode=7
Jul 17 17:21:00.053600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:00.053481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerDied","Data":"05a8fe2daa9d0fb7900adba858e3ee20ef44d6b4074833624f065b0273ffa9fe"}
Jul 17 17:21:00.053600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:00.053542    2561 scope.go:117] "RemoveContainer" containerID="f9fa47646776ba2aa44847c1eba88213574ccd2db4a5dd2cf5231087787cb455"
Jul 17 17:21:00.053853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:00.053837    2561 scope.go:117] "RemoveContainer" containerID="05a8fe2daa9d0fb7900adba858e3ee20ef44d6b4074833624f065b0273ffa9fe"
Jul 17 17:21:00.054092 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:00.054073    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n_odh-ai-gateway-infra(e203d77b-75cd-456c-bea5-43143483db56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" podUID="e203d77b-75cd-456c-bea5-43143483db56"
Jul 17 17:21:00.086351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:00.086320    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:21:01.196228 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:01.196198    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:21:01.226367 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:01.226337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca\") pod \"e203d77b-75cd-456c-bea5-43143483db56\" (UID: \"e203d77b-75cd-456c-bea5-43143483db56\") "
Jul 17 17:21:01.226681 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:01.226658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e203d77b-75cd-456c-bea5-43143483db56" (UID: "e203d77b-75cd-456c-bea5-43143483db56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:21:01.327433 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:01.327342    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e203d77b-75cd-456c-bea5-43143483db56-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:21:01.509000 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:01.508964    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:21:02.065348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.065316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"
Jul 17 17:21:02.065348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.065326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n" event={"ID":"e203d77b-75cd-456c-bea5-43143483db56","Type":"ContainerDied","Data":"852da637bf92b0737cec14e8b9e9c9a431a439551a0b0f20ae5f2415f041b4cf"}
Jul 17 17:21:02.065655 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.065391    2561 scope.go:117] "RemoveContainer" containerID="05a8fe2daa9d0fb7900adba858e3ee20ef44d6b4074833624f065b0273ffa9fe"
Jul 17 17:21:02.067379 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.067355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerStarted","Data":"57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"}
Jul 17 17:21:02.067582 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.067564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:21:02.095772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.095746    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:21:02.099682 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.099662    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-qfb9n"]
Jul 17 17:21:02.513662 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:02.513582    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e203d77b-75cd-456c-bea5-43143483db56" path="/var/lib/kubelet/pods/e203d77b-75cd-456c-bea5-43143483db56/volumes"
Jul 17 17:21:03.075132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:03.075101    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38" exitCode=1
Jul 17 17:21:03.075342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:03.075142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"}
Jul 17 17:21:03.075342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:03.075190    2561 scope.go:117] "RemoveContainer" containerID="1006910d23971ed430508803d1fb05e3f04950dd620639e42edbe458eb67289a"
Jul 17 17:21:03.075449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:03.075409    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:03.075629 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:03.075611    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:04.080951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:04.080923    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:04.081381 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:04.081153    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:04.969000 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:04.968961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:21:05.086423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:05.086388    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:05.086883 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:05.086618    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:06.616105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:06.616076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:21:06.629958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:06.629935    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:21:11.508461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:11.508429    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:21:11.508867 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:11.508636    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:21:15.505311 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505279    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:15.505805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505767    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.505805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505786    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505817    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505826    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505846    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505855    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505871    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505879    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505910    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.505923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505930    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.505957    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506038    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506069    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506081    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506089    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506097    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:15.506507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.506112    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:15.510800 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.510781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:15.513330 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.513312    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:21:15.517656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.517634    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:15.551574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.551540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qndwd\" (UID: \"c0b744bc-b450-454c-b852-31e5a61c782e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:15.652947 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.652908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qndwd\" (UID: \"c0b744bc-b450-454c-b852-31e5a61c782e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:15.653526 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.653508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-qndwd\" (UID: \"c0b744bc-b450-454c-b852-31e5a61c782e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:15.821656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.821627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:15.959725 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:15.959703    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:15.961838 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:21:15.961807    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b744bc_b450_454c_b852_31e5a61c782e.slice/crio-b86a121d32c4947780c721fb66cc185b041aade86880ce8744b634adf3638feb WatchSource:0}: Error finding container b86a121d32c4947780c721fb66cc185b041aade86880ce8744b634adf3638feb: Status 404 returned error can't find the container with id b86a121d32c4947780c721fb66cc185b041aade86880ce8744b634adf3638feb
Jul 17 17:21:16.140007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:16.139974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerStarted","Data":"aad44e3c1797f35bb77804665f356d727266146cd312a16e7641079dac13cf9a"}
Jul 17 17:21:16.140155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:16.140026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerStarted","Data":"b86a121d32c4947780c721fb66cc185b041aade86880ce8744b634adf3638feb"}
Jul 17 17:21:16.155997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:16.155952    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" podStartSLOduration=1.155929219 podStartE2EDuration="1.155929219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:21:16.155435577 +0000 UTC m=+3310.140523115" watchObservedRunningTime="2026-07-17 17:21:16.155929219 +0000 UTC m=+3310.141016758"
Jul 17 17:21:17.146026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:17.145987    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0b744bc-b450-454c-b852-31e5a61c782e" containerID="aad44e3c1797f35bb77804665f356d727266146cd312a16e7641079dac13cf9a" exitCode=7
Jul 17 17:21:17.146454 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:17.146088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerDied","Data":"aad44e3c1797f35bb77804665f356d727266146cd312a16e7641079dac13cf9a"}
Jul 17 17:21:17.146454 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:17.146265    2561 scope.go:117] "RemoveContainer" containerID="aad44e3c1797f35bb77804665f356d727266146cd312a16e7641079dac13cf9a"
Jul 17 17:21:18.152896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:18.152792    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0b744bc-b450-454c-b852-31e5a61c782e" containerID="3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1" exitCode=7
Jul 17 17:21:18.152896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:18.152882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerDied","Data":"3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1"}
Jul 17 17:21:18.153397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:18.152969    2561 scope.go:117] "RemoveContainer" containerID="aad44e3c1797f35bb77804665f356d727266146cd312a16e7641079dac13cf9a"
Jul 17 17:21:18.153397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:18.153232    2561 scope.go:117] "RemoveContainer" containerID="3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1"
Jul 17 17:21:18.153521 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:18.153480    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qndwd_odh-ai-gateway-infra(c0b744bc-b450-454c-b852-31e5a61c782e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" podUID="c0b744bc-b450-454c-b852-31e5a61c782e"
Jul 17 17:21:18.508647 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:18.508554    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:18.508821 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:18.508753    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:19.160292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:19.160265    2561 scope.go:117] "RemoveContainer" containerID="3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1"
Jul 17 17:21:19.160705 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:19.160476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qndwd_odh-ai-gateway-infra(c0b744bc-b450-454c-b852-31e5a61c782e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" podUID="c0b744bc-b450-454c-b852-31e5a61c782e"
Jul 17 17:21:22.252387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:22.252345    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:21:22.350855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:22.350804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:21:24.508831 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:24.508798    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:21:24.509311 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:24.509093    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:21:25.656779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.656746    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:25.657210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.657136    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:25.657210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.657148    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:25.657210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.657208    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e203d77b-75cd-456c-bea5-43143483db56" containerName="revoke-keys"
Jul 17 17:21:25.657321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.657222    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="64544642-7248-44b4-8838-3c2312827b4e" containerName="revoke-keys"
Jul 17 17:21:25.657321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.657228    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="185a8a6e-67c7-40af-917c-926d98ee8c87" containerName="revoke-keys"
Jul 17 17:21:25.661657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.661636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:25.666901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.666865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:25.747540 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.747506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5\" (UID: \"c2750356-9aba-46de-8c98-1cd1333e9569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:25.848125 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.848091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5\" (UID: \"c2750356-9aba-46de-8c98-1cd1333e9569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:25.848915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.848889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5\" (UID: \"c2750356-9aba-46de-8c98-1cd1333e9569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:25.973824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:25.973739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:26.315348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:26.315319    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:26.316742 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:21:26.316714    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2750356_9aba_46de_8c98_1cd1333e9569.slice/crio-c6a45ad15f129ffbb45c678fa556c59adbf74d7c1b69576c7e75309c6d8b6ba6 WatchSource:0}: Error finding container c6a45ad15f129ffbb45c678fa556c59adbf74d7c1b69576c7e75309c6d8b6ba6: Status 404 returned error can't find the container with id c6a45ad15f129ffbb45c678fa556c59adbf74d7c1b69576c7e75309c6d8b6ba6
Jul 17 17:21:27.200300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:27.200258    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2750356-9aba-46de-8c98-1cd1333e9569" containerID="ea08a3987f467f9ccf59e3b6f753f195d3fc3fbb8de347f85d8cad7db8548d5c" exitCode=7
Jul 17 17:21:27.200804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:27.200295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" event={"ID":"c2750356-9aba-46de-8c98-1cd1333e9569","Type":"ContainerDied","Data":"ea08a3987f467f9ccf59e3b6f753f195d3fc3fbb8de347f85d8cad7db8548d5c"}
Jul 17 17:21:27.200804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:27.200362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" event={"ID":"c2750356-9aba-46de-8c98-1cd1333e9569","Type":"ContainerStarted","Data":"c6a45ad15f129ffbb45c678fa556c59adbf74d7c1b69576c7e75309c6d8b6ba6"}
Jul 17 17:21:27.200804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:27.200581    2561 scope.go:117] "RemoveContainer" containerID="ea08a3987f467f9ccf59e3b6f753f195d3fc3fbb8de347f85d8cad7db8548d5c"
Jul 17 17:21:28.207346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:28.207225    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2750356-9aba-46de-8c98-1cd1333e9569" containerID="6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e" exitCode=7
Jul 17 17:21:28.207346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:28.207295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" event={"ID":"c2750356-9aba-46de-8c98-1cd1333e9569","Type":"ContainerDied","Data":"6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e"}
Jul 17 17:21:28.207829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:28.207361    2561 scope.go:117] "RemoveContainer" containerID="ea08a3987f467f9ccf59e3b6f753f195d3fc3fbb8de347f85d8cad7db8548d5c"
Jul 17 17:21:28.207829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:28.207607    2561 scope.go:117] "RemoveContainer" containerID="6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e"
Jul 17 17:21:28.207905 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:28.207828    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5_odh-ai-gateway-infra(c2750356-9aba-46de-8c98-1cd1333e9569)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" podUID="c2750356-9aba-46de-8c98-1cd1333e9569"
Jul 17 17:21:29.213839 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:29.213809    2561 scope.go:117] "RemoveContainer" containerID="6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e"
Jul 17 17:21:29.214263 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:29.214013    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5_odh-ai-gateway-infra(c2750356-9aba-46de-8c98-1cd1333e9569)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" podUID="c2750356-9aba-46de-8c98-1cd1333e9569"
Jul 17 17:21:30.936272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:30.936237    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:30.941087 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:30.941068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:30.946873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:30.946844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:31.096622 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.096583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx\" (UID: \"d7770455-66b4-476f-906d-5c5d1da513d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:31.197720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.197627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx\" (UID: \"d7770455-66b4-476f-906d-5c5d1da513d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:31.198298 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.198277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx\" (UID: \"d7770455-66b4-476f-906d-5c5d1da513d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:31.253023 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.252982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:31.387114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.387090    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:31.389223 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:21:31.389195    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7770455_66b4_476f_906d_5c5d1da513d6.slice/crio-5ee45d8dba52b99ea6e96b4ba8b38f3fc4999452a6ad26eab75ad1c0fd3276f9 WatchSource:0}: Error finding container 5ee45d8dba52b99ea6e96b4ba8b38f3fc4999452a6ad26eab75ad1c0fd3276f9: Status 404 returned error can't find the container with id 5ee45d8dba52b99ea6e96b4ba8b38f3fc4999452a6ad26eab75ad1c0fd3276f9
Jul 17 17:21:31.508572 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.508489    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:31.508728 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:31.508671    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:31.571224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:31.571190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:21:32.229520 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.229483    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7770455-66b4-476f-906d-5c5d1da513d6" containerID="5b79e4086fa8cf0af00a1527f5eb0a3433482e5d23590f969f904541eca328ba" exitCode=7
Jul 17 17:21:32.229898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.229527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" event={"ID":"d7770455-66b4-476f-906d-5c5d1da513d6","Type":"ContainerDied","Data":"5b79e4086fa8cf0af00a1527f5eb0a3433482e5d23590f969f904541eca328ba"}
Jul 17 17:21:32.229898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.229575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" event={"ID":"d7770455-66b4-476f-906d-5c5d1da513d6","Type":"ContainerStarted","Data":"5ee45d8dba52b99ea6e96b4ba8b38f3fc4999452a6ad26eab75ad1c0fd3276f9"}
Jul 17 17:21:32.229898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.229752    2561 scope.go:117] "RemoveContainer" containerID="5b79e4086fa8cf0af00a1527f5eb0a3433482e5d23590f969f904541eca328ba"
Jul 17 17:21:32.508885 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.508841    2561 scope.go:117] "RemoveContainer" containerID="3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1"
Jul 17 17:21:32.773856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:32.773817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:21:33.236880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.236840    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0b744bc-b450-454c-b852-31e5a61c782e" containerID="a41d2ec803604cea4068f76f0e8bd4287add2dae5505c6764a99d0f93c0fcb07" exitCode=7
Jul 17 17:21:33.237389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.236916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerDied","Data":"a41d2ec803604cea4068f76f0e8bd4287add2dae5505c6764a99d0f93c0fcb07"}
Jul 17 17:21:33.237389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.236977    2561 scope.go:117] "RemoveContainer" containerID="3cf110e741a13e64debf650e2cb721c04787cf5babd4dcf5e9bbe247dfeb89c1"
Jul 17 17:21:33.237389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.237312    2561 scope.go:117] "RemoveContainer" containerID="a41d2ec803604cea4068f76f0e8bd4287add2dae5505c6764a99d0f93c0fcb07"
Jul 17 17:21:33.237584 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:33.237555    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-qndwd_odh-ai-gateway-infra(c0b744bc-b450-454c-b852-31e5a61c782e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" podUID="c0b744bc-b450-454c-b852-31e5a61c782e"
Jul 17 17:21:33.238809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.238788    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7770455-66b4-476f-906d-5c5d1da513d6" containerID="74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139" exitCode=7
Jul 17 17:21:33.238909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.238870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" event={"ID":"d7770455-66b4-476f-906d-5c5d1da513d6","Type":"ContainerDied","Data":"74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139"}
Jul 17 17:21:33.239028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.239013    2561 scope.go:117] "RemoveContainer" containerID="74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139"
Jul 17 17:21:33.239233 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:33.239214    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx_odh-ai-gateway-infra(d7770455-66b4-476f-906d-5c5d1da513d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" podUID="d7770455-66b4-476f-906d-5c5d1da513d6"
Jul 17 17:21:33.272384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.272344    2561 scope.go:117] "RemoveContainer" containerID="5b79e4086fa8cf0af00a1527f5eb0a3433482e5d23590f969f904541eca328ba"
Jul 17 17:21:33.468350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.468268    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:33.472966 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.472945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:33.492840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.492807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:33.622726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.622686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp\" (UID: \"600008d1-41a0-4d8e-a258-c70d907a0aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:33.724064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.723948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp\" (UID: \"600008d1-41a0-4d8e-a258-c70d907a0aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:33.724625 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.724603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp\" (UID: \"600008d1-41a0-4d8e-a258-c70d907a0aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:33.782931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.782893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:33.926465 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:33.926418    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:33.929405 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:21:33.929376    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600008d1_41a0_4d8e_a258_c70d907a0aa3.slice/crio-496b2dde74b3d4e3272212e35a755061a8350b36e9fe55744cfb400fd07b4f0e WatchSource:0}: Error finding container 496b2dde74b3d4e3272212e35a755061a8350b36e9fe55744cfb400fd07b4f0e: Status 404 returned error can't find the container with id 496b2dde74b3d4e3272212e35a755061a8350b36e9fe55744cfb400fd07b4f0e
Jul 17 17:21:34.246705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.246678    2561 scope.go:117] "RemoveContainer" containerID="74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139"
Jul 17 17:21:34.247173 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:34.246924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx_odh-ai-gateway-infra(d7770455-66b4-476f-906d-5c5d1da513d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" podUID="d7770455-66b4-476f-906d-5c5d1da513d6"
Jul 17 17:21:34.248008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.247983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerStarted","Data":"6bb663f8aefe471774bb4930df5dcd24aba007f7992c3cf8a2e15de05d718d8b"}
Jul 17 17:21:34.248163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.248066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerStarted","Data":"496b2dde74b3d4e3272212e35a755061a8350b36e9fe55744cfb400fd07b4f0e"}
Jul 17 17:21:34.264790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.264759    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:34.312652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.312597    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" podStartSLOduration=1.312580896 podStartE2EDuration="1.312580896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:21:34.310577063 +0000 UTC m=+3328.295664603" watchObservedRunningTime="2026-07-17 17:21:34.312580896 +0000 UTC m=+3328.297668435"
Jul 17 17:21:34.396003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.395979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:34.532691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.532663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca\") pod \"c0b744bc-b450-454c-b852-31e5a61c782e\" (UID: \"c0b744bc-b450-454c-b852-31e5a61c782e\") "
Jul 17 17:21:34.533013 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.532989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0b744bc-b450-454c-b852-31e5a61c782e" (UID: "c0b744bc-b450-454c-b852-31e5a61c782e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:21:34.633973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:34.633943    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0b744bc-b450-454c-b852-31e5a61c782e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:21:35.254693 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.254602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd" event={"ID":"c0b744bc-b450-454c-b852-31e5a61c782e","Type":"ContainerDied","Data":"b86a121d32c4947780c721fb66cc185b041aade86880ce8744b634adf3638feb"}
Jul 17 17:21:35.255166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.254612    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"
Jul 17 17:21:35.255166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.254715    2561 scope.go:117] "RemoveContainer" containerID="a41d2ec803604cea4068f76f0e8bd4287add2dae5505c6764a99d0f93c0fcb07"
Jul 17 17:21:35.256379 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.256357    2561 generic.go:356] "Generic (PLEG): container finished" podID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerID="6bb663f8aefe471774bb4930df5dcd24aba007f7992c3cf8a2e15de05d718d8b" exitCode=7
Jul 17 17:21:35.256498 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.256441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerDied","Data":"6bb663f8aefe471774bb4930df5dcd24aba007f7992c3cf8a2e15de05d718d8b"}
Jul 17 17:21:35.256644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.256611    2561 scope.go:117] "RemoveContainer" containerID="6bb663f8aefe471774bb4930df5dcd24aba007f7992c3cf8a2e15de05d718d8b"
Jul 17 17:21:35.288543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.288507    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:35.295443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:35.295418    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-qndwd"]
Jul 17 17:21:36.264221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:36.264122    2561 generic.go:356] "Generic (PLEG): container finished" podID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerID="877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128" exitCode=7
Jul 17 17:21:36.264221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:36.264163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerDied","Data":"877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128"}
Jul 17 17:21:36.264683 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:36.264222    2561 scope.go:117] "RemoveContainer" containerID="6bb663f8aefe471774bb4930df5dcd24aba007f7992c3cf8a2e15de05d718d8b"
Jul 17 17:21:36.264683 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:36.264395    2561 scope.go:117] "RemoveContainer" containerID="877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128"
Jul 17 17:21:36.264683 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:36.264636    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp_odh-ai-gateway-infra(600008d1-41a0-4d8e-a258-c70d907a0aa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3"
Jul 17 17:21:36.514327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:36.514251    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" path="/var/lib/kubelet/pods/c0b744bc-b450-454c-b852-31e5a61c782e/volumes"
Jul 17 17:21:37.270438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:37.270408    2561 scope.go:117] "RemoveContainer" containerID="877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128"
Jul 17 17:21:37.270827 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:37.270600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp_odh-ai-gateway-infra(600008d1-41a0-4d8e-a258-c70d907a0aa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3"
Jul 17 17:21:38.508215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:38.508183    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:21:38.508735 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:38.508369    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:21:41.472836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:41.472800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:21:42.508488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:42.508454    2561 scope.go:117] "RemoveContainer" containerID="6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e"
Jul 17 17:21:43.300788 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:43.300753    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2750356-9aba-46de-8c98-1cd1333e9569" containerID="e7ea4e1b87af32d6dcc98dafc4d8fe537d37c16be72933d1ef3de35008a8b976" exitCode=7
Jul 17 17:21:43.300990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:43.300827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" event={"ID":"c2750356-9aba-46de-8c98-1cd1333e9569","Type":"ContainerDied","Data":"e7ea4e1b87af32d6dcc98dafc4d8fe537d37c16be72933d1ef3de35008a8b976"}
Jul 17 17:21:43.300990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:43.300898    2561 scope.go:117] "RemoveContainer" containerID="6d985860748cbb2b1b637c5ebaceef0b43ff8a32f11ebe8878121fd556e22a8e"
Jul 17 17:21:43.301321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:43.301305    2561 scope.go:117] "RemoveContainer" containerID="e7ea4e1b87af32d6dcc98dafc4d8fe537d37c16be72933d1ef3de35008a8b976"
Jul 17 17:21:43.301545 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:43.301525    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5_odh-ai-gateway-infra(c2750356-9aba-46de-8c98-1cd1333e9569)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" podUID="c2750356-9aba-46de-8c98-1cd1333e9569"
Jul 17 17:21:44.326174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:44.326137    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:44.458717 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:44.458693    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:44.518889 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:44.518863    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca\") pod \"c2750356-9aba-46de-8c98-1cd1333e9569\" (UID: \"c2750356-9aba-46de-8c98-1cd1333e9569\") "
Jul 17 17:21:44.519237 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:44.519213    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2750356-9aba-46de-8c98-1cd1333e9569" (UID: "c2750356-9aba-46de-8c98-1cd1333e9569"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:21:44.620376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:44.620283    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2750356-9aba-46de-8c98-1cd1333e9569-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:21:45.312818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.312788    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"
Jul 17 17:21:45.312818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.312801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5" event={"ID":"c2750356-9aba-46de-8c98-1cd1333e9569","Type":"ContainerDied","Data":"c6a45ad15f129ffbb45c678fa556c59adbf74d7c1b69576c7e75309c6d8b6ba6"}
Jul 17 17:21:45.313113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.312859    2561 scope.go:117] "RemoveContainer" containerID="e7ea4e1b87af32d6dcc98dafc4d8fe537d37c16be72933d1ef3de35008a8b976"
Jul 17 17:21:45.339220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.339189    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:45.345643 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.345620    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rt6p5"]
Jul 17 17:21:45.509098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:45.509066    2561 scope.go:117] "RemoveContainer" containerID="74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139"
Jul 17 17:21:46.319302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.319269    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7770455-66b4-476f-906d-5c5d1da513d6" containerID="363aa273726779ec34984ec78e475b682ea2f0ee541fc9131ca0168f9e0ff8ff" exitCode=7
Jul 17 17:21:46.319493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.319346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" event={"ID":"d7770455-66b4-476f-906d-5c5d1da513d6","Type":"ContainerDied","Data":"363aa273726779ec34984ec78e475b682ea2f0ee541fc9131ca0168f9e0ff8ff"}
Jul 17 17:21:46.319493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.319431    2561 scope.go:117] "RemoveContainer" containerID="74374be3ca12679b458ab0e2993aa97f34190bffd2359a3196b4ece83601f139"
Jul 17 17:21:46.319773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.319746    2561 scope.go:117] "RemoveContainer" containerID="363aa273726779ec34984ec78e475b682ea2f0ee541fc9131ca0168f9e0ff8ff"
Jul 17 17:21:46.320054 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:46.320019    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx_odh-ai-gateway-infra(d7770455-66b4-476f-906d-5c5d1da513d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" podUID="d7770455-66b4-476f-906d-5c5d1da513d6"
Jul 17 17:21:46.510789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.510709    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:21:46.511213 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:46.510887    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:21:46.514373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:46.514344    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" path="/var/lib/kubelet/pods/c2750356-9aba-46de-8c98-1cd1333e9569/volumes"
Jul 17 17:21:47.348895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:47.348857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:47.486493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:47.486468    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:47.546145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:47.546109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca\") pod \"d7770455-66b4-476f-906d-5c5d1da513d6\" (UID: \"d7770455-66b4-476f-906d-5c5d1da513d6\") "
Jul 17 17:21:47.546565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:47.546446    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7770455-66b4-476f-906d-5c5d1da513d6" (UID: "d7770455-66b4-476f-906d-5c5d1da513d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:21:47.546645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:47.546629    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7770455-66b4-476f-906d-5c5d1da513d6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:21:48.332596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.332562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"
Jul 17 17:21:48.332785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.332594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx" event={"ID":"d7770455-66b4-476f-906d-5c5d1da513d6","Type":"ContainerDied","Data":"5ee45d8dba52b99ea6e96b4ba8b38f3fc4999452a6ad26eab75ad1c0fd3276f9"}
Jul 17 17:21:48.332785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.332658    2561 scope.go:117] "RemoveContainer" containerID="363aa273726779ec34984ec78e475b682ea2f0ee541fc9131ca0168f9e0ff8ff"
Jul 17 17:21:48.355569 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.355534    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:48.362201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.362172    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-ksgkx"]
Jul 17 17:21:48.508869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.508837    2561 scope.go:117] "RemoveContainer" containerID="877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128"
Jul 17 17:21:48.514156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:48.514131    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" path="/var/lib/kubelet/pods/d7770455-66b4-476f-906d-5c5d1da513d6/volumes"
Jul 17 17:21:49.344715 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:49.344679    2561 generic.go:356] "Generic (PLEG): container finished" podID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerID="f78c9897ebb3dd70778f0fe0fa7c71b2eb50fdbc6966c370d0e343ba81f26ceb" exitCode=7
Jul 17 17:21:49.345196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:49.344754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerDied","Data":"f78c9897ebb3dd70778f0fe0fa7c71b2eb50fdbc6966c370d0e343ba81f26ceb"}
Jul 17 17:21:49.345196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:49.344838    2561 scope.go:117] "RemoveContainer" containerID="877935917d7d8d636e2fc1896de90695b9d6986af837828cc37cd0d51488a128"
Jul 17 17:21:49.345196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:49.345124    2561 scope.go:117] "RemoveContainer" containerID="f78c9897ebb3dd70778f0fe0fa7c71b2eb50fdbc6966c370d0e343ba81f26ceb"
Jul 17 17:21:49.345388 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:49.345317    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp_odh-ai-gateway-infra(600008d1-41a0-4d8e-a258-c70d907a0aa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3"
Jul 17 17:21:50.367858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.367818    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:50.505069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.505031    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:50.508408 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.508385    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:21:50.508580 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:21:50.508561    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:21:50.573822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.573789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca\") pod \"600008d1-41a0-4d8e-a258-c70d907a0aa3\" (UID: \"600008d1-41a0-4d8e-a258-c70d907a0aa3\") "
Jul 17 17:21:50.574251 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.574218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "600008d1-41a0-4d8e-a258-c70d907a0aa3" (UID: "600008d1-41a0-4d8e-a258-c70d907a0aa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:21:50.675610 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:50.675529    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600008d1-41a0-4d8e-a258-c70d907a0aa3-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:21:51.356568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:51.356536    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"
Jul 17 17:21:51.356568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:51.356545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp" event={"ID":"600008d1-41a0-4d8e-a258-c70d907a0aa3","Type":"ContainerDied","Data":"496b2dde74b3d4e3272212e35a755061a8350b36e9fe55744cfb400fd07b4f0e"}
Jul 17 17:21:51.356821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:51.356612    2561 scope.go:117] "RemoveContainer" containerID="f78c9897ebb3dd70778f0fe0fa7c71b2eb50fdbc6966c370d0e343ba81f26ceb"
Jul 17 17:21:51.379686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:51.379659    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:51.385808 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:51.385781    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vwrkp"]
Jul 17 17:21:52.514092 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:21:52.514057    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" path="/var/lib/kubelet/pods/600008d1-41a0-4d8e-a258-c70d907a0aa3/volumes"
Jul 17 17:22:01.508927 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:01.508894    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:22:01.509342 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:01.509113    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:22:04.517125 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:04.517094    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:22:04.517590 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:04.517275    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:22:05.542393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542359    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542703    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542717    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542728    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542741    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542747    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542757    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542763    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.542769 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542770    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542775    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542781    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542786    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542800    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542807    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542812    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542817    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542822    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542876    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542883    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542889    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542894    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542898    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542902    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542907    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542912    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542917    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:05.543077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.542922    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:05.547182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.547165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:05.549395 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.549374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:22:05.552888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.552854    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:05.609232 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.609198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-zc76c\" (UID: \"800d34bf-dddd-4ec1-8752-912f9119af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:05.710648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.710611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-zc76c\" (UID: \"800d34bf-dddd-4ec1-8752-912f9119af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:05.711239 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.711221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-zc76c\" (UID: \"800d34bf-dddd-4ec1-8752-912f9119af97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:05.858564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.858478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:05.989951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:05.989925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:05.993318 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:22:05.993290    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800d34bf_dddd_4ec1_8752_912f9119af97.slice/crio-ee3cda76c9d07c95d80fed53781fe1872ef4fd1b1f73561ed98f0f068e99eca9 WatchSource:0}: Error finding container ee3cda76c9d07c95d80fed53781fe1872ef4fd1b1f73561ed98f0f068e99eca9: Status 404 returned error can't find the container with id ee3cda76c9d07c95d80fed53781fe1872ef4fd1b1f73561ed98f0f068e99eca9
Jul 17 17:22:06.428844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:06.428809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerStarted","Data":"176822c2ff26cd9a69714fc2d200b8a3a12d07cf33b004e450f482c332af71fe"}
Jul 17 17:22:06.429016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:06.428859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerStarted","Data":"ee3cda76c9d07c95d80fed53781fe1872ef4fd1b1f73561ed98f0f068e99eca9"}
Jul 17 17:22:06.444703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:06.444662    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" podStartSLOduration=1.444648454 podStartE2EDuration="1.444648454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:22:06.443511849 +0000 UTC m=+3360.428599386" watchObservedRunningTime="2026-07-17 17:22:06.444648454 +0000 UTC m=+3360.429735992"
Jul 17 17:22:07.440783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:07.440747    2561 generic.go:356] "Generic (PLEG): container finished" podID="800d34bf-dddd-4ec1-8752-912f9119af97" containerID="176822c2ff26cd9a69714fc2d200b8a3a12d07cf33b004e450f482c332af71fe" exitCode=7
Jul 17 17:22:07.441267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:07.440825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerDied","Data":"176822c2ff26cd9a69714fc2d200b8a3a12d07cf33b004e450f482c332af71fe"}
Jul 17 17:22:07.441267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:07.441020    2561 scope.go:117] "RemoveContainer" containerID="176822c2ff26cd9a69714fc2d200b8a3a12d07cf33b004e450f482c332af71fe"
Jul 17 17:22:08.447564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:08.447526    2561 generic.go:356] "Generic (PLEG): container finished" podID="800d34bf-dddd-4ec1-8752-912f9119af97" containerID="613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1" exitCode=7
Jul 17 17:22:08.447564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:08.447566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerDied","Data":"613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1"}
Jul 17 17:22:08.448072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:08.447612    2561 scope.go:117] "RemoveContainer" containerID="176822c2ff26cd9a69714fc2d200b8a3a12d07cf33b004e450f482c332af71fe"
Jul 17 17:22:08.448072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:08.447825    2561 scope.go:117] "RemoveContainer" containerID="613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1"
Jul 17 17:22:08.448182 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:08.448086    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-zc76c_odh-ai-gateway-infra(800d34bf-dddd-4ec1-8752-912f9119af97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" podUID="800d34bf-dddd-4ec1-8752-912f9119af97"
Jul 17 17:22:09.453564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:09.453536    2561 scope.go:117] "RemoveContainer" containerID="613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1"
Jul 17 17:22:09.453980 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:09.453723    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-zc76c_odh-ai-gateway-infra(800d34bf-dddd-4ec1-8752-912f9119af97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" podUID="800d34bf-dddd-4ec1-8752-912f9119af97"
Jul 17 17:22:12.508346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:12.508319    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:22:12.508730 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:12.508491    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:22:15.697753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.697719    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698100    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698111    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698125    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698131    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2750356-9aba-46de-8c98-1cd1333e9569" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698141    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698147    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="600008d1-41a0-4d8e-a258-c70d907a0aa3" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698206    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b744bc-b450-454c-b852-31e5a61c782e" containerName="revoke-keys"
Jul 17 17:22:15.698269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.698215    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7770455-66b4-476f-906d-5c5d1da513d6" containerName="revoke-keys"
Jul 17 17:22:15.702196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.702177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:15.708495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.708424    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:15.797056 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.797018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2\" (UID: \"3ff63668-eb15-4d41-a8f1-6a05b171e524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:15.898672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.898638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2\" (UID: \"3ff63668-eb15-4d41-a8f1-6a05b171e524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:15.899330 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:15.899309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2\" (UID: \"3ff63668-eb15-4d41-a8f1-6a05b171e524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:16.012704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.012631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:16.147522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.147500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:16.151415 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:22:16.151385    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff63668_eb15_4d41_a8f1_6a05b171e524.slice/crio-ff4713c512e2219573bcd61f75009a2f53872c6ca138e687a96c8936e954419c WatchSource:0}: Error finding container ff4713c512e2219573bcd61f75009a2f53872c6ca138e687a96c8936e954419c: Status 404 returned error can't find the container with id ff4713c512e2219573bcd61f75009a2f53872c6ca138e687a96c8936e954419c
Jul 17 17:22:16.489281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.489242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerStarted","Data":"db0cced7d8e01c1e8087cdb6ea2d2c1a257ad11e47d805c265440842d7566fac"}
Jul 17 17:22:16.489467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.489299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerStarted","Data":"ff4713c512e2219573bcd61f75009a2f53872c6ca138e687a96c8936e954419c"}
Jul 17 17:22:16.507425 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.507375    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" podStartSLOduration=1.507357384 podStartE2EDuration="1.507357384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:22:16.505411072 +0000 UTC m=+3370.490498610" watchObservedRunningTime="2026-07-17 17:22:16.507357384 +0000 UTC m=+3370.492444922"
Jul 17 17:22:16.514094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:16.514074    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:22:16.514278 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:16.514259    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:22:17.496069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:17.495966    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerID="db0cced7d8e01c1e8087cdb6ea2d2c1a257ad11e47d805c265440842d7566fac" exitCode=7
Jul 17 17:22:17.496069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:17.496011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerDied","Data":"db0cced7d8e01c1e8087cdb6ea2d2c1a257ad11e47d805c265440842d7566fac"}
Jul 17 17:22:17.496484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:17.496291    2561 scope.go:117] "RemoveContainer" containerID="db0cced7d8e01c1e8087cdb6ea2d2c1a257ad11e47d805c265440842d7566fac"
Jul 17 17:22:18.502392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:18.502306    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerID="5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb" exitCode=7
Jul 17 17:22:18.502798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:18.502392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerDied","Data":"5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb"}
Jul 17 17:22:18.502798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:18.502454    2561 scope.go:117] "RemoveContainer" containerID="db0cced7d8e01c1e8087cdb6ea2d2c1a257ad11e47d805c265440842d7566fac"
Jul 17 17:22:18.502798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:18.502574    2561 scope.go:117] "RemoveContainer" containerID="5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb"
Jul 17 17:22:18.502798 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:18.502783    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2_odh-ai-gateway-infra(3ff63668-eb15-4d41-a8f1-6a05b171e524)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524"
Jul 17 17:22:19.509149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:19.509120    2561 scope.go:117] "RemoveContainer" containerID="5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb"
Jul 17 17:22:19.509580 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:19.509330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2_odh-ai-gateway-infra(3ff63668-eb15-4d41-a8f1-6a05b171e524)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524"
Jul 17 17:22:20.979626 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:20.979589    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:20.984491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:20.984472    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:20.989387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:20.989362    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:21.155770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.155727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg\" (UID: \"0f76f5b2-0607-4f84-bdae-8c66f847c1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:21.256403 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.256312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg\" (UID: \"0f76f5b2-0607-4f84-bdae-8c66f847c1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:21.256961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.256940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg\" (UID: \"0f76f5b2-0607-4f84-bdae-8c66f847c1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:21.296191 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.296154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:21.430829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.430805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:21.432781 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:22:21.432755    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f76f5b2_0607_4f84_bdae_8c66f847c1d2.slice/crio-f3a2294815cccaa41d3c47094445b2fdffd96d97fbc7ac722afbf7719bbf7321 WatchSource:0}: Error finding container f3a2294815cccaa41d3c47094445b2fdffd96d97fbc7ac722afbf7719bbf7321: Status 404 returned error can't find the container with id f3a2294815cccaa41d3c47094445b2fdffd96d97fbc7ac722afbf7719bbf7321
Jul 17 17:22:21.519710 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:21.519636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" event={"ID":"0f76f5b2-0607-4f84-bdae-8c66f847c1d2","Type":"ContainerStarted","Data":"f3a2294815cccaa41d3c47094445b2fdffd96d97fbc7ac722afbf7719bbf7321"}
Jul 17 17:22:22.509165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:22.509126    2561 scope.go:117] "RemoveContainer" containerID="613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1"
Jul 17 17:22:22.532129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:22.532098    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerID="e7672f22a402898fef373e0cc80292d4e37c7ed8c49fb7028c2b89471d7d8c31" exitCode=7
Jul 17 17:22:22.532357 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:22.532162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" event={"ID":"0f76f5b2-0607-4f84-bdae-8c66f847c1d2","Type":"ContainerDied","Data":"e7672f22a402898fef373e0cc80292d4e37c7ed8c49fb7028c2b89471d7d8c31"}
Jul 17 17:22:22.532434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:22.532417    2561 scope.go:117] "RemoveContainer" containerID="e7672f22a402898fef373e0cc80292d4e37c7ed8c49fb7028c2b89471d7d8c31"
Jul 17 17:22:23.488640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.488601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:23.492053 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.492028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:23.499546 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.499284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:23.539034 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.539002    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerID="24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57" exitCode=7
Jul 17 17:22:23.539497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.539090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" event={"ID":"0f76f5b2-0607-4f84-bdae-8c66f847c1d2","Type":"ContainerDied","Data":"24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57"}
Jul 17 17:22:23.539497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.539157    2561 scope.go:117] "RemoveContainer" containerID="e7672f22a402898fef373e0cc80292d4e37c7ed8c49fb7028c2b89471d7d8c31"
Jul 17 17:22:23.539497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.539392    2561 scope.go:117] "RemoveContainer" containerID="24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57"
Jul 17 17:22:23.539654 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:23.539630    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg_odh-ai-gateway-infra(0f76f5b2-0607-4f84-bdae-8c66f847c1d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2"
Jul 17 17:22:23.541026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.541005    2561 generic.go:356] "Generic (PLEG): container finished" podID="800d34bf-dddd-4ec1-8752-912f9119af97" containerID="88584b9b25c92e4035c4db0d4795b20fbc2bff79281647b36111de3be76d0675" exitCode=7
Jul 17 17:22:23.541135 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.541070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerDied","Data":"88584b9b25c92e4035c4db0d4795b20fbc2bff79281647b36111de3be76d0675"}
Jul 17 17:22:23.541362 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.541347    2561 scope.go:117] "RemoveContainer" containerID="88584b9b25c92e4035c4db0d4795b20fbc2bff79281647b36111de3be76d0675"
Jul 17 17:22:23.541539 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:23.541523    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-zc76c_odh-ai-gateway-infra(800d34bf-dddd-4ec1-8752-912f9119af97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" podUID="800d34bf-dddd-4ec1-8752-912f9119af97"
Jul 17 17:22:23.550014 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.549996    2561 scope.go:117] "RemoveContainer" containerID="613a6f0b8f4fcff41eced8bd47341451d6ba00b75cc6ace083d87010588234d1"
Jul 17 17:22:23.680108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.680000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf\" (UID: \"c88f93be-d9cf-42f7-bdbe-eff3bc024efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:23.781488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.781441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf\" (UID: \"c88f93be-d9cf-42f7-bdbe-eff3bc024efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:23.782115 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.782095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf\" (UID: \"c88f93be-d9cf-42f7-bdbe-eff3bc024efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:23.802797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.802766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:23.934063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:23.934024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:23.936118 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:22:23.936085    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88f93be_d9cf_42f7_bdbe_eff3bc024efe.slice/crio-ffaa56827292118ab6dcb895c6e5e3ef3443781bf13e835c7789dfec1a6b3cf4 WatchSource:0}: Error finding container ffaa56827292118ab6dcb895c6e5e3ef3443781bf13e835c7789dfec1a6b3cf4: Status 404 returned error can't find the container with id ffaa56827292118ab6dcb895c6e5e3ef3443781bf13e835c7789dfec1a6b3cf4
Jul 17 17:22:24.548877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.548834    2561 generic.go:356] "Generic (PLEG): container finished" podID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerID="64bf7a3c7780a21619bfa8bcacce600ff9087f8340609ac81eea6271b3958a96" exitCode=7
Jul 17 17:22:24.549272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.548916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" event={"ID":"c88f93be-d9cf-42f7-bdbe-eff3bc024efe","Type":"ContainerDied","Data":"64bf7a3c7780a21619bfa8bcacce600ff9087f8340609ac81eea6271b3958a96"}
Jul 17 17:22:24.549272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.548966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" event={"ID":"c88f93be-d9cf-42f7-bdbe-eff3bc024efe","Type":"ContainerStarted","Data":"ffaa56827292118ab6dcb895c6e5e3ef3443781bf13e835c7789dfec1a6b3cf4"}
Jul 17 17:22:24.549272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.549160    2561 scope.go:117] "RemoveContainer" containerID="64bf7a3c7780a21619bfa8bcacce600ff9087f8340609ac81eea6271b3958a96"
Jul 17 17:22:24.550709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.550693    2561 scope.go:117] "RemoveContainer" containerID="24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57"
Jul 17 17:22:24.550859 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:24.550843    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg_odh-ai-gateway-infra(0f76f5b2-0607-4f84-bdae-8c66f847c1d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2"
Jul 17 17:22:24.577346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.577323    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:24.699331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.699301    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:24.892154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.892112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca\") pod \"800d34bf-dddd-4ec1-8752-912f9119af97\" (UID: \"800d34bf-dddd-4ec1-8752-912f9119af97\") "
Jul 17 17:22:24.892467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.892442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "800d34bf-dddd-4ec1-8752-912f9119af97" (UID: "800d34bf-dddd-4ec1-8752-912f9119af97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:22:24.993312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:24.993273    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/800d34bf-dddd-4ec1-8752-912f9119af97-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:22:25.564486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.564460    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"
Jul 17 17:22:25.564977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.564457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c" event={"ID":"800d34bf-dddd-4ec1-8752-912f9119af97","Type":"ContainerDied","Data":"ee3cda76c9d07c95d80fed53781fe1872ef4fd1b1f73561ed98f0f068e99eca9"}
Jul 17 17:22:25.564977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.564613    2561 scope.go:117] "RemoveContainer" containerID="88584b9b25c92e4035c4db0d4795b20fbc2bff79281647b36111de3be76d0675"
Jul 17 17:22:25.566425 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.566401    2561 generic.go:356] "Generic (PLEG): container finished" podID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerID="faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107" exitCode=7
Jul 17 17:22:25.566521 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.566434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" event={"ID":"c88f93be-d9cf-42f7-bdbe-eff3bc024efe","Type":"ContainerDied","Data":"faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107"}
Jul 17 17:22:25.566718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.566704    2561 scope.go:117] "RemoveContainer" containerID="faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107"
Jul 17 17:22:25.566884 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:25.566866    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf_odh-ai-gateway-infra(c88f93be-d9cf-42f7-bdbe-eff3bc024efe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe"
Jul 17 17:22:25.573640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.573623    2561 scope.go:117] "RemoveContainer" containerID="64bf7a3c7780a21619bfa8bcacce600ff9087f8340609ac81eea6271b3958a96"
Jul 17 17:22:25.598205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.598182    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:25.601703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:25.601680    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-zc76c"]
Jul 17 17:22:26.514117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:26.514086    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" path="/var/lib/kubelet/pods/800d34bf-dddd-4ec1-8752-912f9119af97/volumes"
Jul 17 17:22:26.573881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:26.573855    2561 scope.go:117] "RemoveContainer" containerID="faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107"
Jul 17 17:22:26.574278 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:26.574071    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf_odh-ai-gateway-infra(c88f93be-d9cf-42f7-bdbe-eff3bc024efe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe"
Jul 17 17:22:27.508842 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:27.508808    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:22:27.509033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:27.508919    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:22:27.509103 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:27.509030    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:22:27.509103 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:27.509078    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:22:32.509060 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:32.509026    2561 scope.go:117] "RemoveContainer" containerID="5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb"
Jul 17 17:22:33.614659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:33.614625    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerID="8b638423c5c82ae13da000829c08e5a9e4be0444f144cac8fad77b28825cbf80" exitCode=7
Jul 17 17:22:33.615091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:33.614694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerDied","Data":"8b638423c5c82ae13da000829c08e5a9e4be0444f144cac8fad77b28825cbf80"}
Jul 17 17:22:33.615091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:33.614783    2561 scope.go:117] "RemoveContainer" containerID="5b618d2a084ee857259d4548ceb730eb620a3d419e6205398d48dce6908d20fb"
Jul 17 17:22:33.615091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:33.615089    2561 scope.go:117] "RemoveContainer" containerID="8b638423c5c82ae13da000829c08e5a9e4be0444f144cac8fad77b28825cbf80"
Jul 17 17:22:33.615299 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:33.615281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2_odh-ai-gateway-infra(3ff63668-eb15-4d41-a8f1-6a05b171e524)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524"
Jul 17 17:22:34.639874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:34.639836    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:34.771594 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:34.771568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:34.878529 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:34.878496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca\") pod \"3ff63668-eb15-4d41-a8f1-6a05b171e524\" (UID: \"3ff63668-eb15-4d41-a8f1-6a05b171e524\") "
Jul 17 17:22:34.878857 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:34.878834    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ff63668-eb15-4d41-a8f1-6a05b171e524" (UID: "3ff63668-eb15-4d41-a8f1-6a05b171e524"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:22:34.980104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:34.980023    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff63668-eb15-4d41-a8f1-6a05b171e524-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:22:35.626877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:35.626850    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"
Jul 17 17:22:35.627065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:35.626849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2" event={"ID":"3ff63668-eb15-4d41-a8f1-6a05b171e524","Type":"ContainerDied","Data":"ff4713c512e2219573bcd61f75009a2f53872c6ca138e687a96c8936e954419c"}
Jul 17 17:22:35.627065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:35.626985    2561 scope.go:117] "RemoveContainer" containerID="8b638423c5c82ae13da000829c08e5a9e4be0444f144cac8fad77b28825cbf80"
Jul 17 17:22:35.647322 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:35.647300    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:35.653255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:35.653235    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-rrxg2"]
Jul 17 17:22:36.513739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:36.513704    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" path="/var/lib/kubelet/pods/3ff63668-eb15-4d41-a8f1-6a05b171e524/volumes"
Jul 17 17:22:37.508292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:37.508254    2561 scope.go:117] "RemoveContainer" containerID="24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57"
Jul 17 17:22:38.643716 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:38.643680    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerID="778d0ef172b88b52dc5205a7f183cdd2368cbf9c92e1b7818a491a4f1b4c008e" exitCode=7
Jul 17 17:22:38.644198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:38.643753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" event={"ID":"0f76f5b2-0607-4f84-bdae-8c66f847c1d2","Type":"ContainerDied","Data":"778d0ef172b88b52dc5205a7f183cdd2368cbf9c92e1b7818a491a4f1b4c008e"}
Jul 17 17:22:38.644198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:38.643813    2561 scope.go:117] "RemoveContainer" containerID="24f1d810cb84d3370c8d91b7e1c12ff3f6b553259985854f82b250b9280dec57"
Jul 17 17:22:38.644198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:38.644168    2561 scope.go:117] "RemoveContainer" containerID="778d0ef172b88b52dc5205a7f183cdd2368cbf9c92e1b7818a491a4f1b4c008e"
Jul 17 17:22:38.644427 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:38.644406    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg_odh-ai-gateway-infra(0f76f5b2-0607-4f84-bdae-8c66f847c1d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2"
Jul 17 17:22:39.670367 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:39.670334    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:39.806858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:39.806834    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:39.925399 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:39.925307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca\") pod \"0f76f5b2-0607-4f84-bdae-8c66f847c1d2\" (UID: \"0f76f5b2-0607-4f84-bdae-8c66f847c1d2\") "
Jul 17 17:22:39.925670 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:39.925648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f76f5b2-0607-4f84-bdae-8c66f847c1d2" (UID: "0f76f5b2-0607-4f84-bdae-8c66f847c1d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:22:40.026275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.026240    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f76f5b2-0607-4f84-bdae-8c66f847c1d2-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:22:40.508382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.508344    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:22:40.508561 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.508490    2561 scope.go:117] "RemoveContainer" containerID="faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107"
Jul 17 17:22:40.508647 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:40.508553    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:22:40.655621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.655594    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"
Jul 17 17:22:40.655621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.655602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg" event={"ID":"0f76f5b2-0607-4f84-bdae-8c66f847c1d2","Type":"ContainerDied","Data":"f3a2294815cccaa41d3c47094445b2fdffd96d97fbc7ac722afbf7719bbf7321"}
Jul 17 17:22:40.655832 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.655662    2561 scope.go:117] "RemoveContainer" containerID="778d0ef172b88b52dc5205a7f183cdd2368cbf9c92e1b7818a491a4f1b4c008e"
Jul 17 17:22:40.671338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.671314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:40.675285 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:40.675264    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-fgkvg"]
Jul 17 17:22:41.663195 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:41.663160    2561 generic.go:356] "Generic (PLEG): container finished" podID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerID="1b40762ba8fdfdc24ae47e36893b8d361a6d0c72befdd1039585e8f45caf6c69" exitCode=7
Jul 17 17:22:41.663347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:41.663236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" event={"ID":"c88f93be-d9cf-42f7-bdbe-eff3bc024efe","Type":"ContainerDied","Data":"1b40762ba8fdfdc24ae47e36893b8d361a6d0c72befdd1039585e8f45caf6c69"}
Jul 17 17:22:41.663347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:41.663289    2561 scope.go:117] "RemoveContainer" containerID="faf7460038fb54089e0551d23d9d07909eb530ae115459f69ca63e8381787107"
Jul 17 17:22:41.663579 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:41.663559    2561 scope.go:117] "RemoveContainer" containerID="1b40762ba8fdfdc24ae47e36893b8d361a6d0c72befdd1039585e8f45caf6c69"
Jul 17 17:22:41.663792 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:41.663771    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf_odh-ai-gateway-infra(c88f93be-d9cf-42f7-bdbe-eff3bc024efe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe"
Jul 17 17:22:42.514188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.514154    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" path="/var/lib/kubelet/pods/0f76f5b2-0607-4f84-bdae-8c66f847c1d2/volumes"
Jul 17 17:22:42.514943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.514929    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:22:42.515110 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:42.515093    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:22:42.690839 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.690801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:42.819792 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.819769    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:42.953716 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.953683    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca\") pod \"c88f93be-d9cf-42f7-bdbe-eff3bc024efe\" (UID: \"c88f93be-d9cf-42f7-bdbe-eff3bc024efe\") "
Jul 17 17:22:42.954079 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:42.954056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c88f93be-d9cf-42f7-bdbe-eff3bc024efe" (UID: "c88f93be-d9cf-42f7-bdbe-eff3bc024efe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:22:43.054836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.054752    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88f93be-d9cf-42f7-bdbe-eff3bc024efe-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:22:43.681734 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.681707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"
Jul 17 17:22:43.681734 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.681713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf" event={"ID":"c88f93be-d9cf-42f7-bdbe-eff3bc024efe","Type":"ContainerDied","Data":"ffaa56827292118ab6dcb895c6e5e3ef3443781bf13e835c7789dfec1a6b3cf4"}
Jul 17 17:22:43.682274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.681780    2561 scope.go:117] "RemoveContainer" containerID="1b40762ba8fdfdc24ae47e36893b8d361a6d0c72befdd1039585e8f45caf6c69"
Jul 17 17:22:43.703910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.703884    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:43.710480 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:43.710460    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vqccf"]
Jul 17 17:22:44.514350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:44.514321    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" path="/var/lib/kubelet/pods/c88f93be-d9cf-42f7-bdbe-eff3bc024efe/volumes"
Jul 17 17:22:52.508460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:52.508430    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:22:52.508959 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:52.508623    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:22:54.508512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:22:54.508483    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:22:54.508916 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:22:54.508654    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:05.585364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585276    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:05.585823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585775    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.585823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585795    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.585823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585809    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.585823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585818    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585833    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585842    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585871    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585881    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585889    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585898    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585907    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585920    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585928    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585939    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585948    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585958    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.586039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.585966    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586083    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586097    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586107    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586117    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586126    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586134    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586143    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586151    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.586722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.586161    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.589467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.589447    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:05.591669 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.591646    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:23:05.596105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.596082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:05.760665 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.760630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-695s4\" (UID: \"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:05.782963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.782935    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:05.783336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783317    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.783336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783334    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff63668-eb15-4d41-a8f1-6a05b171e524" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783345    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783351    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783364    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783423    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f76f5b2-0607-4f84-bdae-8c66f847c1d2" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783433    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88f93be-d9cf-42f7-bdbe-eff3bc024efe" containerName="revoke-keys"
Jul 17 17:23:05.783491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.783440    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="800d34bf-dddd-4ec1-8752-912f9119af97" containerName="revoke-keys"
Jul 17 17:23:05.786566 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.786544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:05.793903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.793690    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:05.862214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.862121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-695s4\" (UID: \"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:05.862799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.862772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-695s4\" (UID: \"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:05.901142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.901104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:05.963703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:05.963625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt\" (UID: \"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:06.029545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.029520    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:06.031661 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:23:06.031633    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbaebfb_f97a_48f1_aeb2_9ee6bc20dd17.slice/crio-f10fbab07b50e6320f0922150ab1a02da8322fb5dbb1ada99d193d4930fabd41 WatchSource:0}: Error finding container f10fbab07b50e6320f0922150ab1a02da8322fb5dbb1ada99d193d4930fabd41: Status 404 returned error can't find the container with id f10fbab07b50e6320f0922150ab1a02da8322fb5dbb1ada99d193d4930fabd41
Jul 17 17:23:06.064964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.064941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt\" (UID: \"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:06.065518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.065500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt\" (UID: \"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:06.098225 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.098199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:06.232793 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.232767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:06.315094 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:23:06.315061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8b53f6_e7f5_41bc_a803_dc7ee9ebc335.slice/crio-48caf47094e2a0f47f2ac0bb726c431ea10407fef7d514ecc80d06fc43998cfe WatchSource:0}: Error finding container 48caf47094e2a0f47f2ac0bb726c431ea10407fef7d514ecc80d06fc43998cfe: Status 404 returned error can't find the container with id 48caf47094e2a0f47f2ac0bb726c431ea10407fef7d514ecc80d06fc43998cfe
Jul 17 17:23:06.510597 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.510521    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:23:06.510750 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:06.510715    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:06.794423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.794386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerStarted","Data":"4d506a2271a7aeeb62979fea6c59b2882b3f377ec4ac86905138860d8394a003"}
Jul 17 17:23:06.794917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.794448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerStarted","Data":"48caf47094e2a0f47f2ac0bb726c431ea10407fef7d514ecc80d06fc43998cfe"}
Jul 17 17:23:06.795806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.795783    2561 generic.go:356] "Generic (PLEG): container finished" podID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerID="394400072aab6503808d5a0ef4ce4290c2e6330bba0ce57f926eeac720fa3a25" exitCode=7
Jul 17 17:23:06.795880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.795826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerDied","Data":"394400072aab6503808d5a0ef4ce4290c2e6330bba0ce57f926eeac720fa3a25"}
Jul 17 17:23:06.795880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.795856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerStarted","Data":"f10fbab07b50e6320f0922150ab1a02da8322fb5dbb1ada99d193d4930fabd41"}
Jul 17 17:23:06.796033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.796020    2561 scope.go:117] "RemoveContainer" containerID="394400072aab6503808d5a0ef4ce4290c2e6330bba0ce57f926eeac720fa3a25"
Jul 17 17:23:06.814578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:06.814523    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" podStartSLOduration=1.814501687 podStartE2EDuration="1.814501687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:23:06.812502669 +0000 UTC m=+3420.797590189" watchObservedRunningTime="2026-07-17 17:23:06.814501687 +0000 UTC m=+3420.799589226"
Jul 17 17:23:07.509016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.508931    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:23:07.509209 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:07.509154    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:23:07.802746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.802713    2561 generic.go:356] "Generic (PLEG): container finished" podID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerID="b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf" exitCode=7
Jul 17 17:23:07.803206 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.802804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerDied","Data":"b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf"}
Jul 17 17:23:07.803206 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.802866    2561 scope.go:117] "RemoveContainer" containerID="394400072aab6503808d5a0ef4ce4290c2e6330bba0ce57f926eeac720fa3a25"
Jul 17 17:23:07.803206 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.803002    2561 scope.go:117] "RemoveContainer" containerID="b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf"
Jul 17 17:23:07.803347 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:07.803229    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-695s4_odh-ai-gateway-infra(edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17"
Jul 17 17:23:07.804371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.804344    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerID="4d506a2271a7aeeb62979fea6c59b2882b3f377ec4ac86905138860d8394a003" exitCode=7
Jul 17 17:23:07.804483 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.804383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerDied","Data":"4d506a2271a7aeeb62979fea6c59b2882b3f377ec4ac86905138860d8394a003"}
Jul 17 17:23:07.804577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:07.804564    2561 scope.go:117] "RemoveContainer" containerID="4d506a2271a7aeeb62979fea6c59b2882b3f377ec4ac86905138860d8394a003"
Jul 17 17:23:08.810932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:08.810898    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerID="b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c" exitCode=7
Jul 17 17:23:08.811385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:08.810975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerDied","Data":"b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c"}
Jul 17 17:23:08.811385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:08.811035    2561 scope.go:117] "RemoveContainer" containerID="4d506a2271a7aeeb62979fea6c59b2882b3f377ec4ac86905138860d8394a003"
Jul 17 17:23:08.811385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:08.811267    2561 scope.go:117] "RemoveContainer" containerID="b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c"
Jul 17 17:23:08.811529 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:08.811497    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt_odh-ai-gateway-infra(5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335"
Jul 17 17:23:08.812687 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:08.812673    2561 scope.go:117] "RemoveContainer" containerID="b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf"
Jul 17 17:23:08.812847 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:08.812829    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-695s4_odh-ai-gateway-infra(edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17"
Jul 17 17:23:09.819191 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:09.819165    2561 scope.go:117] "RemoveContainer" containerID="b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c"
Jul 17 17:23:09.819580 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:09.819370    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt_odh-ai-gateway-infra(5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335"
Jul 17 17:23:11.026471 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.026435    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:11.029754 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.029737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:11.038155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.038127    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:11.109699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.109664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7\" (UID: \"f12b6847-28ca-4f38-9006-0b93d3bf748e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:11.210325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.210289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7\" (UID: \"f12b6847-28ca-4f38-9006-0b93d3bf748e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:11.210878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.210860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7\" (UID: \"f12b6847-28ca-4f38-9006-0b93d3bf748e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:11.340342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.340257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:11.469533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.469507    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:11.471685 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:23:11.471661    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12b6847_28ca_4f38_9006_0b93d3bf748e.slice/crio-9e31d1e7676c283808fd90610a6aa2a6dcff7c4d1cfc4185df8a5a73182ba5e8 WatchSource:0}: Error finding container 9e31d1e7676c283808fd90610a6aa2a6dcff7c4d1cfc4185df8a5a73182ba5e8: Status 404 returned error can't find the container with id 9e31d1e7676c283808fd90610a6aa2a6dcff7c4d1cfc4185df8a5a73182ba5e8
Jul 17 17:23:11.829701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.829666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerStarted","Data":"69358ef763e5621ef416825af4f75b509d7aa2bfe5253f8e644efdf331564695"}
Jul 17 17:23:11.829864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.829716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerStarted","Data":"9e31d1e7676c283808fd90610a6aa2a6dcff7c4d1cfc4185df8a5a73182ba5e8"}
Jul 17 17:23:11.844774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:11.844724    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" podStartSLOduration=0.844709768 podStartE2EDuration="844.709768ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:23:11.843564731 +0000 UTC m=+3425.828652290" watchObservedRunningTime="2026-07-17 17:23:11.844709768 +0000 UTC m=+3425.829797306"
Jul 17 17:23:12.835991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:12.835960    2561 generic.go:356] "Generic (PLEG): container finished" podID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerID="69358ef763e5621ef416825af4f75b509d7aa2bfe5253f8e644efdf331564695" exitCode=7
Jul 17 17:23:12.836507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:12.836072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerDied","Data":"69358ef763e5621ef416825af4f75b509d7aa2bfe5253f8e644efdf331564695"}
Jul 17 17:23:12.836507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:12.836299    2561 scope.go:117] "RemoveContainer" containerID="69358ef763e5621ef416825af4f75b509d7aa2bfe5253f8e644efdf331564695"
Jul 17 17:23:13.842664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:13.842632    2561 generic.go:356] "Generic (PLEG): container finished" podID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerID="a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137" exitCode=7
Jul 17 17:23:13.843091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:13.842684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerDied","Data":"a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137"}
Jul 17 17:23:13.843091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:13.842750    2561 scope.go:117] "RemoveContainer" containerID="69358ef763e5621ef416825af4f75b509d7aa2bfe5253f8e644efdf331564695"
Jul 17 17:23:13.843091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:13.842897    2561 scope.go:117] "RemoveContainer" containerID="a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137"
Jul 17 17:23:13.843224 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:13.843142    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7_odh-ai-gateway-infra(f12b6847-28ca-4f38-9006-0b93d3bf748e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e"
Jul 17 17:23:14.848654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:14.848631    2561 scope.go:117] "RemoveContainer" containerID="a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137"
Jul 17 17:23:14.851054 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:14.848827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7_odh-ai-gateway-infra(f12b6847-28ca-4f38-9006-0b93d3bf748e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e"
Jul 17 17:23:17.508649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:17.508622    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:23:17.509084 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:17.508827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:20.508219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:20.508145    2561 scope.go:117] "RemoveContainer" containerID="b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c"
Jul 17 17:23:20.876691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:20.876660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerStarted","Data":"9ccae397c6f0773794d5a1058a9a343ed6c7ac762d5de5c5fdbc1de54f708d9f"}
Jul 17 17:23:21.883828 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:21.883793    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerID="9ccae397c6f0773794d5a1058a9a343ed6c7ac762d5de5c5fdbc1de54f708d9f" exitCode=7
Jul 17 17:23:21.884252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:21.883861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerDied","Data":"9ccae397c6f0773794d5a1058a9a343ed6c7ac762d5de5c5fdbc1de54f708d9f"}
Jul 17 17:23:21.884252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:21.883918    2561 scope.go:117] "RemoveContainer" containerID="b1633e47b6b98424e66637af5d394c1beab4a3c907ea8dd5d7dad6146f41c20c"
Jul 17 17:23:21.884252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:21.884195    2561 scope.go:117] "RemoveContainer" containerID="9ccae397c6f0773794d5a1058a9a343ed6c7ac762d5de5c5fdbc1de54f708d9f"
Jul 17 17:23:21.884439 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:21.884416    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt_odh-ai-gateway-infra(5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335"
Jul 17 17:23:21.907621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:21.907593    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:22.508629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:22.508601    2561 scope.go:117] "RemoveContainer" containerID="b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf"
Jul 17 17:23:22.508783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:22.508693    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:23:22.508883 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:22.508837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:23:22.890611 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:22.890568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerStarted","Data":"ef6494094cd644f97a2e7dac3e83f32b84943e2e0099d0932a73b56a1a9add76"}
Jul 17 17:23:22.907550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:22.907500    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" podStartSLOduration=17.907484815 podStartE2EDuration="17.907484815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:23:22.905719064 +0000 UTC m=+3436.890806603" watchObservedRunningTime="2026-07-17 17:23:22.907484815 +0000 UTC m=+3436.892572353"
Jul 17 17:23:23.023647 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.023624    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:23.107966 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.107879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca\") pod \"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335\" (UID: \"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335\") "
Jul 17 17:23:23.108249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.108229    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" (UID: "5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:23:23.208931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.208899    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:23:23.532459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532424    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:23.532825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.532880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532826    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.532880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532837    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.532880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532843    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.532880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.532880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532869    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.533037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532945    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.533037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.532954    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:23:23.535902 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.535884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:23.544190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.544165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:23.612378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.612348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6\" (UID: \"a5934933-b1c2-4886-b69f-5b9cc85271bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:23.713620 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.713587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6\" (UID: \"a5934933-b1c2-4886-b69f-5b9cc85271bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:23.714266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.714247    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6\" (UID: \"a5934933-b1c2-4886-b69f-5b9cc85271bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:23.846167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.846072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:23.901314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.901251    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"
Jul 17 17:23:23.901314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.901263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt" event={"ID":"5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335","Type":"ContainerDied","Data":"48caf47094e2a0f47f2ac0bb726c431ea10407fef7d514ecc80d06fc43998cfe"}
Jul 17 17:23:23.901842 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.901325    2561 scope.go:117] "RemoveContainer" containerID="9ccae397c6f0773794d5a1058a9a343ed6c7ac762d5de5c5fdbc1de54f708d9f"
Jul 17 17:23:23.906349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.905318    2561 generic.go:356] "Generic (PLEG): container finished" podID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerID="ef6494094cd644f97a2e7dac3e83f32b84943e2e0099d0932a73b56a1a9add76" exitCode=7
Jul 17 17:23:23.906349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.905445    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerDied","Data":"ef6494094cd644f97a2e7dac3e83f32b84943e2e0099d0932a73b56a1a9add76"}
Jul 17 17:23:23.906349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.905794    2561 scope.go:117] "RemoveContainer" containerID="ef6494094cd644f97a2e7dac3e83f32b84943e2e0099d0932a73b56a1a9add76"
Jul 17 17:23:23.906349 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:23.906020    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-695s4_odh-ai-gateway-infra(edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17"
Jul 17 17:23:23.916735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.916522    2561 scope.go:117] "RemoveContainer" containerID="b6561356455be50c8b8a46153dae542756754853985d89cb87f13e178aea2bbf"
Jul 17 17:23:23.919823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.919802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:23.952018 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.951993    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:23.957511 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.957484    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-z4bmt"]
Jul 17 17:23:23.989875 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:23.989846    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:23.993466 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:23:23.993440    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5934933_b1c2_4886_b69f_5b9cc85271bf.slice/crio-d1487580444a32abee58f58e80693dce6ee9611386a272f609ef56a23b9d36b4 WatchSource:0}: Error finding container d1487580444a32abee58f58e80693dce6ee9611386a272f609ef56a23b9d36b4: Status 404 returned error can't find the container with id d1487580444a32abee58f58e80693dce6ee9611386a272f609ef56a23b9d36b4
Jul 17 17:23:24.514374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:24.514302    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" path="/var/lib/kubelet/pods/5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335/volumes"
Jul 17 17:23:24.912323 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:24.912284    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerID="45544fca23eca2ac3e6d98606bf649c70fb002cdf213e60a9bf91500c2b70c46" exitCode=7
Jul 17 17:23:24.912323 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:24.912320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerDied","Data":"45544fca23eca2ac3e6d98606bf649c70fb002cdf213e60a9bf91500c2b70c46"}
Jul 17 17:23:24.912783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:24.912371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerStarted","Data":"d1487580444a32abee58f58e80693dce6ee9611386a272f609ef56a23b9d36b4"}
Jul 17 17:23:24.912783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:24.912568    2561 scope.go:117] "RemoveContainer" containerID="45544fca23eca2ac3e6d98606bf649c70fb002cdf213e60a9bf91500c2b70c46"
Jul 17 17:23:25.134627 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.134605    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:25.228192 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.228106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca\") pod \"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17\" (UID: \"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17\") "
Jul 17 17:23:25.228501 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.228482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" (UID: "edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:23:25.329782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.329735    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:23:25.920723 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.920683    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerID="b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009" exitCode=7
Jul 17 17:23:25.921252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.920774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerDied","Data":"b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009"}
Jul 17 17:23:25.921252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.920833    2561 scope.go:117] "RemoveContainer" containerID="45544fca23eca2ac3e6d98606bf649c70fb002cdf213e60a9bf91500c2b70c46"
Jul 17 17:23:25.921252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.921078    2561 scope.go:117] "RemoveContainer" containerID="b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009"
Jul 17 17:23:25.921420 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:25.921367    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6_odh-ai-gateway-infra(a5934933-b1c2-4886-b69f-5b9cc85271bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf"
Jul 17 17:23:25.922107 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.922088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4" event={"ID":"edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17","Type":"ContainerDied","Data":"f10fbab07b50e6320f0922150ab1a02da8322fb5dbb1ada99d193d4930fabd41"}
Jul 17 17:23:25.922187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.922123    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"
Jul 17 17:23:25.931883 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.931861    2561 scope.go:117] "RemoveContainer" containerID="ef6494094cd644f97a2e7dac3e83f32b84943e2e0099d0932a73b56a1a9add76"
Jul 17 17:23:25.952135 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.952096    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:25.958418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:25.958392    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-695s4"]
Jul 17 17:23:26.513420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:26.513388    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" path="/var/lib/kubelet/pods/edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17/volumes"
Jul 17 17:23:26.928445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:26.928416    2561 scope.go:117] "RemoveContainer" containerID="b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009"
Jul 17 17:23:26.928888 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:26.928654    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6_odh-ai-gateway-infra(a5934933-b1c2-4886-b69f-5b9cc85271bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf"
Jul 17 17:23:29.508743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:29.508715    2561 scope.go:117] "RemoveContainer" containerID="a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137"
Jul 17 17:23:29.945091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:29.945032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerStarted","Data":"6e8992f53bc2da3989ce167523905a474547bcfc4d49645d0d14c13c87ce55c3"}
Jul 17 17:23:30.509302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.509224    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:23:30.509660 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.509631    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:23:30.951571 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.951537    2561 generic.go:356] "Generic (PLEG): container finished" podID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerID="6e8992f53bc2da3989ce167523905a474547bcfc4d49645d0d14c13c87ce55c3" exitCode=7
Jul 17 17:23:30.951744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.951585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerDied","Data":"6e8992f53bc2da3989ce167523905a474547bcfc4d49645d0d14c13c87ce55c3"}
Jul 17 17:23:30.951744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.951642    2561 scope.go:117] "RemoveContainer" containerID="a58c50de54e3f21603a7bb459bd71372b571a1d1ef9aa978245504d216dd4137"
Jul 17 17:23:30.951936 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.951920    2561 scope.go:117] "RemoveContainer" containerID="6e8992f53bc2da3989ce167523905a474547bcfc4d49645d0d14c13c87ce55c3"
Jul 17 17:23:30.952168 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:30.952148    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7_odh-ai-gateway-infra(f12b6847-28ca-4f38-9006-0b93d3bf748e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e"
Jul 17 17:23:30.974942 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:30.974917    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:31.072690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.072656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:31.526741 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.526702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:31.626364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.626330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:31.726873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.726786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:31.957773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.957741    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582" exitCode=1
Jul 17 17:23:31.957937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.957809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"}
Jul 17 17:23:31.957937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.957860    2561 scope.go:117] "RemoveContainer" containerID="4245b650d9994e1550fc88674a969d784edd17bd5115074afc331f08da14c1e7"
Jul 17 17:23:31.958270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:31.958253    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:23:31.958518 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:31.958497    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:32.089726 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.089701    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:32.192583 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.192534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca\") pod \"f12b6847-28ca-4f38-9006-0b93d3bf748e\" (UID: \"f12b6847-28ca-4f38-9006-0b93d3bf748e\") "
Jul 17 17:23:32.192903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.192881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f12b6847-28ca-4f38-9006-0b93d3bf748e" (UID: "f12b6847-28ca-4f38-9006-0b93d3bf748e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:23:32.294054 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.294018    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12b6847-28ca-4f38-9006-0b93d3bf748e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:23:32.967060 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.967008    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"
Jul 17 17:23:32.967496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.967066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7" event={"ID":"f12b6847-28ca-4f38-9006-0b93d3bf748e","Type":"ContainerDied","Data":"9e31d1e7676c283808fd90610a6aa2a6dcff7c4d1cfc4185df8a5a73182ba5e8"}
Jul 17 17:23:32.967496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.967155    2561 scope.go:117] "RemoveContainer" containerID="6e8992f53bc2da3989ce167523905a474547bcfc4d49645d0d14c13c87ce55c3"
Jul 17 17:23:32.986067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.986029    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:32.999160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:32.999134    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6f7d7"]
Jul 17 17:23:33.235377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:33.235289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:34.514358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:34.514327    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" path="/var/lib/kubelet/pods/f12b6847-28ca-4f38-9006-0b93d3bf748e/volumes"
Jul 17 17:23:34.657929 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:34.657898    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:23:35.175146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:35.175112    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:23:35.175493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:35.175477    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:23:35.175679 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:35.175660    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:36.511196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:36.511170    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:23:36.511659 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:36.511388    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:23:37.508576 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:37.508530    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:23:37.508927 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:37.508913    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:23:37.509147 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:37.509127    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:23:38.514506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:38.514479    2561 scope.go:117] "RemoveContainer" containerID="b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009"
Jul 17 17:23:38.997704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:38.997666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerStarted","Data":"eb381783dfd81a3f75a711130feb9a85b986ba580d87c210fcbc15b4ea9599e1"}
Jul 17 17:23:39.016580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.016534    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" podStartSLOduration=16.01652087 podStartE2EDuration="16.01652087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:23:39.014884023 +0000 UTC m=+3452.999971588" watchObservedRunningTime="2026-07-17 17:23:39.01652087 +0000 UTC m=+3453.001608406"
Jul 17 17:23:39.125935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.125905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:39.237785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.237753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:39.330697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.330660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:39.423236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.423141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:39.860835 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:39.860803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:40.004298 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:40.004265    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerID="eb381783dfd81a3f75a711130feb9a85b986ba580d87c210fcbc15b4ea9599e1" exitCode=7
Jul 17 17:23:40.004477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:40.004336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerDied","Data":"eb381783dfd81a3f75a711130feb9a85b986ba580d87c210fcbc15b4ea9599e1"}
Jul 17 17:23:40.004477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:40.004396    2561 scope.go:117] "RemoveContainer" containerID="b356eb46992812494db420b6f50c4a19bf6812f602226e35601a8b5f217b0009"
Jul 17 17:23:40.004681 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:40.004664    2561 scope.go:117] "RemoveContainer" containerID="eb381783dfd81a3f75a711130feb9a85b986ba580d87c210fcbc15b4ea9599e1"
Jul 17 17:23:40.004899 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:40.004871    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6_odh-ai-gateway-infra(a5934933-b1c2-4886-b69f-5b9cc85271bf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf"
Jul 17 17:23:40.033818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:40.033791    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:41.138707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:41.138682    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:41.277659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:41.277634    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca\") pod \"a5934933-b1c2-4886-b69f-5b9cc85271bf\" (UID: \"a5934933-b1c2-4886-b69f-5b9cc85271bf\") "
Jul 17 17:23:41.277997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:41.277975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5934933-b1c2-4886-b69f-5b9cc85271bf" (UID: "a5934933-b1c2-4886-b69f-5b9cc85271bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:23:41.378238 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:41.378201    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5934933-b1c2-4886-b69f-5b9cc85271bf-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:23:42.015876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.015848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6" event={"ID":"a5934933-b1c2-4886-b69f-5b9cc85271bf","Type":"ContainerDied","Data":"d1487580444a32abee58f58e80693dce6ee9611386a272f609ef56a23b9d36b4"}
Jul 17 17:23:42.015876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.015873    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"
Jul 17 17:23:42.016144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.015908    2561 scope.go:117] "RemoveContainer" containerID="eb381783dfd81a3f75a711130feb9a85b986ba580d87c210fcbc15b4ea9599e1"
Jul 17 17:23:42.036201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.036176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:42.040813 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.040792    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-hwnc6"]
Jul 17 17:23:42.513916 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.513881    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" path="/var/lib/kubelet/pods/a5934933-b1c2-4886-b69f-5b9cc85271bf/volumes"
Jul 17 17:23:42.887040 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:42.887011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:23:49.508884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:49.508845    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:23:49.509316 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:49.509082    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:23:51.508298 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:23:51.508265    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:23:51.508679 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:23:51.508434    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:01.508963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:01.508930    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:24:01.509494 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:01.509141    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:24:02.508574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:02.508538    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:24:02.508779 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:02.508718    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:05.628142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628110    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628476    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628484    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628500    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628513    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628519    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628530    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628544    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628550    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628607    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628615    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628621    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628626    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628631    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628638    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628644    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8b53f6-e7f5-41bc-a803-dc7ee9ebc335" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628650    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.628844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.628656    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:24:05.631729 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.631712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:05.633953 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.633932    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:24:05.638169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.638139    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:05.703343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.703298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-fpr4w\" (UID: \"04a1d35c-fe79-469e-b041-2471fd9982b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:05.803858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.803825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-fpr4w\" (UID: \"04a1d35c-fe79-469e-b041-2471fd9982b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:05.804703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.804675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-fpr4w\" (UID: \"04a1d35c-fe79-469e-b041-2471fd9982b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:05.847538 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.847505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:05.943065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:05.942972    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:06.071397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:06.071373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:06.073493 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:24:06.073466    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a1d35c_fe79_469e_b041_2471fd9982b4.slice/crio-7eb021a57ed690ba1c976fbe899e9ca0fc6ab1f8b4638525cb4a899bc54456c9 WatchSource:0}: Error finding container 7eb021a57ed690ba1c976fbe899e9ca0fc6ab1f8b4638525cb4a899bc54456c9: Status 404 returned error can't find the container with id 7eb021a57ed690ba1c976fbe899e9ca0fc6ab1f8b4638525cb4a899bc54456c9
Jul 17 17:24:06.130898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:06.130871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" event={"ID":"04a1d35c-fe79-469e-b041-2471fd9982b4","Type":"ContainerStarted","Data":"7eb021a57ed690ba1c976fbe899e9ca0fc6ab1f8b4638525cb4a899bc54456c9"}
Jul 17 17:24:07.137564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:07.137519    2561 generic.go:356] "Generic (PLEG): container finished" podID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerID="724a00735629e9544fee48655c3fadf86546cead274c5f892a32a3ee65d918ee" exitCode=7
Jul 17 17:24:07.137978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:07.137567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" event={"ID":"04a1d35c-fe79-469e-b041-2471fd9982b4","Type":"ContainerDied","Data":"724a00735629e9544fee48655c3fadf86546cead274c5f892a32a3ee65d918ee"}
Jul 17 17:24:07.137978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:07.137814    2561 scope.go:117] "RemoveContainer" containerID="724a00735629e9544fee48655c3fadf86546cead274c5f892a32a3ee65d918ee"
Jul 17 17:24:08.144598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:08.144564    2561 generic.go:356] "Generic (PLEG): container finished" podID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerID="6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659" exitCode=7
Jul 17 17:24:08.145062 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:08.144649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" event={"ID":"04a1d35c-fe79-469e-b041-2471fd9982b4","Type":"ContainerDied","Data":"6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659"}
Jul 17 17:24:08.145062 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:08.144702    2561 scope.go:117] "RemoveContainer" containerID="724a00735629e9544fee48655c3fadf86546cead274c5f892a32a3ee65d918ee"
Jul 17 17:24:08.145062 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:08.144977    2561 scope.go:117] "RemoveContainer" containerID="6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659"
Jul 17 17:24:08.145223 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:08.145204    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-fpr4w_odh-ai-gateway-infra(04a1d35c-fe79-469e-b041-2471fd9982b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4"
Jul 17 17:24:09.150829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:09.150803    2561 scope.go:117] "RemoveContainer" containerID="6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659"
Jul 17 17:24:09.151216 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:09.151020    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-fpr4w_odh-ai-gateway-infra(04a1d35c-fe79-469e-b041-2471fd9982b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4"
Jul 17 17:24:11.946366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:11.946332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:12.050229 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:12.050197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:12.150383 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:12.150350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:13.508713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:13.508681    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:24:13.509143 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:13.508875    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:24:16.510881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:16.510843    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:24:16.511305 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:16.511077    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:21.508266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:21.508233    2561 scope.go:117] "RemoveContainer" containerID="6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659"
Jul 17 17:24:22.215680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:22.215648    2561 generic.go:356] "Generic (PLEG): container finished" podID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerID="10d3b42d425b8e16f6d87d69159c6651e930489278a53e5daab904ae28656ccc" exitCode=7
Jul 17 17:24:22.215869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:22.215727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" event={"ID":"04a1d35c-fe79-469e-b041-2471fd9982b4","Type":"ContainerDied","Data":"10d3b42d425b8e16f6d87d69159c6651e930489278a53e5daab904ae28656ccc"}
Jul 17 17:24:22.215869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:22.215801    2561 scope.go:117] "RemoveContainer" containerID="6a8b473a95fe13c12f7180d697035c790420ceeedb2b4fc80eba345313acc659"
Jul 17 17:24:22.216219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:22.216200    2561 scope.go:117] "RemoveContainer" containerID="10d3b42d425b8e16f6d87d69159c6651e930489278a53e5daab904ae28656ccc"
Jul 17 17:24:22.216437 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:22.216404    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-fpr4w_odh-ai-gateway-infra(04a1d35c-fe79-469e-b041-2471fd9982b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4"
Jul 17 17:24:23.241031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:23.240996    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:23.371069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:23.371030    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:23.474198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:23.474166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca\") pod \"04a1d35c-fe79-469e-b041-2471fd9982b4\" (UID: \"04a1d35c-fe79-469e-b041-2471fd9982b4\") "
Jul 17 17:24:23.474510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:23.474490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04a1d35c-fe79-469e-b041-2471fd9982b4" (UID: "04a1d35c-fe79-469e-b041-2471fd9982b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:24:23.575124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:23.575095    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a1d35c-fe79-469e-b041-2471fd9982b4-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:24:24.228304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.228274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"
Jul 17 17:24:24.228477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.228304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w" event={"ID":"04a1d35c-fe79-469e-b041-2471fd9982b4","Type":"ContainerDied","Data":"7eb021a57ed690ba1c976fbe899e9ca0fc6ab1f8b4638525cb4a899bc54456c9"}
Jul 17 17:24:24.228477 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.228363    2561 scope.go:117] "RemoveContainer" containerID="10d3b42d425b8e16f6d87d69159c6651e930489278a53e5daab904ae28656ccc"
Jul 17 17:24:24.250375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.250350    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:24.256744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.256723    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-fpr4w"]
Jul 17 17:24:24.513790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:24.513713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" path="/var/lib/kubelet/pods/04a1d35c-fe79-469e-b041-2471fd9982b4/volumes"
Jul 17 17:24:26.510202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:26.510168    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:24:26.510708 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:26.510368    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:24:28.097576 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:28.097529    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:28.508765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:28.508643    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:24:28.508963 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:28.508869    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:33.738800 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:33.738769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:33.847267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:33.847235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:33.964587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:33.964551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:39.508289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:39.508257    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:24:39.508669 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:39.508450    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:40.508997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:40.508962    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:24:40.509415 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:40.509191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:24:44.472764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:44.472728    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:45.258877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:45.258843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:51.552563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:51.552527    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:51.660575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:51.660545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:51.752168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:51.752137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:51.853214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:51.853130    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:52.587683 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:52.587651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:52.688029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:52.687996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:52.787593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:52.787563    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:53.508200 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:53.508169    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:24:53.508382 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:53.508349    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:24:54.431235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:54.431203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:24:54.509022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:54.508995    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:24:54.509216 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:24:54.509190    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:24:56.163558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:24:56.163522    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:25:00.395189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:00.395156    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:00.488321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:00.488274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:01.077148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:01.077113    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:05.508812 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:05.508782    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:25:05.509337 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:05.508973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:25:06.568541 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:06.568508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:25:06.996274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:06.996190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:07.091938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:07.091905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:07.227231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:07.227198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:07.541361 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:07.541324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:07.744057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:07.744020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:09.508349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:09.508321    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:25:09.508776 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:09.508517    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:25:10.599789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.599760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:10.695291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.695264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:10.763439 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:10.763958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763936    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763960    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763976    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.763998    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764027    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764035    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12b6847-28ca-4f38-9006-0b93d3bf748e" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:25:10.764080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764072    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="edbaebfb-f97a-48f1-aeb2-9ee6bc20dd17" containerName="revoke-keys"
Jul 17 17:25:10.764434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764198    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764209    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764222    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:10.764434 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.764239    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5934933-b1c2-4886-b69f-5b9cc85271bf" containerName="revoke-keys"
Jul 17 17:25:10.767730 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.767707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.770545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.770523    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f1f5b638-serving-cert\""
Jul 17 17:25:10.778978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.778945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:10.811194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.811165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:10.817222 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.817198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.817354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.817244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.817354 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.817316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrtv\" (UniqueName: \"kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.918296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.918201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.918296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.918267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.918544 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.918354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrtv\" (UniqueName: \"kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.918828 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.918806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.920704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.920687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.929373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.929343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrtv\" (UniqueName: \"kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv\") pod \"maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:10.950004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:10.949979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:11.079705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.079671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:11.146936 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.146904    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:11.147488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.147464    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:11.147770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.147650    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a1d35c-fe79-469e-b041-2471fd9982b4" containerName="revoke-keys"
Jul 17 17:25:11.151018 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.150999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:11.153120 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.153099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:25:11.156413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.156393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:11.220901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.220809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:11.220901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.220846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2\" (UID: \"701fdfaf-a168-46d6-8795-bdc887d477e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:11.223781 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:25:11.223760    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b2e337_03f7_4073_aa2b_38bae63d64af.slice/crio-3b8314b06ff0beb550602868f397c1ea1c7e4d2921df0fda41acdc03e3faef67 WatchSource:0}: Error finding container 3b8314b06ff0beb550602868f397c1ea1c7e4d2921df0fda41acdc03e3faef67: Status 404 returned error can't find the container with id 3b8314b06ff0beb550602868f397c1ea1c7e4d2921df0fda41acdc03e3faef67
Jul 17 17:25:11.263967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.263932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:11.321845 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.321813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2\" (UID: \"701fdfaf-a168-46d6-8795-bdc887d477e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:11.322468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.322450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2\" (UID: \"701fdfaf-a168-46d6-8795-bdc887d477e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:11.366972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.366940    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:11.457289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.457257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" event={"ID":"b2b2e337-03f7-4073-aa2b-38bae63d64af","Type":"ContainerStarted","Data":"3b8314b06ff0beb550602868f397c1ea1c7e4d2921df0fda41acdc03e3faef67"}
Jul 17 17:25:11.463317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.463291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:11.590785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:11.590760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:11.594402 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:25:11.594367    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701fdfaf_a168_46d6_8795_bdc887d477e3.slice/crio-0131463c5617d78e636d859996c15703d16763a98f4b474808639e0b2ebe8bd7 WatchSource:0}: Error finding container 0131463c5617d78e636d859996c15703d16763a98f4b474808639e0b2ebe8bd7: Status 404 returned error can't find the container with id 0131463c5617d78e636d859996c15703d16763a98f4b474808639e0b2ebe8bd7
Jul 17 17:25:12.463535 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.463503    2561 generic.go:356] "Generic (PLEG): container finished" podID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerID="07efecdd5266543d19c990547543736e0396076ea801a4198c344542053e4615" exitCode=7
Jul 17 17:25:12.464003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.463588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" event={"ID":"701fdfaf-a168-46d6-8795-bdc887d477e3","Type":"ContainerDied","Data":"07efecdd5266543d19c990547543736e0396076ea801a4198c344542053e4615"}
Jul 17 17:25:12.464003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.463653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" event={"ID":"701fdfaf-a168-46d6-8795-bdc887d477e3","Type":"ContainerStarted","Data":"0131463c5617d78e636d859996c15703d16763a98f4b474808639e0b2ebe8bd7"}
Jul 17 17:25:12.464003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.463780    2561 scope.go:117] "RemoveContainer" containerID="07efecdd5266543d19c990547543736e0396076ea801a4198c344542053e4615"
Jul 17 17:25:12.465305 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.465276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" event={"ID":"b2b2e337-03f7-4073-aa2b-38bae63d64af","Type":"ContainerStarted","Data":"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"}
Jul 17 17:25:12.465401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.465330    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:12.507310 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:12.507266    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" podStartSLOduration=2.072769261 podStartE2EDuration="2.507246369s" totalImagesPullingTime="434.477108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:12.505287878 +0000 UTC m=+3546.490375421" watchObservedRunningTime="2026-07-17 17:25:12.507246369 +0000 UTC m=+3546.492333909"
Jul 17 17:25:13.471324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:13.471293    2561 generic.go:356] "Generic (PLEG): container finished" podID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerID="834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c" exitCode=7
Jul 17 17:25:13.471762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:13.471385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" event={"ID":"701fdfaf-a168-46d6-8795-bdc887d477e3","Type":"ContainerDied","Data":"834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c"}
Jul 17 17:25:13.471762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:13.471452    2561 scope.go:117] "RemoveContainer" containerID="07efecdd5266543d19c990547543736e0396076ea801a4198c344542053e4615"
Jul 17 17:25:13.471762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:13.471552    2561 scope.go:117] "RemoveContainer" containerID="834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c"
Jul 17 17:25:13.471930 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:13.471822    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2_odh-ai-gateway-infra(701fdfaf-a168-46d6-8795-bdc887d477e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3"
Jul 17 17:25:14.477114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:14.477081    2561 scope.go:117] "RemoveContainer" containerID="834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c"
Jul 17 17:25:14.477510 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:14.477283    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2_odh-ai-gateway-infra(701fdfaf-a168-46d6-8795-bdc887d477e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3"
Jul 17 17:25:15.068231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:15.068188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:15.721189 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:15.721152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:15.820948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:15.820916    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:16.139823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:16.139792    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:18.477263 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:18.477234    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:18.508838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:18.508811    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:25:18.509095 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:18.509070    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:25:21.785569 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:21.785521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:21.885472 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:21.885438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:22.147127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.147035    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"]
Jul 17 17:25:22.150486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.150469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:22.164040 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.164001    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"]
Jul 17 17:25:22.196392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.196360    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:25:22.212798 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.212767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs\" (UID: \"7d905f7b-be0b-4781-bcbb-750ed1bc70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:22.313444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.313404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs\" (UID: \"7d905f7b-be0b-4781-bcbb-750ed1bc70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:22.314009 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.313990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs\" (UID: \"7d905f7b-be0b-4781-bcbb-750ed1bc70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:22.459991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.459887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:22.597415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:22.597386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"]
Jul 17 17:25:22.599010 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:25:22.598979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d905f7b_be0b_4781_bcbb_750ed1bc70d2.slice/crio-3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3 WatchSource:0}: Error finding container 3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3: Status 404 returned error can't find the container with id 3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3
Jul 17 17:25:23.525124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:23.525089    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d905f7b-be0b-4781-bcbb-750ed1bc70d2" containerID="c724f7fbcce6ffc978e311d6e63a9fda95b208f0b9d4cbce12739a0950416294" exitCode=0
Jul 17 17:25:23.525504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:23.525171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs" event={"ID":"7d905f7b-be0b-4781-bcbb-750ed1bc70d2","Type":"ContainerDied","Data":"c724f7fbcce6ffc978e311d6e63a9fda95b208f0b9d4cbce12739a0950416294"}
Jul 17 17:25:23.525504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:23.525222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs" event={"ID":"7d905f7b-be0b-4781-bcbb-750ed1bc70d2","Type":"ContainerStarted","Data":"3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3"}
Jul 17 17:25:24.509118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:24.509090    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:25:24.509314 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:24.509294    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:25:24.665460 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:24.665435    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:24.734089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:24.734013    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca\") pod \"7d905f7b-be0b-4781-bcbb-750ed1bc70d2\" (UID: \"7d905f7b-be0b-4781-bcbb-750ed1bc70d2\") "
Jul 17 17:25:24.734397 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:24.734372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d905f7b-be0b-4781-bcbb-750ed1bc70d2" (UID: "7d905f7b-be0b-4781-bcbb-750ed1bc70d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:25:24.835619 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:24.835586    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d905f7b-be0b-4781-bcbb-750ed1bc70d2-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:25:25.537580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:25.537554    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"
Jul 17 17:25:25.537580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:25.537561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs" event={"ID":"7d905f7b-be0b-4781-bcbb-750ed1bc70d2","Type":"ContainerDied","Data":"3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3"}
Jul 17 17:25:25.537806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:25.537610    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3868af4b8dbc8c5fdb8fa579b1385b4e9af9231f810fb153379238ab641563c3"
Jul 17 17:25:28.508380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:28.508348    2561 scope.go:117] "RemoveContainer" containerID="834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c"
Jul 17 17:25:29.509327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:29.509213    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:25:29.509758 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:29.509422    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:25:29.556837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:29.556803    2561 generic.go:356] "Generic (PLEG): container finished" podID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerID="af8b19a704310e6e85077a06b98d5ce8ef18fdd57d3d869e08694ab89282e4f0" exitCode=7
Jul 17 17:25:29.557064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:29.556851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" event={"ID":"701fdfaf-a168-46d6-8795-bdc887d477e3","Type":"ContainerDied","Data":"af8b19a704310e6e85077a06b98d5ce8ef18fdd57d3d869e08694ab89282e4f0"}
Jul 17 17:25:29.557064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:29.556912    2561 scope.go:117] "RemoveContainer" containerID="834d71633d33894e35f9d315e86b6e21cffeec374bf390f64afe29aaaa7ff28c"
Jul 17 17:25:29.557264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:29.557245    2561 scope.go:117] "RemoveContainer" containerID="af8b19a704310e6e85077a06b98d5ce8ef18fdd57d3d869e08694ab89282e4f0"
Jul 17 17:25:29.557467 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:29.557446    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2_odh-ai-gateway-infra(701fdfaf-a168-46d6-8795-bdc887d477e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3"
Jul 17 17:25:30.584224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:30.584184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:30.722065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:30.722017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:30.789895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:30.789862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca\") pod \"701fdfaf-a168-46d6-8795-bdc887d477e3\" (UID: \"701fdfaf-a168-46d6-8795-bdc887d477e3\") "
Jul 17 17:25:30.790242 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:30.790218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "701fdfaf-a168-46d6-8795-bdc887d477e3" (UID: "701fdfaf-a168-46d6-8795-bdc887d477e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:25:30.891106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:30.890982    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701fdfaf-a168-46d6-8795-bdc887d477e3-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:25:31.569951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:31.569920    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"
Jul 17 17:25:31.570166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:31.569950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2" event={"ID":"701fdfaf-a168-46d6-8795-bdc887d477e3","Type":"ContainerDied","Data":"0131463c5617d78e636d859996c15703d16763a98f4b474808639e0b2ebe8bd7"}
Jul 17 17:25:31.570166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:31.570009    2561 scope.go:117] "RemoveContainer" containerID="af8b19a704310e6e85077a06b98d5ce8ef18fdd57d3d869e08694ab89282e4f0"
Jul 17 17:25:31.590867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:31.590840    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:31.596846 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:31.596818    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-qgqf2"]
Jul 17 17:25:32.514451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.514420    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" path="/var/lib/kubelet/pods/701fdfaf-a168-46d6-8795-bdc887d477e3/volumes"
Jul 17 17:25:32.544790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.544761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:32.544995 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.544970    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" podUID="b2b2e337-03f7-4073-aa2b-38bae63d64af" containerName="maas-api" containerID="cri-o://dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced" gracePeriod=30
Jul 17 17:25:32.577281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.577254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:32.789217 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.789188    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:32.909394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.909361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls\") pod \"b2b2e337-03f7-4073-aa2b-38bae63d64af\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") "
Jul 17 17:25:32.909591 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.909419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqrtv\" (UniqueName: \"kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv\") pod \"b2b2e337-03f7-4073-aa2b-38bae63d64af\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") "
Jul 17 17:25:32.909591 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.909459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca\") pod \"b2b2e337-03f7-4073-aa2b-38bae63d64af\" (UID: \"b2b2e337-03f7-4073-aa2b-38bae63d64af\") "
Jul 17 17:25:32.909870 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.909846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2b2e337-03f7-4073-aa2b-38bae63d64af" (UID: "b2b2e337-03f7-4073-aa2b-38bae63d64af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:25:32.911713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.911690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv" (OuterVolumeSpecName: "kube-api-access-qqrtv") pod "b2b2e337-03f7-4073-aa2b-38bae63d64af" (UID: "b2b2e337-03f7-4073-aa2b-38bae63d64af"). InnerVolumeSpecName "kube-api-access-qqrtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:25:32.911849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:32.911715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2b2e337-03f7-4073-aa2b-38bae63d64af" (UID: "b2b2e337-03f7-4073-aa2b-38bae63d64af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:25:33.010375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.010342    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2b2e337-03f7-4073-aa2b-38bae63d64af-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:25:33.010375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.010370    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrtv\" (UniqueName: \"kubernetes.io/projected/b2b2e337-03f7-4073-aa2b-38bae63d64af-kube-api-access-qqrtv\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:25:33.010375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.010379    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2e337-03f7-4073-aa2b-38bae63d64af-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:25:33.581276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.581238    2561 generic.go:356] "Generic (PLEG): container finished" podID="b2b2e337-03f7-4073-aa2b-38bae63d64af" containerID="dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced" exitCode=0
Jul 17 17:25:33.581449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.581286    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"
Jul 17 17:25:33.581449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.581286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" event={"ID":"b2b2e337-03f7-4073-aa2b-38bae63d64af","Type":"ContainerDied","Data":"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"}
Jul 17 17:25:33.581449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.581340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp" event={"ID":"b2b2e337-03f7-4073-aa2b-38bae63d64af","Type":"ContainerDied","Data":"3b8314b06ff0beb550602868f397c1ea1c7e4d2921df0fda41acdc03e3faef67"}
Jul 17 17:25:33.581449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.581364    2561 scope.go:117] "RemoveContainer" containerID="dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"
Jul 17 17:25:33.596261 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.596242    2561 scope.go:117] "RemoveContainer" containerID="dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"
Jul 17 17:25:33.596538 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:33.596519    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced\": container with ID starting with dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced not found: ID does not exist" containerID="dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"
Jul 17 17:25:33.596598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.596547    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced"} err="failed to get container status \"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced\": rpc error: code = NotFound desc = could not find container \"dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced\": container with ID starting with dc01fb1f5f5f1c7a0509d41b5f6c3c1636ca357a3f0343985ecb11b1653e3ced not found: ID does not exist"
Jul 17 17:25:33.605874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.605850    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:33.609590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:33.609571    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1f5b638-7f6454f9cf-j5kmp"]
Jul 17 17:25:34.515830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:34.515799    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b2e337-03f7-4073-aa2b-38bae63d64af" path="/var/lib/kubelet/pods/b2b2e337-03f7-4073-aa2b-38bae63d64af/volumes"
Jul 17 17:25:35.559532 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:35.559504    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:25:37.233355 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:37.233324    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"]
Jul 17 17:25:37.236017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:37.235993    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f1f5b638-s5vhs"]
Jul 17 17:25:37.237292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:37.237265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:37.595605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:37.595575    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:38.508856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:38.508828    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:25:38.509385 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:38.509000    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:25:38.517997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:38.517967    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d905f7b-be0b-4781-bcbb-750ed1bc70d2" path="/var/lib/kubelet/pods/7d905f7b-be0b-4781-bcbb-750ed1bc70d2/volumes"
Jul 17 17:25:41.414956 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:41.414915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:41.470271 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:41.470246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:43.359468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:43.359423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:43.461037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:43.460977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:43.606969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:43.606938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:43.823223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:43.823194    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:43.957943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:43.957902    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:44.044652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:44.044610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:44.165002 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:44.164911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:44.508315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:44.508244    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:25:44.508663 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:44.508414    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:25:45.741750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.741718    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742103    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742121    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742127    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b2e337-03f7-4073-aa2b-38bae63d64af" containerName="maas-api"
Jul 17 17:25:45.742152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b2e337-03f7-4073-aa2b-38bae63d64af" containerName="maas-api"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742159    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d905f7b-be0b-4781-bcbb-750ed1bc70d2" containerName="revoke-keys"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742165    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d905f7b-be0b-4781-bcbb-750ed1bc70d2" containerName="revoke-keys"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742217    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b2e337-03f7-4073-aa2b-38bae63d64af" containerName="maas-api"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742224    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742230    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.742347 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.742236    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d905f7b-be0b-4781-bcbb-750ed1bc70d2" containerName="revoke-keys"
Jul 17 17:25:45.746646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.746627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:25:45.748935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.748914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:25:45.751725 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.751701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:25:45.820010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.819975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2vb6c\" (UID: \"0db56f90-9cab-4788-8ee4-9eb27f372fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:25:45.920503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.920471    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:25:45.920789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.920767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2vb6c\" (UID: \"0db56f90-9cab-4788-8ee4-9eb27f372fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:25:45.920873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.920860    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.920932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.920875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.920969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.920942    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="701fdfaf-a168-46d6-8795-bdc887d477e3" containerName="revoke-keys"
Jul 17 17:25:45.921384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.921366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-2vb6c\" (UID: \"0db56f90-9cab-4788-8ee4-9eb27f372fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:25:45.924068 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.924033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:25:45.930731 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:45.930717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:25:46.021274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.021213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df\" (UID: \"8143e97b-141e-4118-bf1b-060775eb9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:25:46.057630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.057612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:25:46.122027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.122002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df\" (UID: \"8143e97b-141e-4118-bf1b-060775eb9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:25:46.122860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.122809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df\" (UID: \"8143e97b-141e-4118-bf1b-060775eb9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:25:46.181272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.181249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:25:46.183375 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:25:46.183346    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db56f90_9cab_4788_8ee4_9eb27f372fe1.slice/crio-60d6bfeabffde07e96b92e5ddef3bb7f6ad8e8ecadb3094c12a99d7cc1000796 WatchSource:0}: Error finding container 60d6bfeabffde07e96b92e5ddef3bb7f6ad8e8ecadb3094c12a99d7cc1000796: Status 404 returned error can't find the container with id 60d6bfeabffde07e96b92e5ddef3bb7f6ad8e8ecadb3094c12a99d7cc1000796
Jul 17 17:25:46.235155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.235131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:25:46.363154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.363128    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:25:46.364864 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:25:46.364843    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8143e97b_141e_4118_bf1b_060775eb9ff7.slice/crio-ea066878665602df855afb2b4ed1c99c2509b36f1540982330eb8cb0291ef9a3 WatchSource:0}: Error finding container ea066878665602df855afb2b4ed1c99c2509b36f1540982330eb8cb0291ef9a3: Status 404 returned error can't find the container with id ea066878665602df855afb2b4ed1c99c2509b36f1540982330eb8cb0291ef9a3
Jul 17 17:25:46.647361 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.647327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerStarted","Data":"df12a94f169ea4c64a508fec8eae47b49a8ab9dc6a5b2db14fb2d880be876682"}
Jul 17 17:25:46.647490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.647382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerStarted","Data":"ea066878665602df855afb2b4ed1c99c2509b36f1540982330eb8cb0291ef9a3"}
Jul 17 17:25:46.648785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.648758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerStarted","Data":"c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590"}
Jul 17 17:25:46.648881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.648813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerStarted","Data":"60d6bfeabffde07e96b92e5ddef3bb7f6ad8e8ecadb3094c12a99d7cc1000796"}
Jul 17 17:25:46.664749 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.664700    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" podStartSLOduration=1.664685272 podStartE2EDuration="1.664685272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:46.661955663 +0000 UTC m=+3580.647043201" watchObservedRunningTime="2026-07-17 17:25:46.664685272 +0000 UTC m=+3580.649772810"
Jul 17 17:25:46.678118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:46.678073    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" podStartSLOduration=1.678060008 podStartE2EDuration="1.678060008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:25:46.676411295 +0000 UTC m=+3580.661498835" watchObservedRunningTime="2026-07-17 17:25:46.678060008 +0000 UTC m=+3580.663147543"
Jul 17 17:25:46.839275 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:46.839244    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db56f90_9cab_4788_8ee4_9eb27f372fe1.slice/crio-c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 17:25:47.655028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.654998    2561 generic.go:356] "Generic (PLEG): container finished" podID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerID="df12a94f169ea4c64a508fec8eae47b49a8ab9dc6a5b2db14fb2d880be876682" exitCode=7
Jul 17 17:25:47.655247 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.655037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerDied","Data":"df12a94f169ea4c64a508fec8eae47b49a8ab9dc6a5b2db14fb2d880be876682"}
Jul 17 17:25:47.655330 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.655246    2561 scope.go:117] "RemoveContainer" containerID="df12a94f169ea4c64a508fec8eae47b49a8ab9dc6a5b2db14fb2d880be876682"
Jul 17 17:25:47.656664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.656642    2561 generic.go:356] "Generic (PLEG): container finished" podID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerID="c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590" exitCode=7
Jul 17 17:25:47.656783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.656687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerDied","Data":"c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590"}
Jul 17 17:25:47.656906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:47.656881    2561 scope.go:117] "RemoveContainer" containerID="c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590"
Jul 17 17:25:48.663078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.663024    2561 generic.go:356] "Generic (PLEG): container finished" podID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerID="b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba" exitCode=7
Jul 17 17:25:48.663557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.663084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerDied","Data":"b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba"}
Jul 17 17:25:48.663557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.663144    2561 scope.go:117] "RemoveContainer" containerID="df12a94f169ea4c64a508fec8eae47b49a8ab9dc6a5b2db14fb2d880be876682"
Jul 17 17:25:48.663557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.663357    2561 scope.go:117] "RemoveContainer" containerID="b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba"
Jul 17 17:25:48.663719 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:48.663585    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df_odh-ai-gateway-infra(8143e97b-141e-4118-bf1b-060775eb9ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7"
Jul 17 17:25:48.664976 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.664955    2561 generic.go:356] "Generic (PLEG): container finished" podID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerID="32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb" exitCode=7
Jul 17 17:25:48.665116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.664982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerDied","Data":"32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb"}
Jul 17 17:25:48.665201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.665188    2561 scope.go:117] "RemoveContainer" containerID="32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb"
Jul 17 17:25:48.665366 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:48.665350    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2vb6c_odh-ai-gateway-infra(0db56f90-9cab-4788-8ee4-9eb27f372fe1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1"
Jul 17 17:25:48.673823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:48.673799    2561 scope.go:117] "RemoveContainer" containerID="c7524b51419292eec84a990a15905d7b8f7885b0fd959ccae7c2eba95d8d4590"
Jul 17 17:25:49.671943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:49.671906    2561 scope.go:117] "RemoveContainer" containerID="b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba"
Jul 17 17:25:49.672405 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:49.672132    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df_odh-ai-gateway-infra(8143e97b-141e-4118-bf1b-060775eb9ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7"
Jul 17 17:25:49.673453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:49.673437    2561 scope.go:117] "RemoveContainer" containerID="32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb"
Jul 17 17:25:49.673593 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:49.673576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2vb6c_odh-ai-gateway-infra(0db56f90-9cab-4788-8ee4-9eb27f372fe1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1"
Jul 17 17:25:49.848356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:49.848320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:49.952414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:49.952338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:50.059928 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:50.059895    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:50.276310 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:50.276280    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:53.509173 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:53.509136    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:25:53.509649 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:53.509364    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:25:55.982783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:55.982749    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:56.086910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:56.086866    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:25:56.515065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:25:56.511838    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:25:56.515065 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:25:56.512075    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:26:01.508506 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:01.508425    2561 scope.go:117] "RemoveContainer" containerID="b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba"
Jul 17 17:26:02.508400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.508327    2561 scope.go:117] "RemoveContainer" containerID="32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb"
Jul 17 17:26:02.737883 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.737847    2561 generic.go:356] "Generic (PLEG): container finished" podID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerID="309335dd035d9e36c3d241f47a9ee966faac8d433e51208e24b1a23fd3c3d462" exitCode=7
Jul 17 17:26:02.738281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.737923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerDied","Data":"309335dd035d9e36c3d241f47a9ee966faac8d433e51208e24b1a23fd3c3d462"}
Jul 17 17:26:02.738281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.738007    2561 scope.go:117] "RemoveContainer" containerID="b4e9f1233813a2479868a10a33ba699a4176821465bba77cc8cfdca5a47d1cba"
Jul 17 17:26:02.738688 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.738415    2561 scope.go:117] "RemoveContainer" containerID="309335dd035d9e36c3d241f47a9ee966faac8d433e51208e24b1a23fd3c3d462"
Jul 17 17:26:02.738688 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:02.738672    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df_odh-ai-gateway-infra(8143e97b-141e-4118-bf1b-060775eb9ff7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7"
Jul 17 17:26:02.830488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:02.830456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:03.161057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.161013    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:03.283618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.283587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:03.746415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.746385    2561 generic.go:356] "Generic (PLEG): container finished" podID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerID="c2ec1e01288a6af86fd26dcf04aaef62ebca9f8139e68b3ee98c598e20bb725d" exitCode=7
Jul 17 17:26:03.746856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.746454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerDied","Data":"c2ec1e01288a6af86fd26dcf04aaef62ebca9f8139e68b3ee98c598e20bb725d"}
Jul 17 17:26:03.746856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.746512    2561 scope.go:117] "RemoveContainer" containerID="32e7b75b3e4fe97110ab0429974891c495c56b85ed7113d7906c2904668332eb"
Jul 17 17:26:03.746856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.746801    2561 scope.go:117] "RemoveContainer" containerID="c2ec1e01288a6af86fd26dcf04aaef62ebca9f8139e68b3ee98c598e20bb725d"
Jul 17 17:26:03.747004 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:03.746985    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-2vb6c_odh-ai-gateway-infra(0db56f90-9cab-4788-8ee4-9eb27f372fe1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1"
Jul 17 17:26:03.768545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.768517    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:26:03.892067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.892021    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:26:03.988458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.988423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca\") pod \"8143e97b-141e-4118-bf1b-060775eb9ff7\" (UID: \"8143e97b-141e-4118-bf1b-060775eb9ff7\") "
Jul 17 17:26:03.988785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:03.988762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8143e97b-141e-4118-bf1b-060775eb9ff7" (UID: "8143e97b-141e-4118-bf1b-060775eb9ff7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:26:04.089145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.089110    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8143e97b-141e-4118-bf1b-060775eb9ff7-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:26:04.753748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.753713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df" event={"ID":"8143e97b-141e-4118-bf1b-060775eb9ff7","Type":"ContainerDied","Data":"ea066878665602df855afb2b4ed1c99c2509b36f1540982330eb8cb0291ef9a3"}
Jul 17 17:26:04.753748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.753747    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"
Jul 17 17:26:04.754259 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.753777    2561 scope.go:117] "RemoveContainer" containerID="309335dd035d9e36c3d241f47a9ee966faac8d433e51208e24b1a23fd3c3d462"
Jul 17 17:26:04.769495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.769465    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:26:04.771584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.771536    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-r77df"]
Jul 17 17:26:04.774717 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.774699    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:26:04.906201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.906174    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:26:04.996885 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.996859    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca\") pod \"0db56f90-9cab-4788-8ee4-9eb27f372fe1\" (UID: \"0db56f90-9cab-4788-8ee4-9eb27f372fe1\") "
Jul 17 17:26:04.997211 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:04.997191    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0db56f90-9cab-4788-8ee4-9eb27f372fe1" (UID: "0db56f90-9cab-4788-8ee4-9eb27f372fe1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:26:05.097548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.097519    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db56f90-9cab-4788-8ee4-9eb27f372fe1-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:26:05.508870 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.508800    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:26:05.762422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.762337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c" event={"ID":"0db56f90-9cab-4788-8ee4-9eb27f372fe1","Type":"ContainerDied","Data":"60d6bfeabffde07e96b92e5ddef3bb7f6ad8e8ecadb3094c12a99d7cc1000796"}
Jul 17 17:26:05.762422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.762363    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"
Jul 17 17:26:05.762422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.762393    2561 scope.go:117] "RemoveContainer" containerID="c2ec1e01288a6af86fd26dcf04aaef62ebca9f8139e68b3ee98c598e20bb725d"
Jul 17 17:26:05.783404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.783382    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:26:05.788122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.788099    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-2vb6c"]
Jul 17 17:26:05.975342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:05.975309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:06.062606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.062574    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:06.207612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207583    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:26:06.207960 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207947    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207962    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207971    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207976    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207987    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.207994    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208008    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208014    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208026    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208030    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208106    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208116    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208123    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208128    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208135    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.208175 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.208141    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" containerName="revoke-keys"
Jul 17 17:26:06.211078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.211061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.213342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.213320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9777fc6b-serving-cert\""
Jul 17 17:26:06.220672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.220649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:26:06.247272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.247245    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:06.305886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.305858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.305991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.305902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.306040 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.305987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgtl\" (UniqueName: \"kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.407063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.406972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.407063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.407017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.407228 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.407067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgtl\" (UniqueName: \"kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.407644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.407627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.409804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.409785    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9777fc6b-serving-cert\""
Jul 17 17:26:06.420213 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.420187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.423194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.423173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgtl\" (UniqueName: \"kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl\") pod \"maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.479174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.479138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:06.514329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.514297    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" path="/var/lib/kubelet/pods/0db56f90-9cab-4788-8ee4-9eb27f372fe1/volumes"
Jul 17 17:26:06.514640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.514627    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8143e97b-141e-4118-bf1b-060775eb9ff7" path="/var/lib/kubelet/pods/8143e97b-141e-4118-bf1b-060775eb9ff7/volumes"
Jul 17 17:26:06.538542 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.538509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:06.662703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.662620    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:26:06.672137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.672089    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:26:06.699543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.699452    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:26:06.701770 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:26:06.701747    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb70c163_82b5_4e59_b13b_68d3a1887040.slice/crio-c2a312228102026f2c7fcdef87702da76bded8077d9b2e280c9a46bf4652574a WatchSource:0}: Error finding container c2a312228102026f2c7fcdef87702da76bded8077d9b2e280c9a46bf4652574a: Status 404 returned error can't find the container with id c2a312228102026f2c7fcdef87702da76bded8077d9b2e280c9a46bf4652574a
Jul 17 17:26:06.745493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.745465    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:06.768743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.768716    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255" exitCode=1
Jul 17 17:26:06.769155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.768790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"}
Jul 17 17:26:06.769155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.768843    2561 scope.go:117] "RemoveContainer" containerID="57a075fabfe147fe09b8a04aa9c3f40937728b9177b53728d9d98fdb6ae94b38"
Jul 17 17:26:06.769272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.769251    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:06.769492 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:06.769470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:06.770185 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.770169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" event={"ID":"eb70c163-82b5-4e59-b13b-68d3a1887040","Type":"ContainerStarted","Data":"c2a312228102026f2c7fcdef87702da76bded8077d9b2e280c9a46bf4652574a"}
Jul 17 17:26:06.847403 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:06.847376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:07.067821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:07.067796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:26:07.777440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:07.777400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" event={"ID":"eb70c163-82b5-4e59-b13b-68d3a1887040","Type":"ContainerStarted","Data":"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"}
Jul 17 17:26:07.777906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:07.777491    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:07.779130 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:07.779113    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:07.779274 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:07.779257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:07.795553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:07.795505    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" podStartSLOduration=1.470152942 podStartE2EDuration="1.795493914s" totalImagesPullingTime="325.340972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:07.793529765 +0000 UTC m=+3601.778617306" watchObservedRunningTime="2026-07-17 17:26:07.795493914 +0000 UTC m=+3601.780581505"
Jul 17 17:26:10.456154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:10.456115    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:11.188030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.187999    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:11.188376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.188364    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:11.188420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.188378    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db56f90-9cab-4788-8ee4-9eb27f372fe1" containerName="revoke-keys"
Jul 17 17:26:11.191613 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.191598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:11.193890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.193869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:26:11.199535 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.199511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:11.241385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.241359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj\" (UID: \"d6885824-b241-4b65-899c-f87fdc5b461f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:11.342057 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.342016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj\" (UID: \"d6885824-b241-4b65-899c-f87fdc5b461f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:11.342577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.342556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj\" (UID: \"d6885824-b241-4b65-899c-f87fdc5b461f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:11.504297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.504210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:11.508834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.508812    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:26:11.508993 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:11.508974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:26:11.631274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.631211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:11.633791 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:26:11.633766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6885824_b241_4b65_899c_f87fdc5b461f.slice/crio-8b36bcbf781596ac4474e3a8a83ee9f73a7a9ac29f245328e00712ac7d897033 WatchSource:0}: Error finding container 8b36bcbf781596ac4474e3a8a83ee9f73a7a9ac29f245328e00712ac7d897033: Status 404 returned error can't find the container with id 8b36bcbf781596ac4474e3a8a83ee9f73a7a9ac29f245328e00712ac7d897033
Jul 17 17:26:11.799660 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:11.799627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerStarted","Data":"8b36bcbf781596ac4474e3a8a83ee9f73a7a9ac29f245328e00712ac7d897033"}
Jul 17 17:26:12.162346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:12.162272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:26:12.805744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:12.805715    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6885824-b241-4b65-899c-f87fdc5b461f" containerID="509bb95c2234a6cb140536e033327bf3f0e94a6f94ac46f5b8b3921c83b7ab51" exitCode=7
Jul 17 17:26:12.806158 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:12.805787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerDied","Data":"509bb95c2234a6cb140536e033327bf3f0e94a6f94ac46f5b8b3921c83b7ab51"}
Jul 17 17:26:12.806158 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:12.805967    2561 scope.go:117] "RemoveContainer" containerID="509bb95c2234a6cb140536e033327bf3f0e94a6f94ac46f5b8b3921c83b7ab51"
Jul 17 17:26:13.789715 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:13.789690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:13.813018 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:13.812986    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6885824-b241-4b65-899c-f87fdc5b461f" containerID="dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103" exitCode=7
Jul 17 17:26:13.813735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:13.813077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerDied","Data":"dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103"}
Jul 17 17:26:13.813735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:13.813151    2561 scope.go:117] "RemoveContainer" containerID="509bb95c2234a6cb140536e033327bf3f0e94a6f94ac46f5b8b3921c83b7ab51"
Jul 17 17:26:13.813735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:13.813291    2561 scope.go:117] "RemoveContainer" containerID="dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103"
Jul 17 17:26:13.813735 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:13.813551    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj_odh-ai-gateway-infra(d6885824-b241-4b65-899c-f87fdc5b461f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" podUID="d6885824-b241-4b65-899c-f87fdc5b461f"
Jul 17 17:26:14.820026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:14.820001    2561 scope.go:117] "RemoveContainer" containerID="dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103"
Jul 17 17:26:14.820395 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:14.820244    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj_odh-ai-gateway-infra(d6885824-b241-4b65-899c-f87fdc5b461f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" podUID="d6885824-b241-4b65-899c-f87fdc5b461f"
Jul 17 17:26:14.969358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:14.969315    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:26:14.969724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:14.969711    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:14.969900 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:14.969868    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:22.183839 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:22.183805    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:22.286521 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:22.286491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:23.301416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.301384    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:26:23.304949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.304929    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.307328 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.307305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9777fc6b\"/\"openshift-service-ca.crt\""
Jul 17 17:26:23.307574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.307558    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9777fc6b\"/\"kube-root-ca.crt\""
Jul 17 17:26:23.308440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.308423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9777fc6b\"/\"default-dockercfg-52z8x\""
Jul 17 17:26:23.308514 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.308423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9777fc6b\"/\"e2e-lifecycle-model-9777fc6b-kserve-self-signed-certs\""
Jul 17 17:26:23.313319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.313296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:26:23.328475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.328448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:23.343243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.343333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.343380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.343380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrp2t\" (UniqueName: \"kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.343463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.343463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.343427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.444598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.444598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrp2t\" (UniqueName: \"kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.444833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.444833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.444833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.445035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.444866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.445035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.445009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.445222 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.445201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.445274 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.445235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.446942 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.446916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.447137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.447120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.452507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.452480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrp2t\" (UniqueName: \"kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t\") pod \"e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") " pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.615612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.615528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:23.749562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.749538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:26:23.751972 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:26:23.751939    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9608a8a4_dbca_4a4d_9985_559653b6f3e4.slice/crio-e5390523335bcb248de8b4402c2363118794489adc7b3db93363cfe7956aced0 WatchSource:0}: Error finding container e5390523335bcb248de8b4402c2363118794489adc7b3db93363cfe7956aced0: Status 404 returned error can't find the container with id e5390523335bcb248de8b4402c2363118794489adc7b3db93363cfe7956aced0
Jul 17 17:26:23.866407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.866332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" event={"ID":"9608a8a4-dbca-4a4d-9985-559653b6f3e4","Type":"ContainerStarted","Data":"f2a4bd091fd40fb2acd87465696034391d4015024559ef2ed5ff89348db7984a"}
Jul 17 17:26:23.866407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.866376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" event={"ID":"9608a8a4-dbca-4a4d-9985-559653b6f3e4","Type":"ContainerStarted","Data":"e5390523335bcb248de8b4402c2363118794489adc7b3db93363cfe7956aced0"}
Jul 17 17:26:23.882621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:23.882569    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" podStartSLOduration=0.882553516 podStartE2EDuration="882.553516ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:23.882234589 +0000 UTC m=+3617.867322129" watchObservedRunningTime="2026-07-17 17:26:23.882553516 +0000 UTC m=+3617.867641054"
Jul 17 17:26:24.311892 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:24.311863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:24.412317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:24.412285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:25.508691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:25.508659    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:26:25.509113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:25.508761    2561 scope.go:117] "RemoveContainer" containerID="dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103"
Jul 17 17:26:25.509113 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:25.508822    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:26:25.877764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:25.877726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerStarted","Data":"621dd72c51dd48adab80d67f28b86f8dcb00917afa882278fbfaa4e373ddb403"}
Jul 17 17:26:25.894033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:25.893982    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" podStartSLOduration=14.893968534999999 podStartE2EDuration="14.893968535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:26:25.89207184 +0000 UTC m=+3619.877159401" watchObservedRunningTime="2026-07-17 17:26:25.893968535 +0000 UTC m=+3619.879056074"
Jul 17 17:26:26.885001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:26.884974    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6885824-b241-4b65-899c-f87fdc5b461f" containerID="621dd72c51dd48adab80d67f28b86f8dcb00917afa882278fbfaa4e373ddb403" exitCode=7
Jul 17 17:26:26.885423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:26.885056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerDied","Data":"621dd72c51dd48adab80d67f28b86f8dcb00917afa882278fbfaa4e373ddb403"}
Jul 17 17:26:26.885423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:26.885106    2561 scope.go:117] "RemoveContainer" containerID="dbdf3ad09379484f061264841b1be041cd9146710b6d8825934ae9a4a0777103"
Jul 17 17:26:26.885423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:26.885373    2561 scope.go:117] "RemoveContainer" containerID="621dd72c51dd48adab80d67f28b86f8dcb00917afa882278fbfaa4e373ddb403"
Jul 17 17:26:26.885563 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:26.885545    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj_odh-ai-gateway-infra(d6885824-b241-4b65-899c-f87fdc5b461f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" podUID="d6885824-b241-4b65-899c-f87fdc5b461f"
Jul 17 17:26:26.902505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:26.902482    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:27.491449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:27.491419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:27.594700 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:27.594675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:27.660766 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:27.660736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:26:28.022750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.022727    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:28.085612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.085588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca\") pod \"d6885824-b241-4b65-899c-f87fdc5b461f\" (UID: \"d6885824-b241-4b65-899c-f87fdc5b461f\") "
Jul 17 17:26:28.085921 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.085901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6885824-b241-4b65-899c-f87fdc5b461f" (UID: "d6885824-b241-4b65-899c-f87fdc5b461f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:26:28.187080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.186999    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6885824-b241-4b65-899c-f87fdc5b461f-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:26:28.896459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.896432    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"
Jul 17 17:26:28.896459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.896451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj" event={"ID":"d6885824-b241-4b65-899c-f87fdc5b461f","Type":"ContainerDied","Data":"8b36bcbf781596ac4474e3a8a83ee9f73a7a9ac29f245328e00712ac7d897033"}
Jul 17 17:26:28.896661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.896526    2561 scope.go:117] "RemoveContainer" containerID="621dd72c51dd48adab80d67f28b86f8dcb00917afa882278fbfaa4e373ddb403"
Jul 17 17:26:28.912671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.912648    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:28.918945 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:28.918925    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-jh9sj"]
Jul 17 17:26:29.508771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:29.508747    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:29.509187 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:29.508935    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:30.514141 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:30.514104    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" path="/var/lib/kubelet/pods/d6885824-b241-4b65-899c-f87fdc5b461f/volumes"
Jul 17 17:26:33.616279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:33.616251    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:33.616639 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:33.616294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:33.628453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:33.628430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:33.932988 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:33.932923    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:26:38.509012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:38.508979    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:26:38.509443 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:38.509186    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:26:40.660772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:40.660739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:26:42.508466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:42.508439    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:42.508938 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:42.508624    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:45.780212 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780180    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:26:45.780574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780544    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.780574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780555    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.780651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780586    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.780651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780594    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.780713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780657    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.780713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.780664    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:45.784984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.784967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:26:45.787423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.787405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:26:45.793238 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.793217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:26:45.941630 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:45.941595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-84txk\" (UID: \"0644c1af-de1f-4c77-a701-e89c9c32d0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:26:46.042624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.042549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-84txk\" (UID: \"0644c1af-de1f-4c77-a701-e89c9c32d0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:26:46.043218 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.043196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-84txk\" (UID: \"0644c1af-de1f-4c77-a701-e89c9c32d0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:26:46.096602 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.096572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:26:46.223014 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.222989    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:26:46.224753 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:26:46.224726    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0644c1af_de1f_4c77_a701_e89c9c32d0dc.slice/crio-105a69529d4c12819d52b0efb89d6f94e3d0654f92a377ff1dc79bc40676fc72 WatchSource:0}: Error finding container 105a69529d4c12819d52b0efb89d6f94e3d0654f92a377ff1dc79bc40676fc72: Status 404 returned error can't find the container with id 105a69529d4c12819d52b0efb89d6f94e3d0654f92a377ff1dc79bc40676fc72
Jul 17 17:26:46.984951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.984918    2561 generic.go:356] "Generic (PLEG): container finished" podID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerID="294ed44a8aa908fec470268dbb5cafe98fb8125b981389f6234319f9a0cf9a96" exitCode=7
Jul 17 17:26:46.985317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.985007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerDied","Data":"294ed44a8aa908fec470268dbb5cafe98fb8125b981389f6234319f9a0cf9a96"}
Jul 17 17:26:46.985317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.985095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerStarted","Data":"105a69529d4c12819d52b0efb89d6f94e3d0654f92a377ff1dc79bc40676fc72"}
Jul 17 17:26:46.985317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:46.985226    2561 scope.go:117] "RemoveContainer" containerID="294ed44a8aa908fec470268dbb5cafe98fb8125b981389f6234319f9a0cf9a96"
Jul 17 17:26:47.992026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:47.991944    2561 generic.go:356] "Generic (PLEG): container finished" podID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerID="23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e" exitCode=7
Jul 17 17:26:47.992026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:47.992004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerDied","Data":"23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e"}
Jul 17 17:26:47.992524 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:47.992087    2561 scope.go:117] "RemoveContainer" containerID="294ed44a8aa908fec470268dbb5cafe98fb8125b981389f6234319f9a0cf9a96"
Jul 17 17:26:47.992524 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:47.992306    2561 scope.go:117] "RemoveContainer" containerID="23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e"
Jul 17 17:26:47.992634 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:47.992534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-84txk_odh-ai-gateway-infra(0644c1af-de1f-4c77-a701-e89c9c32d0dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc"
Jul 17 17:26:48.997899 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:48.997873    2561 scope.go:117] "RemoveContainer" containerID="23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e"
Jul 17 17:26:48.998391 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:48.998154    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-84txk_odh-ai-gateway-infra(0644c1af-de1f-4c77-a701-e89c9c32d0dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc"
Jul 17 17:26:51.508682 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:51.508647    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:26:51.509136 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:51.508837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:26:53.694671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.694625    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:26:53.695188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.695165    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:53.695188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.695185    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:53.695340 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.695326    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6885824-b241-4b65-899c-f87fdc5b461f" containerName="revoke-keys"
Jul 17 17:26:53.698384 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.698363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:26:53.706104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.706079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:26:53.814203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.814171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg\" (UID: \"d69ebf51-aa6e-4f00-b104-bcd79c288892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:26:53.914995 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.914964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg\" (UID: \"d69ebf51-aa6e-4f00-b104-bcd79c288892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:26:53.915624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:53.915600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg\" (UID: \"d69ebf51-aa6e-4f00-b104-bcd79c288892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:26:54.009250 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:54.009165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:26:54.141747 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:54.141717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:26:54.143014 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:26:54.142987    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69ebf51_aa6e_4f00_b104_bcd79c288892.slice/crio-4851da0fa5b93675ded9379e3f38c516ec542c0c1abc4e2212cc62ec235165e9 WatchSource:0}: Error finding container 4851da0fa5b93675ded9379e3f38c516ec542c0c1abc4e2212cc62ec235165e9: Status 404 returned error can't find the container with id 4851da0fa5b93675ded9379e3f38c516ec542c0c1abc4e2212cc62ec235165e9
Jul 17 17:26:55.026776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:55.026745    2561 generic.go:356] "Generic (PLEG): container finished" podID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerID="838b114267bd25f25a688f39852d2a79fec70f28a6932d07c3ed5dbcf909e705" exitCode=7
Jul 17 17:26:55.027161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:55.026792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerDied","Data":"838b114267bd25f25a688f39852d2a79fec70f28a6932d07c3ed5dbcf909e705"}
Jul 17 17:26:55.027161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:55.026835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerStarted","Data":"4851da0fa5b93675ded9379e3f38c516ec542c0c1abc4e2212cc62ec235165e9"}
Jul 17 17:26:55.027161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:55.026996    2561 scope.go:117] "RemoveContainer" containerID="838b114267bd25f25a688f39852d2a79fec70f28a6932d07c3ed5dbcf909e705"
Jul 17 17:26:56.033278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:56.033244    2561 generic.go:356] "Generic (PLEG): container finished" podID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerID="6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad" exitCode=7
Jul 17 17:26:56.033679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:56.033283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerDied","Data":"6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad"}
Jul 17 17:26:56.033679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:56.033347    2561 scope.go:117] "RemoveContainer" containerID="838b114267bd25f25a688f39852d2a79fec70f28a6932d07c3ed5dbcf909e705"
Jul 17 17:26:56.033679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:56.033519    2561 scope.go:117] "RemoveContainer" containerID="6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad"
Jul 17 17:26:56.033847 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:56.033753    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg_odh-ai-gateway-infra(d69ebf51-aa6e-4f00-b104-bcd79c288892)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892"
Jul 17 17:26:56.510910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:56.510841    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:26:56.511074 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:56.511030    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:26:57.040064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:57.040024    2561 scope.go:117] "RemoveContainer" containerID="6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad"
Jul 17 17:26:57.040494 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:26:57.040250    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg_odh-ai-gateway-infra(d69ebf51-aa6e-4f00-b104-bcd79c288892)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892"
Jul 17 17:26:58.057024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:58.056992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:26:59.296767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.296733    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:26:59.297241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.296946    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" podUID="eb70c163-82b5-4e59-b13b-68d3a1887040" containerName="maas-api" containerID="cri-o://81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a" gracePeriod=30
Jul 17 17:26:59.336606 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.336580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:26:59.541343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.541320    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:26:59.557493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.557446    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jgtl\" (UniqueName: \"kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl\") pod \"eb70c163-82b5-4e59-b13b-68d3a1887040\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") "
Jul 17 17:26:59.557493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.557477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca\") pod \"eb70c163-82b5-4e59-b13b-68d3a1887040\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") "
Jul 17 17:26:59.557640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.557501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls\") pod \"eb70c163-82b5-4e59-b13b-68d3a1887040\" (UID: \"eb70c163-82b5-4e59-b13b-68d3a1887040\") "
Jul 17 17:26:59.557862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.557836    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb70c163-82b5-4e59-b13b-68d3a1887040" (UID: "eb70c163-82b5-4e59-b13b-68d3a1887040"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:26:59.558056 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.558022    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb70c163-82b5-4e59-b13b-68d3a1887040-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:26:59.559640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.559608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb70c163-82b5-4e59-b13b-68d3a1887040" (UID: "eb70c163-82b5-4e59-b13b-68d3a1887040"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:26:59.559737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.559658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl" (OuterVolumeSpecName: "kube-api-access-8jgtl") pod "eb70c163-82b5-4e59-b13b-68d3a1887040" (UID: "eb70c163-82b5-4e59-b13b-68d3a1887040"). InnerVolumeSpecName "kube-api-access-8jgtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:26:59.658498 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.658472    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgtl\" (UniqueName: \"kubernetes.io/projected/eb70c163-82b5-4e59-b13b-68d3a1887040-kube-api-access-8jgtl\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:26:59.658498 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:26:59.658496    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb70c163-82b5-4e59-b13b-68d3a1887040-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:00.054463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.054429    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb70c163-82b5-4e59-b13b-68d3a1887040" containerID="81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a" exitCode=0
Jul 17 17:27:00.054605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.054477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"
Jul 17 17:27:00.054605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.054515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" event={"ID":"eb70c163-82b5-4e59-b13b-68d3a1887040","Type":"ContainerDied","Data":"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"}
Jul 17 17:27:00.054605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.054572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d" event={"ID":"eb70c163-82b5-4e59-b13b-68d3a1887040","Type":"ContainerDied","Data":"c2a312228102026f2c7fcdef87702da76bded8077d9b2e280c9a46bf4652574a"}
Jul 17 17:27:00.054605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.054594    2561 scope.go:117] "RemoveContainer" containerID="81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"
Jul 17 17:27:00.063634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.063616    2561 scope.go:117] "RemoveContainer" containerID="81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"
Jul 17 17:27:00.063878 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:00.063861    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a\": container with ID starting with 81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a not found: ID does not exist" containerID="81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"
Jul 17 17:27:00.063918 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.063888    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a"} err="failed to get container status \"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a\": rpc error: code = NotFound desc = could not find container \"81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a\": container with ID starting with 81e3c36fff46d49c31f1c520a7b3a21ef218898a796b39083941b139a291f43a not found: ID does not exist"
Jul 17 17:27:00.076633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.076612    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:27:00.081064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.081033    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9777fc6b-755797b76f-vhp2d"]
Jul 17 17:27:00.514366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:00.514281    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb70c163-82b5-4e59-b13b-68d3a1887040" path="/var/lib/kubelet/pods/eb70c163-82b5-4e59-b13b-68d3a1887040/volumes"
Jul 17 17:27:02.508999 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:02.508968    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:27:02.509412 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:02.509181    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:27:03.508885 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:03.508852    2561 scope.go:117] "RemoveContainer" containerID="23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e"
Jul 17 17:27:03.876698 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:03.876669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:04.075320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:04.075280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerStarted","Data":"c25767cacb1b4b3fc88afe6c65dd391d5c6ca64c198110b1989170e339da2eea"}
Jul 17 17:27:04.093195 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:04.093150    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" podStartSLOduration=19.093135169 podStartE2EDuration="19.093135169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:27:04.091181928 +0000 UTC m=+3658.076269466" watchObservedRunningTime="2026-07-17 17:27:04.093135169 +0000 UTC m=+3658.078222706"
Jul 17 17:27:05.082239 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:05.082194    2561 generic.go:356] "Generic (PLEG): container finished" podID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerID="c25767cacb1b4b3fc88afe6c65dd391d5c6ca64c198110b1989170e339da2eea" exitCode=7
Jul 17 17:27:05.082645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:05.082252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerDied","Data":"c25767cacb1b4b3fc88afe6c65dd391d5c6ca64c198110b1989170e339da2eea"}
Jul 17 17:27:05.082645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:05.082313    2561 scope.go:117] "RemoveContainer" containerID="23edc4f34d996df426cf809c47661e71e982200d725f3cf5f0b9914fbf55b23e"
Jul 17 17:27:05.082645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:05.082592    2561 scope.go:117] "RemoveContainer" containerID="c25767cacb1b4b3fc88afe6c65dd391d5c6ca64c198110b1989170e339da2eea"
Jul 17 17:27:05.082865 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:05.082844    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-84txk_odh-ai-gateway-infra(0644c1af-de1f-4c77-a701-e89c9c32d0dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc"
Jul 17 17:27:05.108704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:05.108678    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:27:06.080269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.080216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:06.223295 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.223266    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:27:06.313276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.313233    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca\") pod \"0644c1af-de1f-4c77-a701-e89c9c32d0dc\" (UID: \"0644c1af-de1f-4c77-a701-e89c9c32d0dc\") "
Jul 17 17:27:06.313658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.313630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0644c1af-de1f-4c77-a701-e89c9c32d0dc" (UID: "0644c1af-de1f-4c77-a701-e89c9c32d0dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:27:06.360345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.360275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:27:06.414335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:06.414296    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0644c1af-de1f-4c77-a701-e89c9c32d0dc-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:07.094391 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:07.094362    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"
Jul 17 17:27:07.094550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:07.094386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk" event={"ID":"0644c1af-de1f-4c77-a701-e89c9c32d0dc","Type":"ContainerDied","Data":"105a69529d4c12819d52b0efb89d6f94e3d0654f92a377ff1dc79bc40676fc72"}
Jul 17 17:27:07.094550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:07.094451    2561 scope.go:117] "RemoveContainer" containerID="c25767cacb1b4b3fc88afe6c65dd391d5c6ca64c198110b1989170e339da2eea"
Jul 17 17:27:07.110968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:07.110943    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:27:07.117652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:07.117622    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-84txk"]
Jul 17 17:27:08.509116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:08.509079    2561 scope.go:117] "RemoveContainer" containerID="6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad"
Jul 17 17:27:08.509612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:08.509249    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:27:08.509612 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:08.509445    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:27:08.514789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:08.514767    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" path="/var/lib/kubelet/pods/0644c1af-de1f-4c77-a701-e89c9c32d0dc/volumes"
Jul 17 17:27:09.105587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:09.105554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerStarted","Data":"d4a1f15601bcb646dc9aa92f3a6d5e3d2731ab6137353bc03d19846c92f7b7ee"}
Jul 17 17:27:09.120548 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:09.120499    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" podStartSLOduration=16.120481638 podStartE2EDuration="16.120481638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:27:09.119889233 +0000 UTC m=+3663.104976768" watchObservedRunningTime="2026-07-17 17:27:09.120481638 +0000 UTC m=+3663.105569178"
Jul 17 17:27:10.111676 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:10.111642    2561 generic.go:356] "Generic (PLEG): container finished" podID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerID="d4a1f15601bcb646dc9aa92f3a6d5e3d2731ab6137353bc03d19846c92f7b7ee" exitCode=7
Jul 17 17:27:10.112178 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:10.111719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerDied","Data":"d4a1f15601bcb646dc9aa92f3a6d5e3d2731ab6137353bc03d19846c92f7b7ee"}
Jul 17 17:27:10.112178 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:10.111792    2561 scope.go:117] "RemoveContainer" containerID="6f56a80cd4cf83a3f619b18a738562601bdacfce621ce1aabe9d8fac21eb89ad"
Jul 17 17:27:10.112178 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:10.112141    2561 scope.go:117] "RemoveContainer" containerID="d4a1f15601bcb646dc9aa92f3a6d5e3d2731ab6137353bc03d19846c92f7b7ee"
Jul 17 17:27:10.112360 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:10.112342    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg_odh-ai-gateway-infra(d69ebf51-aa6e-4f00-b104-bcd79c288892)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892"
Jul 17 17:27:10.136327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:10.136302    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:27:11.052073 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.052026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:27:11.095805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.095773    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:11.245688 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.245665    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:27:11.357546 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.357471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca\") pod \"d69ebf51-aa6e-4f00-b104-bcd79c288892\" (UID: \"d69ebf51-aa6e-4f00-b104-bcd79c288892\") "
Jul 17 17:27:11.357829 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.357805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d69ebf51-aa6e-4f00-b104-bcd79c288892" (UID: "d69ebf51-aa6e-4f00-b104-bcd79c288892"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:27:11.458881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:11.458839    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69ebf51-aa6e-4f00-b104-bcd79c288892-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:12.124948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.124910    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"
Jul 17 17:27:12.124948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.124940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg" event={"ID":"d69ebf51-aa6e-4f00-b104-bcd79c288892","Type":"ContainerDied","Data":"4851da0fa5b93675ded9379e3f38c516ec542c0c1abc4e2212cc62ec235165e9"}
Jul 17 17:27:12.125172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.125005    2561 scope.go:117] "RemoveContainer" containerID="d4a1f15601bcb646dc9aa92f3a6d5e3d2731ab6137353bc03d19846c92f7b7ee"
Jul 17 17:27:12.147626 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.147604    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:27:12.151382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.151359    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-p2xmg"]
Jul 17 17:27:12.513774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:12.513694    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" path="/var/lib/kubelet/pods/d69ebf51-aa6e-4f00-b104-bcd79c288892/volumes"
Jul 17 17:27:16.510369 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:16.510335    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:27:16.510843 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:16.510537    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:27:20.509149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:20.509071    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:27:20.509528 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:20.509257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:27:26.072204 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.072139    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" podUID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" containerName="main" containerID="cri-o://f2a4bd091fd40fb2acd87465696034391d4015024559ef2ed5ff89348db7984a" gracePeriod=45
Jul 17 17:27:26.198061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.198016    2561 generic.go:356] "Generic (PLEG): container finished" podID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" containerID="f2a4bd091fd40fb2acd87465696034391d4015024559ef2ed5ff89348db7984a" exitCode=0
Jul 17 17:27:26.198180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.198096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" event={"ID":"9608a8a4-dbca-4a4d-9985-559653b6f3e4","Type":"ContainerDied","Data":"f2a4bd091fd40fb2acd87465696034391d4015024559ef2ed5ff89348db7984a"}
Jul 17 17:27:26.320024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.320002    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:27:26.392064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.391980    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrp2t\" (UniqueName: \"kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm\") pod \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\" (UID: \"9608a8a4-dbca-4a4d-9985-559653b6f3e4\") "
Jul 17 17:27:26.392488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:27:26.392488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:27:26.392488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home" (OuterVolumeSpecName: "home") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:27:26.392488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.392459    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:26.394302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.394278    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t" (OuterVolumeSpecName: "kube-api-access-jrp2t") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "kube-api-access-jrp2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:27:26.394302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.394297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:27:26.394464 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.394295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm" (OuterVolumeSpecName: "dshm") pod "9608a8a4-dbca-4a4d-9985-559653b6f3e4" (UID: "9608a8a4-dbca-4a4d-9985-559653b6f3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:27:26.493415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.493393    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:26.493415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.493416    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9608a8a4-dbca-4a4d-9985-559653b6f3e4-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:26.493563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.493426    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:26.493563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.493435    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrp2t\" (UniqueName: \"kubernetes.io/projected/9608a8a4-dbca-4a4d-9985-559653b6f3e4-kube-api-access-jrp2t\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:26.493563 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:26.493444    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9608a8a4-dbca-4a4d-9985-559653b6f3e4-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:27:27.203479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.203442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6" event={"ID":"9608a8a4-dbca-4a4d-9985-559653b6f3e4","Type":"ContainerDied","Data":"e5390523335bcb248de8b4402c2363118794489adc7b3db93363cfe7956aced0"}
Jul 17 17:27:27.203479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.203473    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"
Jul 17 17:27:27.204151 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.203497    2561 scope.go:117] "RemoveContainer" containerID="f2a4bd091fd40fb2acd87465696034391d4015024559ef2ed5ff89348db7984a"
Jul 17 17:27:27.220913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.220882    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:27:27.222727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.222708    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9777fc6b/e2e-lifecycle-model-9777fc6b-kserve-f9b66867f-75rm6"]
Jul 17 17:27:27.508180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:27.508095    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:27:27.508342 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:27.508277    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:27:28.513768 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:28.513670    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" path="/var/lib/kubelet/pods/9608a8a4-dbca-4a4d-9985-559653b6f3e4/volumes"
Jul 17 17:27:32.508592 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:32.508562    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:27:32.508985 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:32.508744    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:27:40.508283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:40.508246    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:27:40.508802 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:40.508448    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:27:43.509155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.509121    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:27:43.509548 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:43.509279    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:27:43.732177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732143    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:27:43.732573 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732555    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb70c163-82b5-4e59-b13b-68d3a1887040" containerName="maas-api"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732575    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb70c163-82b5-4e59-b13b-68d3a1887040" containerName="maas-api"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732598    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732611    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732631    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732639    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732666    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" containerName="main"
Jul 17 17:27:43.732672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732675    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" containerName="main"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732685    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732692    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732796    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb70c163-82b5-4e59-b13b-68d3a1887040" containerName="maas-api"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732810    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732818    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9608a8a4-dbca-4a4d-9985-559653b6f3e4" containerName="main"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732828    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732837    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:43.733132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.732846    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:43.737634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.737615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:27:43.741677 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.741364    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:27:43.748604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.748578    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:27:43.753531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.753509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn\" (UID: \"d51d4e7d-766d-4305-aaa8-962173a8c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:27:43.854421 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.854394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn\" (UID: \"d51d4e7d-766d-4305-aaa8-962173a8c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:27:43.854962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:43.854942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn\" (UID: \"d51d4e7d-766d-4305-aaa8-962173a8c0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:27:44.057568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:44.057539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:27:44.186536 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:44.186508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:27:44.189273 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:27:44.189243    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51d4e7d_766d_4305_aaa8_962173a8c0aa.slice/crio-26049d44886d9a66c53198c33a4bba7b88f727104717913e7195ad19988a95e4 WatchSource:0}: Error finding container 26049d44886d9a66c53198c33a4bba7b88f727104717913e7195ad19988a95e4: Status 404 returned error can't find the container with id 26049d44886d9a66c53198c33a4bba7b88f727104717913e7195ad19988a95e4
Jul 17 17:27:44.281786 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:44.281754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" event={"ID":"d51d4e7d-766d-4305-aaa8-962173a8c0aa","Type":"ContainerStarted","Data":"26049d44886d9a66c53198c33a4bba7b88f727104717913e7195ad19988a95e4"}
Jul 17 17:27:45.287718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.287684    2561 generic.go:356] "Generic (PLEG): container finished" podID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerID="6d405e748c1d6db0a2093503f2ea9ee79b9049db20b0ef7537884f8bc8de769d" exitCode=7
Jul 17 17:27:45.288320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.287774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" event={"ID":"d51d4e7d-766d-4305-aaa8-962173a8c0aa","Type":"ContainerDied","Data":"6d405e748c1d6db0a2093503f2ea9ee79b9049db20b0ef7537884f8bc8de769d"}
Jul 17 17:27:45.288320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.287983    2561 scope.go:117] "RemoveContainer" containerID="6d405e748c1d6db0a2093503f2ea9ee79b9049db20b0ef7537884f8bc8de769d"
Jul 17 17:27:45.819823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.819792    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:27:45.820186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820173    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:45.820233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820188    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:45.820233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820210    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:45.820233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820216    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:45.820335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820269    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69ebf51-aa6e-4f00-b104-bcd79c288892" containerName="revoke-keys"
Jul 17 17:27:45.820335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.820277    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0644c1af-de1f-4c77-a701-e89c9c32d0dc" containerName="revoke-keys"
Jul 17 17:27:45.823236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.823222    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:27:45.828900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.828874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:27:45.870820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.870747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-l9bwh\" (UID: \"e311e2cb-ef23-4522-967f-d2141f833af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:27:45.971663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.971640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-l9bwh\" (UID: \"e311e2cb-ef23-4522-967f-d2141f833af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:27:45.972270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:45.972236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-l9bwh\" (UID: \"e311e2cb-ef23-4522-967f-d2141f833af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:27:46.135308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.135246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:27:46.279600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.279571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:27:46.281087 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:27:46.281029    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode311e2cb_ef23_4522_967f_d2141f833af3.slice/crio-604a54f9ee1be06dca08e987d8801b2d4ae8ba92937818384a879e878647ea7d WatchSource:0}: Error finding container 604a54f9ee1be06dca08e987d8801b2d4ae8ba92937818384a879e878647ea7d: Status 404 returned error can't find the container with id 604a54f9ee1be06dca08e987d8801b2d4ae8ba92937818384a879e878647ea7d
Jul 17 17:27:46.293428 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.293402    2561 generic.go:356] "Generic (PLEG): container finished" podID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerID="6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90" exitCode=7
Jul 17 17:27:46.293772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.293491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" event={"ID":"d51d4e7d-766d-4305-aaa8-962173a8c0aa","Type":"ContainerDied","Data":"6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90"}
Jul 17 17:27:46.293772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.293564    2561 scope.go:117] "RemoveContainer" containerID="6d405e748c1d6db0a2093503f2ea9ee79b9049db20b0ef7537884f8bc8de769d"
Jul 17 17:27:46.293772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.293750    2561 scope.go:117] "RemoveContainer" containerID="6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90"
Jul 17 17:27:46.294005 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:46.293974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn_odh-ai-gateway-infra(d51d4e7d-766d-4305-aaa8-962173a8c0aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa"
Jul 17 17:27:46.294962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:46.294728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" event={"ID":"e311e2cb-ef23-4522-967f-d2141f833af3","Type":"ContainerStarted","Data":"604a54f9ee1be06dca08e987d8801b2d4ae8ba92937818384a879e878647ea7d"}
Jul 17 17:27:47.300523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:47.300494    2561 scope.go:117] "RemoveContainer" containerID="6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90"
Jul 17 17:27:47.301039 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:47.300769    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn_odh-ai-gateway-infra(d51d4e7d-766d-4305-aaa8-962173a8c0aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa"
Jul 17 17:27:47.302138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:47.302113    2561 generic.go:356] "Generic (PLEG): container finished" podID="e311e2cb-ef23-4522-967f-d2141f833af3" containerID="d272022d8db8535a81ec4128671563fcaa973f322fd978552c0313cb38fda743" exitCode=7
Jul 17 17:27:47.302263 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:47.302188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" event={"ID":"e311e2cb-ef23-4522-967f-d2141f833af3","Type":"ContainerDied","Data":"d272022d8db8535a81ec4128671563fcaa973f322fd978552c0313cb38fda743"}
Jul 17 17:27:47.302336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:47.302319    2561 scope.go:117] "RemoveContainer" containerID="d272022d8db8535a81ec4128671563fcaa973f322fd978552c0313cb38fda743"
Jul 17 17:27:47.738487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:47.738454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:48.197744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:48.197659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:48.308881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:48.308850    2561 generic.go:356] "Generic (PLEG): container finished" podID="e311e2cb-ef23-4522-967f-d2141f833af3" containerID="11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4" exitCode=7
Jul 17 17:27:48.309292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:48.308947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" event={"ID":"e311e2cb-ef23-4522-967f-d2141f833af3","Type":"ContainerDied","Data":"11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4"}
Jul 17 17:27:48.309292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:48.309037    2561 scope.go:117] "RemoveContainer" containerID="d272022d8db8535a81ec4128671563fcaa973f322fd978552c0313cb38fda743"
Jul 17 17:27:48.309292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:48.309178    2561 scope.go:117] "RemoveContainer" containerID="11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4"
Jul 17 17:27:48.309421 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:48.309386    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-l9bwh_odh-ai-gateway-infra(e311e2cb-ef23-4522-967f-d2141f833af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" podUID="e311e2cb-ef23-4522-967f-d2141f833af3"
Jul 17 17:27:49.314580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:49.314552    2561 scope.go:117] "RemoveContainer" containerID="11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4"
Jul 17 17:27:49.314957 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:49.314747    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-l9bwh_odh-ai-gateway-infra(e311e2cb-ef23-4522-967f-d2141f833af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" podUID="e311e2cb-ef23-4522-967f-d2141f833af3"
Jul 17 17:27:53.917224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:53.917188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:54.016613 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:54.016583    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:54.202389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:54.202308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:54.438266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:54.438233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:55.484526 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:55.484484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:55.508627 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:55.508607    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:27:55.508797 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:55.508778    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:27:55.581141 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:55.581091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:55.716806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:55.716775    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:56.382333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:56.382275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:56.471055 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:56.471011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:56.510674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:56.510653    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:27:56.511062 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:27:56.510849    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:27:56.570964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:56.570942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:57.883449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:57.883394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:27:58.465360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:27:58.465323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:28:00.509025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:00.508992    2561 scope.go:117] "RemoveContainer" containerID="6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90"
Jul 17 17:28:01.370994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:01.370912    2561 generic.go:356] "Generic (PLEG): container finished" podID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerID="1a07d969c7bf3209263becc8227d70e5688aa90961b7bbc9fbffb5479749a6c1" exitCode=7
Jul 17 17:28:01.370994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:01.370985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" event={"ID":"d51d4e7d-766d-4305-aaa8-962173a8c0aa","Type":"ContainerDied","Data":"1a07d969c7bf3209263becc8227d70e5688aa90961b7bbc9fbffb5479749a6c1"}
Jul 17 17:28:01.371214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:01.371052    2561 scope.go:117] "RemoveContainer" containerID="6d0d100b457362a0aa84dec15fcb8644e63cdc04440ea5a2060682e510e6be90"
Jul 17 17:28:01.371314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:01.371297    2561 scope.go:117] "RemoveContainer" containerID="1a07d969c7bf3209263becc8227d70e5688aa90961b7bbc9fbffb5479749a6c1"
Jul 17 17:28:01.371524 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:01.371505    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn_odh-ai-gateway-infra(d51d4e7d-766d-4305-aaa8-962173a8c0aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa"
Jul 17 17:28:02.395165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.395133    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:28:02.508906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.508884    2561 scope.go:117] "RemoveContainer" containerID="11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4"
Jul 17 17:28:02.522745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.522725    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:28:02.621685 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.621654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca\") pod \"d51d4e7d-766d-4305-aaa8-962173a8c0aa\" (UID: \"d51d4e7d-766d-4305-aaa8-962173a8c0aa\") "
Jul 17 17:28:02.622039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.622020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d51d4e7d-766d-4305-aaa8-962173a8c0aa" (UID: "d51d4e7d-766d-4305-aaa8-962173a8c0aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:28:02.723350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:02.723289    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d51d4e7d-766d-4305-aaa8-962173a8c0aa-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:28:03.386090 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.385988    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"
Jul 17 17:28:03.386090 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.385988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn" event={"ID":"d51d4e7d-766d-4305-aaa8-962173a8c0aa","Type":"ContainerDied","Data":"26049d44886d9a66c53198c33a4bba7b88f727104717913e7195ad19988a95e4"}
Jul 17 17:28:03.386305 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.386125    2561 scope.go:117] "RemoveContainer" containerID="1a07d969c7bf3209263becc8227d70e5688aa90961b7bbc9fbffb5479749a6c1"
Jul 17 17:28:03.388012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.387987    2561 generic.go:356] "Generic (PLEG): container finished" podID="e311e2cb-ef23-4522-967f-d2141f833af3" containerID="a8f43cf0c8abe5915e1508fbe9563dc000e9869128b6fcdd3174a514b7be1a9a" exitCode=7
Jul 17 17:28:03.388139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.388069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" event={"ID":"e311e2cb-ef23-4522-967f-d2141f833af3","Type":"ContainerDied","Data":"a8f43cf0c8abe5915e1508fbe9563dc000e9869128b6fcdd3174a514b7be1a9a"}
Jul 17 17:28:03.388432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.388415    2561 scope.go:117] "RemoveContainer" containerID="a8f43cf0c8abe5915e1508fbe9563dc000e9869128b6fcdd3174a514b7be1a9a"
Jul 17 17:28:03.388688 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:03.388666    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-l9bwh_odh-ai-gateway-infra(e311e2cb-ef23-4522-967f-d2141f833af3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" podUID="e311e2cb-ef23-4522-967f-d2141f833af3"
Jul 17 17:28:03.395287 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.395271    2561 scope.go:117] "RemoveContainer" containerID="11146ead9d00cc17d1a0309a97f3af715a5b45e157494a1d210184c9c2696ec4"
Jul 17 17:28:03.415103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.415079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:28:03.419254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.419235    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-s6lxn"]
Jul 17 17:28:03.914999 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:03.914966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:04.015147 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.015105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:04.412849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.412818    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:28:04.514336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.514304    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" path="/var/lib/kubelet/pods/d51d4e7d-766d-4305-aaa8-962173a8c0aa/volumes"
Jul 17 17:28:04.545487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.545464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:28:04.634849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.634820    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:04.641826 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.641789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca\") pod \"e311e2cb-ef23-4522-967f-d2141f833af3\" (UID: \"e311e2cb-ef23-4522-967f-d2141f833af3\") "
Jul 17 17:28:04.642310 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.642280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e311e2cb-ef23-4522-967f-d2141f833af3" (UID: "e311e2cb-ef23-4522-967f-d2141f833af3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:28:04.743064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:04.742963    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e311e2cb-ef23-4522-967f-d2141f833af3-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:28:05.400264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:05.400232    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"
Jul 17 17:28:05.400415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:05.400263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh" event={"ID":"e311e2cb-ef23-4522-967f-d2141f833af3","Type":"ContainerDied","Data":"604a54f9ee1be06dca08e987d8801b2d4ae8ba92937818384a879e878647ea7d"}
Jul 17 17:28:05.400415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:05.400318    2561 scope.go:117] "RemoveContainer" containerID="a8f43cf0c8abe5915e1508fbe9563dc000e9869128b6fcdd3174a514b7be1a9a"
Jul 17 17:28:05.420841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:05.420816    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:28:05.426131 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:05.426109    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-l9bwh"]
Jul 17 17:28:06.515092 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:06.515059    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" path="/var/lib/kubelet/pods/e311e2cb-ef23-4522-967f-d2141f833af3/volumes"
Jul 17 17:28:10.463292 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:10.463251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:28:10.465375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:10.465341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:10.508922 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:10.508887    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:28:10.509183 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:10.509099    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:10.577937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:10.577902    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:10.667849 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:10.667812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:11.508761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:11.508730    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:28:11.509261 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:11.508958    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:28:21.240751 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:21.240715    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:21.508365 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:21.508297    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:28:21.508510 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:21.508449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:21.512453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:21.512415    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:22.509334 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:22.509306    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:28:22.509710 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:22.509521    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:28:24.119098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:24.119064    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:34.508724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:34.508692    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:28:34.509162 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:34.509115    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:28:35.508582 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:35.508560    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:28:35.508747 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:35.508729    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:28:35.539177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:35.539149    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158" exitCode=1
Jul 17 17:28:35.539272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:35.539221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"}
Jul 17 17:28:35.539314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:35.539272    2561 scope.go:117] "RemoveContainer" containerID="19e80a8194b1670cbc0ab7f0cb9b5f802ec632ba9a27e096dfb06f7bbb49c582"
Jul 17 17:28:35.539545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:35.539524    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:28:35.539705 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:35.539689    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:37.508321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:37.508294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:28:37.508797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:37.508664    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:28:37.508901 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:37.508873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:40.175244 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:40.175208    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:28:40.175658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:40.175597    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:28:40.175803 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:40.175777    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:42.487522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:42.487487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:46.511240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:46.511215    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:28:46.511633 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:46.511384    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:28:48.912637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:48.912599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:49.012983 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:49.012938    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:49.279385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:49.279343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:49.609841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:49.609739    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:49.865968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:49.865841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:51.318906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.318869    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319386    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319408    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319424    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319433    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319458    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319466    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319502    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319587    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319601    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:51.319911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319612    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319624    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.319911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.319635    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:51.323862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.323840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:28:51.325906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.325888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:28:51.329875 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.329856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:28:51.449538 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.449511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn\" (UID: \"bfa5318d-e562-4050-9be7-882b44850194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:28:51.508494 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.508471    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:28:51.508635 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:51.508617    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:28:51.550762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.550743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn\" (UID: \"bfa5318d-e562-4050-9be7-882b44850194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:28:51.551315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.551298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn\" (UID: \"bfa5318d-e562-4050-9be7-882b44850194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:28:51.636254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.636195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:28:51.765121 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:51.765098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:28:51.767172 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:28:51.767144    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa5318d_e562_4050_9be7_882b44850194.slice/crio-552f4aa1c57cb7cbf96fa60bc11e71d937fa13b7eb79db85a74a7b45695fccbe WatchSource:0}: Error finding container 552f4aa1c57cb7cbf96fa60bc11e71d937fa13b7eb79db85a74a7b45695fccbe: Status 404 returned error can't find the container with id 552f4aa1c57cb7cbf96fa60bc11e71d937fa13b7eb79db85a74a7b45695fccbe
Jul 17 17:28:52.621096 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:52.621065    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfa5318d-e562-4050-9be7-882b44850194" containerID="56b602ccf2ecb46b8f2f983b92d4930df91c80158ce4e57f2acd673b35d353e0" exitCode=7
Jul 17 17:28:52.621491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:52.621115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" event={"ID":"bfa5318d-e562-4050-9be7-882b44850194","Type":"ContainerDied","Data":"56b602ccf2ecb46b8f2f983b92d4930df91c80158ce4e57f2acd673b35d353e0"}
Jul 17 17:28:52.621491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:52.621157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" event={"ID":"bfa5318d-e562-4050-9be7-882b44850194","Type":"ContainerStarted","Data":"552f4aa1c57cb7cbf96fa60bc11e71d937fa13b7eb79db85a74a7b45695fccbe"}
Jul 17 17:28:52.621491 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:52.621357    2561 scope.go:117] "RemoveContainer" containerID="56b602ccf2ecb46b8f2f983b92d4930df91c80158ce4e57f2acd673b35d353e0"
Jul 17 17:28:52.943942 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:52.943862    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:53.020141 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:28:53.020551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020533    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:53.020648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020554    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311e2cb-ef23-4522-967f-d2141f833af3" containerName="revoke-keys"
Jul 17 17:28:53.020648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020568    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:53.020648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020576    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:53.020799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.020651    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51d4e7d-766d-4305-aaa8-962173a8c0aa" containerName="revoke-keys"
Jul 17 17:28:53.023771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.023749    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.026118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.026093    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7ca540-serving-cert\""
Jul 17 17:28:53.035824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.035797    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:28:53.047184 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.047156    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:53.122704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.122671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:53.162897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.162865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65ll\" (UniqueName: \"kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.163038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.162914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.163038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.162997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.230847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.230764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:53.264593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.264561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.264729 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.264671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65ll\" (UniqueName: \"kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.264771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.264735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.265202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.265181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.267015 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.266995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.272616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.272590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65ll\" (UniqueName: \"kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll\") pod \"maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.336254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.336227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:53.464154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.464114    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:28:53.466321 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:28:53.466291    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc197c441_ad92_485a_bb85_76c11377c967.slice/crio-c35fee7f515ff18002f3af733b5eb801732e701b79acafc1347755fb7aefacbf WatchSource:0}: Error finding container c35fee7f515ff18002f3af733b5eb801732e701b79acafc1347755fb7aefacbf: Status 404 returned error can't find the container with id c35fee7f515ff18002f3af733b5eb801732e701b79acafc1347755fb7aefacbf
Jul 17 17:28:53.626861 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.626812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" event={"ID":"c197c441-ad92-485a-bb85-76c11377c967","Type":"ContainerStarted","Data":"c35fee7f515ff18002f3af733b5eb801732e701b79acafc1347755fb7aefacbf"}
Jul 17 17:28:53.627876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.627844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:53.628708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.628685    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfa5318d-e562-4050-9be7-882b44850194" containerID="e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9" exitCode=7
Jul 17 17:28:53.628827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.628732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" event={"ID":"bfa5318d-e562-4050-9be7-882b44850194","Type":"ContainerDied","Data":"e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9"}
Jul 17 17:28:53.628827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.628784    2561 scope.go:117] "RemoveContainer" containerID="56b602ccf2ecb46b8f2f983b92d4930df91c80158ce4e57f2acd673b35d353e0"
Jul 17 17:28:53.628971 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.628952    2561 scope.go:117] "RemoveContainer" containerID="e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9"
Jul 17 17:28:53.629240 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:53.629223    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn_odh-ai-gateway-infra(bfa5318d-e562-4050-9be7-882b44850194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" podUID="bfa5318d-e562-4050-9be7-882b44850194"
Jul 17 17:28:53.726166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:53.726138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:54.635394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:54.635368    2561 scope.go:117] "RemoveContainer" containerID="e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9"
Jul 17 17:28:54.635829 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:54.635584    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn_odh-ai-gateway-infra(bfa5318d-e562-4050-9be7-882b44850194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" podUID="bfa5318d-e562-4050-9be7-882b44850194"
Jul 17 17:28:54.636749 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:54.636720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" event={"ID":"c197c441-ad92-485a-bb85-76c11377c967","Type":"ContainerStarted","Data":"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"}
Jul 17 17:28:54.636866 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:54.636772    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:28:54.669877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:54.669829    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" podStartSLOduration=1.324259847 podStartE2EDuration="1.669813704s" totalImagesPullingTime="345.553857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:28:54.668545855 +0000 UTC m=+3768.653633394" watchObservedRunningTime="2026-07-17 17:28:54.669813704 +0000 UTC m=+3768.654901242"
Jul 17 17:28:57.429991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:57.429956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:57.508636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:57.508605    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:28:57.508817 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:28:57.508795    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:28:57.769516 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:57.769442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:58.070385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:58.070339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:28:58.281836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:28:58.281803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:00.646974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:00.646944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:29:00.666997 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:00.666955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:29:00.960830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:00.960746    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:01.048695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:01.048652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:01.119676 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:01.119642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:01.220826 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:01.220744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:01.615438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:01.615410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:01.722433 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:01.722404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:03.508648 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:03.508614    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:29:03.509145 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:03.508837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:29:05.459100 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:05.459068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:06.510776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:06.510750    2561 scope.go:117] "RemoveContainer" containerID="e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9"
Jul 17 17:29:07.698208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:07.698170    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfa5318d-e562-4050-9be7-882b44850194" containerID="f20065d68b8742c6afbbc64a44ce2bc5894cad6c807def62dd75dd6673e48255" exitCode=7
Jul 17 17:29:07.698609 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:07.698245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" event={"ID":"bfa5318d-e562-4050-9be7-882b44850194","Type":"ContainerDied","Data":"f20065d68b8742c6afbbc64a44ce2bc5894cad6c807def62dd75dd6673e48255"}
Jul 17 17:29:07.698609 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:07.698332    2561 scope.go:117] "RemoveContainer" containerID="e4fcc2abeb1563cec4ff92158169d31ddae2b237c1b69bc4a205747dc80a18e9"
Jul 17 17:29:07.698737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:07.698643    2561 scope.go:117] "RemoveContainer" containerID="f20065d68b8742c6afbbc64a44ce2bc5894cad6c807def62dd75dd6673e48255"
Jul 17 17:29:07.698943 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:07.698917    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn_odh-ai-gateway-infra(bfa5318d-e562-4050-9be7-882b44850194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" podUID="bfa5318d-e562-4050-9be7-882b44850194"
Jul 17 17:29:08.725547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.725515    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:29:08.856729 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.856704    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:29:08.995748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.995672    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:29:08.996037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996025    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996065    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996074    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996096    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996154    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996161    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.996350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.996168    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa5318d-e562-4050-9be7-882b44850194" containerName="revoke-keys"
Jul 17 17:29:08.999053 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:08.999026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.001229 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.001207    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ca540\"/\"test-model-7ca540-kserve-self-signed-certs\""
Jul 17 17:29:09.001314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.001287    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ca540\"/\"kube-root-ca.crt\""
Jul 17 17:29:09.001388 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.001375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ca540\"/\"default-dockercfg-qjthj\""
Jul 17 17:29:09.001469 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.001455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ca540\"/\"openshift-service-ca.crt\""
Jul 17 17:29:09.007521 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.007496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:29:09.014804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.014783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca\") pod \"bfa5318d-e562-4050-9be7-882b44850194\" (UID: \"bfa5318d-e562-4050-9be7-882b44850194\") "
Jul 17 17:29:09.015133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.015111    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfa5318d-e562-4050-9be7-882b44850194" (UID: "bfa5318d-e562-4050-9be7-882b44850194"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:29:09.088107 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.088071    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:09.116298 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxg7s\" (UniqueName: \"kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.116615 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.116557    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa5318d-e562-4050-9be7-882b44850194-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:29:09.217396 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxg7s\" (UniqueName: \"kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.217979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.217884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.219780 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.219754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.219977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.219962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.225495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.225474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxg7s\" (UniqueName: \"kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s\") pod \"test-model-7ca540-kserve-5b99dd69df-hr2lk\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") " pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.309249 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.309224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:09.440083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.440034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:29:09.441781 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:29:09.441756    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod341c2d7e_76c5_407d_814b_e00b37879ea8.slice/crio-2807600fa3d04142353e9bf9886475748b7dae7bc95777f58ff1855194feda28 WatchSource:0}: Error finding container 2807600fa3d04142353e9bf9886475748b7dae7bc95777f58ff1855194feda28: Status 404 returned error can't find the container with id 2807600fa3d04142353e9bf9886475748b7dae7bc95777f58ff1855194feda28
Jul 17 17:29:09.710666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.710578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" event={"ID":"341c2d7e-76c5-407d-814b-e00b37879ea8","Type":"ContainerStarted","Data":"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"}
Jul 17 17:29:09.710666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.710627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" event={"ID":"341c2d7e-76c5-407d-814b-e00b37879ea8","Type":"ContainerStarted","Data":"2807600fa3d04142353e9bf9886475748b7dae7bc95777f58ff1855194feda28"}
Jul 17 17:29:09.711793 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.711770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn" event={"ID":"bfa5318d-e562-4050-9be7-882b44850194","Type":"ContainerDied","Data":"552f4aa1c57cb7cbf96fa60bc11e71d937fa13b7eb79db85a74a7b45695fccbe"}
Jul 17 17:29:09.711793 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.711783    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"
Jul 17 17:29:09.711933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.711818    2561 scope.go:117] "RemoveContainer" containerID="f20065d68b8742c6afbbc64a44ce2bc5894cad6c807def62dd75dd6673e48255"
Jul 17 17:29:09.728259 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.728218    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" podStartSLOduration=1.7282042309999999 podStartE2EDuration="1.728204231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:09.726442654 +0000 UTC m=+3783.711530189" watchObservedRunningTime="2026-07-17 17:29:09.728204231 +0000 UTC m=+3783.713291799"
Jul 17 17:29:09.741360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.741336    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:29:09.747321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:09.747299    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-5h7cn"]
Jul 17 17:29:10.513974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:10.513943    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa5318d-e562-4050-9be7-882b44850194" path="/var/lib/kubelet/pods/bfa5318d-e562-4050-9be7-882b44850194/volumes"
Jul 17 17:29:11.509277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:11.509247    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:29:11.509670 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:11.509390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:29:14.413743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:14.413702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:14.508979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:14.508942    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:29:14.509200 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:14.509182    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:29:14.514453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:14.514416    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:16.398865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.398801    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:16.404392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.404369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:16.406777 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.406753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:29:16.409355 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.409328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:16.461457 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.461425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:29:16.481378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.481353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q\" (UID: \"04596f3f-d762-4764-844d-7a0555ac0cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:16.582072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.582014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q\" (UID: \"04596f3f-d762-4764-844d-7a0555ac0cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:16.582659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.582640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q\" (UID: \"04596f3f-d762-4764-844d-7a0555ac0cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:16.717536 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.717451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:16.846840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:16.846818    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:16.848743 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:29:16.848718    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04596f3f_d762_4764_844d_7a0555ac0cda.slice/crio-7054dd0967d5ede759f151f8dbf87619866e60df2ce6f0c75366c03a2c789980 WatchSource:0}: Error finding container 7054dd0967d5ede759f151f8dbf87619866e60df2ce6f0c75366c03a2c789980: Status 404 returned error can't find the container with id 7054dd0967d5ede759f151f8dbf87619866e60df2ce6f0c75366c03a2c789980
Jul 17 17:29:17.751859 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:17.751775    2561 generic.go:356] "Generic (PLEG): container finished" podID="04596f3f-d762-4764-844d-7a0555ac0cda" containerID="c3e53e2b701531b62b3671c5f63974ce91f9ede2517b3c96bf33ecddcd416bb7" exitCode=7
Jul 17 17:29:17.751859 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:17.751841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerDied","Data":"c3e53e2b701531b62b3671c5f63974ce91f9ede2517b3c96bf33ecddcd416bb7"}
Jul 17 17:29:17.752297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:17.751922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerStarted","Data":"7054dd0967d5ede759f151f8dbf87619866e60df2ce6f0c75366c03a2c789980"}
Jul 17 17:29:17.752297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:17.752176    2561 scope.go:117] "RemoveContainer" containerID="c3e53e2b701531b62b3671c5f63974ce91f9ede2517b3c96bf33ecddcd416bb7"
Jul 17 17:29:18.759492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:18.759397    2561 generic.go:356] "Generic (PLEG): container finished" podID="04596f3f-d762-4764-844d-7a0555ac0cda" containerID="47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024" exitCode=7
Jul 17 17:29:18.759843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:18.759495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerDied","Data":"47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024"}
Jul 17 17:29:18.759843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:18.759548    2561 scope.go:117] "RemoveContainer" containerID="c3e53e2b701531b62b3671c5f63974ce91f9ede2517b3c96bf33ecddcd416bb7"
Jul 17 17:29:18.759843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:18.759765    2561 scope.go:117] "RemoveContainer" containerID="47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024"
Jul 17 17:29:18.760036 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:18.760014    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q_odh-ai-gateway-infra(04596f3f-d762-4764-844d-7a0555ac0cda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" podUID="04596f3f-d762-4764-844d-7a0555ac0cda"
Jul 17 17:29:19.309604 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:19.309567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:19.309804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:19.309622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:19.322032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:19.322006    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:19.766782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:19.766699    2561 scope.go:117] "RemoveContainer" containerID="47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024"
Jul 17 17:29:19.767206 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:19.766886    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q_odh-ai-gateway-infra(04596f3f-d762-4764-844d-7a0555ac0cda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" podUID="04596f3f-d762-4764-844d-7a0555ac0cda"
Jul 17 17:29:19.777514 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:19.777492    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:29:22.512691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:22.512660    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:29:22.513283 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:22.512924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:29:22.652816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:22.652778    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:22.753488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:22.753457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:23.401920 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.401885    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:29:23.405819 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.405789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.408213 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.408189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f748a7\"/\"openshift-service-ca.crt\""
Jul 17 17:29:23.409022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.409004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f748a7\"/\"kube-root-ca.crt\""
Jul 17 17:29:23.409164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.409039    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f748a7\"/\"test-model-f748a7-kserve-self-signed-certs\""
Jul 17 17:29:23.409164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.409115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f748a7\"/\"default-dockercfg-8wh9v\""
Jul 17 17:29:23.416004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.415983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:29:23.427523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.427490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:23.446933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.446908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.447061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.446940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.447061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.446971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.447148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.447034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrgw\" (UniqueName: \"kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.447148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.447095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.447148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.447124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548144 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrgw\" (UniqueName: \"kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.548868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.548723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.550512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.550490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.550767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.550753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.556470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.556451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrgw\" (UniqueName: \"kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw\") pod \"test-model-f748a7-kserve-59cdd497ff-gkcs9\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") " pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.715559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.715460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:23.848868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:23.848842    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:29:23.851628 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:29:23.851589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36623b7_776e_4c85_ad26_63f7c057a64c.slice/crio-032f63201c023fb5b3d8cdcab9a10760d5c82dc31e527ccc7cae3d16ac22ed65 WatchSource:0}: Error finding container 032f63201c023fb5b3d8cdcab9a10760d5c82dc31e527ccc7cae3d16ac22ed65: Status 404 returned error can't find the container with id 032f63201c023fb5b3d8cdcab9a10760d5c82dc31e527ccc7cae3d16ac22ed65
Jul 17 17:29:24.794854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:24.794820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" event={"ID":"e36623b7-776e-4c85-ad26-63f7c057a64c","Type":"ContainerStarted","Data":"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"}
Jul 17 17:29:24.794854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:24.794859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" event={"ID":"e36623b7-776e-4c85-ad26-63f7c057a64c","Type":"ContainerStarted","Data":"032f63201c023fb5b3d8cdcab9a10760d5c82dc31e527ccc7cae3d16ac22ed65"}
Jul 17 17:29:24.812293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:24.812241    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" podStartSLOduration=1.8122246450000001 podStartE2EDuration="1.812224645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:24.81071394 +0000 UTC m=+3798.795801477" watchObservedRunningTime="2026-07-17 17:29:24.812224645 +0000 UTC m=+3798.797312236"
Jul 17 17:29:25.949510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:25.949469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:25.952932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:25.952914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:25.964095 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:25.961317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:26.069695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.069659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-65xvv\" (UID: \"1f3a74c7-24e5-4cfb-9111-0898639bbe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:26.170425 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.170383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-65xvv\" (UID: \"1f3a74c7-24e5-4cfb-9111-0898639bbe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:26.171113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.171079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-65xvv\" (UID: \"1f3a74c7-24e5-4cfb-9111-0898639bbe97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:26.268402 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.268308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:26.615240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.615216    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:26.617144 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:29:26.617116    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3a74c7_24e5_4cfb_9111_0898639bbe97.slice/crio-78149aac06eeaec9177dac2cf9ae64c9a2aa32f9fec82af2e0a9359e1f4d4182 WatchSource:0}: Error finding container 78149aac06eeaec9177dac2cf9ae64c9a2aa32f9fec82af2e0a9359e1f4d4182: Status 404 returned error can't find the container with id 78149aac06eeaec9177dac2cf9ae64c9a2aa32f9fec82af2e0a9359e1f4d4182
Jul 17 17:29:26.805980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:26.805942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerStarted","Data":"78149aac06eeaec9177dac2cf9ae64c9a2aa32f9fec82af2e0a9359e1f4d4182"}
Jul 17 17:29:27.812275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:27.812243    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerID="f23ddb55527c5610d081aecdc0c0045d15cb7bd99493f9b803f9deb0cfb904f0" exitCode=7
Jul 17 17:29:27.812657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:27.812326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerDied","Data":"f23ddb55527c5610d081aecdc0c0045d15cb7bd99493f9b803f9deb0cfb904f0"}
Jul 17 17:29:27.812657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:27.812498    2561 scope.go:117] "RemoveContainer" containerID="f23ddb55527c5610d081aecdc0c0045d15cb7bd99493f9b803f9deb0cfb904f0"
Jul 17 17:29:28.508581 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:28.508545    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:29:28.508802 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:28.508777    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:29:28.818686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:28.818655    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerID="64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1" exitCode=7
Jul 17 17:29:28.819076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:28.818731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerDied","Data":"64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1"}
Jul 17 17:29:28.819076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:28.818795    2561 scope.go:117] "RemoveContainer" containerID="f23ddb55527c5610d081aecdc0c0045d15cb7bd99493f9b803f9deb0cfb904f0"
Jul 17 17:29:28.819076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:28.819071    2561 scope.go:117] "RemoveContainer" containerID="64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1"
Jul 17 17:29:28.819298 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:28.819280    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-65xvv_odh-ai-gateway-infra(1f3a74c7-24e5-4cfb-9111-0898639bbe97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97"
Jul 17 17:29:29.826002 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:29.825977    2561 scope.go:117] "RemoveContainer" containerID="64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1"
Jul 17 17:29:29.826410 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:29.826191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-65xvv_odh-ai-gateway-infra(1f3a74c7-24e5-4cfb-9111-0898639bbe97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97"
Jul 17 17:29:30.865122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:30.865080    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:29:33.508280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.508246    2561 scope.go:117] "RemoveContainer" containerID="47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024"
Jul 17 17:29:33.716108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.716069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:33.716268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.716125    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:33.728640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.728592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:33.845933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.845903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerStarted","Data":"ebd4763f90e1ee430aeef5755a00e7e9e36e34ab709d83a523954166dba5a7e4"}
Jul 17 17:29:33.857122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.857098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:29:33.862701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:33.862657    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" podStartSLOduration=17.862643199 podStartE2EDuration="17.862643199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:33.86121697 +0000 UTC m=+3807.846304530" watchObservedRunningTime="2026-07-17 17:29:33.862643199 +0000 UTC m=+3807.847730735"
Jul 17 17:29:34.852711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:34.852679    2561 generic.go:356] "Generic (PLEG): container finished" podID="04596f3f-d762-4764-844d-7a0555ac0cda" containerID="ebd4763f90e1ee430aeef5755a00e7e9e36e34ab709d83a523954166dba5a7e4" exitCode=7
Jul 17 17:29:34.853168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:34.852748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerDied","Data":"ebd4763f90e1ee430aeef5755a00e7e9e36e34ab709d83a523954166dba5a7e4"}
Jul 17 17:29:34.853168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:34.852810    2561 scope.go:117] "RemoveContainer" containerID="47ef954c49cb91b7725077893077a3ee6884c53289417f394de04b5f026d8024"
Jul 17 17:29:34.853303 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:34.853180    2561 scope.go:117] "RemoveContainer" containerID="ebd4763f90e1ee430aeef5755a00e7e9e36e34ab709d83a523954166dba5a7e4"
Jul 17 17:29:34.853397 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:34.853372    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q_odh-ai-gateway-infra(04596f3f-d762-4764-844d-7a0555ac0cda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" podUID="04596f3f-d762-4764-844d-7a0555ac0cda"
Jul 17 17:29:34.874663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:34.874639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:35.508797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:35.508770    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:29:35.509007 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:35.508966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:29:35.994136 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:35.994103    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:36.061553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.061519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca\") pod \"04596f3f-d762-4764-844d-7a0555ac0cda\" (UID: \"04596f3f-d762-4764-844d-7a0555ac0cda\") "
Jul 17 17:29:36.061875 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.061854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04596f3f-d762-4764-844d-7a0555ac0cda" (UID: "04596f3f-d762-4764-844d-7a0555ac0cda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:29:36.162995 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.162910    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04596f3f-d762-4764-844d-7a0555ac0cda-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:29:36.865127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.865099    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"
Jul 17 17:29:36.865301 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.865135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q" event={"ID":"04596f3f-d762-4764-844d-7a0555ac0cda","Type":"ContainerDied","Data":"7054dd0967d5ede759f151f8dbf87619866e60df2ce6f0c75366c03a2c789980"}
Jul 17 17:29:36.865301 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.865193    2561 scope.go:117] "RemoveContainer" containerID="ebd4763f90e1ee430aeef5755a00e7e9e36e34ab709d83a523954166dba5a7e4"
Jul 17 17:29:36.886790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.886766    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:36.895372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:36.895350    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-xkf2q"]
Jul 17 17:29:38.514014 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:38.513976    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" path="/var/lib/kubelet/pods/04596f3f-d762-4764-844d-7a0555ac0cda/volumes"
Jul 17 17:29:39.508574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:39.508546    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:29:39.508759 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:39.508696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:29:41.363199 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363157    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:29:41.363737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363679    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363699    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363725    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363852    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363866    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.363954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.363878    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:29:41.368104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.368082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:41.372972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.372934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:29:41.517309 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.517266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5\" (UID: \"08cb13d3-0639-4314-b979-cced28a87f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:41.618252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.618153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5\" (UID: \"08cb13d3-0639-4314-b979-cced28a87f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:41.618761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.618743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5\" (UID: \"08cb13d3-0639-4314-b979-cced28a87f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:41.679759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.679728    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:41.811992 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.811964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:29:41.814410 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:29:41.814361    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cb13d3_0639_4314_b979_cced28a87f30.slice/crio-52e2722a2a7825159ffcf19269fd818e46273f757758c9fdee00c76b2d5e25e5 WatchSource:0}: Error finding container 52e2722a2a7825159ffcf19269fd818e46273f757758c9fdee00c76b2d5e25e5: Status 404 returned error can't find the container with id 52e2722a2a7825159ffcf19269fd818e46273f757758c9fdee00c76b2d5e25e5
Jul 17 17:29:41.892254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:41.892184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerStarted","Data":"52e2722a2a7825159ffcf19269fd818e46273f757758c9fdee00c76b2d5e25e5"}
Jul 17 17:29:42.508882 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.508853    2561 scope.go:117] "RemoveContainer" containerID="64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1"
Jul 17 17:29:42.898912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.898872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerStarted","Data":"e9e18a4edacec0b8ef4a2241dd256b2584fc6f7e9bd0cece569bfa0a35a22a24"}
Jul 17 17:29:42.900342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.900312    2561 generic.go:356] "Generic (PLEG): container finished" podID="08cb13d3-0639-4314-b979-cced28a87f30" containerID="e3ab7875046216d742f72bf123905118a0caf9e016f0053f908ab78b8219f7b7" exitCode=7
Jul 17 17:29:42.900488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.900348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerDied","Data":"e3ab7875046216d742f72bf123905118a0caf9e016f0053f908ab78b8219f7b7"}
Jul 17 17:29:42.900527 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.900514    2561 scope.go:117] "RemoveContainer" containerID="e3ab7875046216d742f72bf123905118a0caf9e016f0053f908ab78b8219f7b7"
Jul 17 17:29:42.914965 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:42.914924    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" podStartSLOduration=17.914907944 podStartE2EDuration="17.914907944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:29:42.913597017 +0000 UTC m=+3816.898684564" watchObservedRunningTime="2026-07-17 17:29:42.914907944 +0000 UTC m=+3816.899995510"
Jul 17 17:29:43.906928 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.906891    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerID="e9e18a4edacec0b8ef4a2241dd256b2584fc6f7e9bd0cece569bfa0a35a22a24" exitCode=7
Jul 17 17:29:43.907398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.906963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerDied","Data":"e9e18a4edacec0b8ef4a2241dd256b2584fc6f7e9bd0cece569bfa0a35a22a24"}
Jul 17 17:29:43.907398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.907063    2561 scope.go:117] "RemoveContainer" containerID="64e1171e5b40246cfcd5986da105a82467b9526a7e9ed407cd1d8761816ed9b1"
Jul 17 17:29:43.907398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.907378    2561 scope.go:117] "RemoveContainer" containerID="e9e18a4edacec0b8ef4a2241dd256b2584fc6f7e9bd0cece569bfa0a35a22a24"
Jul 17 17:29:43.907644 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:43.907621    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-65xvv_odh-ai-gateway-infra(1f3a74c7-24e5-4cfb-9111-0898639bbe97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97"
Jul 17 17:29:43.908876 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.908856    2561 generic.go:356] "Generic (PLEG): container finished" podID="08cb13d3-0639-4314-b979-cced28a87f30" containerID="bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f" exitCode=7
Jul 17 17:29:43.908977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.908904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerDied","Data":"bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f"}
Jul 17 17:29:43.909137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.909123    2561 scope.go:117] "RemoveContainer" containerID="bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f"
Jul 17 17:29:43.909331 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:43.909313    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5_odh-ai-gateway-infra(08cb13d3-0639-4314-b979-cced28a87f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" podUID="08cb13d3-0639-4314-b979-cced28a87f30"
Jul 17 17:29:43.918001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.917983    2561 scope.go:117] "RemoveContainer" containerID="e3ab7875046216d742f72bf123905118a0caf9e016f0053f908ab78b8219f7b7"
Jul 17 17:29:43.923074 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:43.923038    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:44.917163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:44.917136    2561 scope.go:117] "RemoveContainer" containerID="bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f"
Jul 17 17:29:44.917458 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:44.917423    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5_odh-ai-gateway-infra(08cb13d3-0639-4314-b979-cced28a87f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" podUID="08cb13d3-0639-4314-b979-cced28a87f30"
Jul 17 17:29:45.039791 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.039769    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:45.153887 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.153854    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca\") pod \"1f3a74c7-24e5-4cfb-9111-0898639bbe97\" (UID: \"1f3a74c7-24e5-4cfb-9111-0898639bbe97\") "
Jul 17 17:29:45.154233 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.154211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f3a74c7-24e5-4cfb-9111-0898639bbe97" (UID: "1f3a74c7-24e5-4cfb-9111-0898639bbe97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:29:45.254763 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.254689    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a74c7-24e5-4cfb-9111-0898639bbe97-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:29:45.922932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.922897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv" event={"ID":"1f3a74c7-24e5-4cfb-9111-0898639bbe97","Type":"ContainerDied","Data":"78149aac06eeaec9177dac2cf9ae64c9a2aa32f9fec82af2e0a9359e1f4d4182"}
Jul 17 17:29:45.923356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.922951    2561 scope.go:117] "RemoveContainer" containerID="e9e18a4edacec0b8ef4a2241dd256b2584fc6f7e9bd0cece569bfa0a35a22a24"
Jul 17 17:29:45.923356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.922911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"
Jul 17 17:29:45.947890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.947859    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:45.951751 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:45.951728    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-65xvv"]
Jul 17 17:29:46.514676 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:46.514643    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" path="/var/lib/kubelet/pods/1f3a74c7-24e5-4cfb-9111-0898639bbe97/volumes"
Jul 17 17:29:47.817597 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:47.817558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:48.508709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:48.508672    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:29:48.508969 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:48.508932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:29:48.675950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:48.675911    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:29:48.696644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:48.696607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:49.587719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:49.587679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:29:49.610938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:49.610910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:50.345948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:50.345915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:50.447282 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:50.447248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:29:52.509104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:52.509076    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:29:52.509485 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:52.509271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:29:55.062823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:55.062792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:29:56.510577 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:56.510544    2561 scope.go:117] "RemoveContainer" containerID="bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f"
Jul 17 17:29:56.976864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:56.976834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerStarted","Data":"55a9b91520f27f6a31563f3e0716523f6b36fd29a35ff2fd0d2b01b9b9b9625b"}
Jul 17 17:29:57.983504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:57.983471    2561 generic.go:356] "Generic (PLEG): container finished" podID="08cb13d3-0639-4314-b979-cced28a87f30" containerID="55a9b91520f27f6a31563f3e0716523f6b36fd29a35ff2fd0d2b01b9b9b9625b" exitCode=7
Jul 17 17:29:57.983944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:57.983554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerDied","Data":"55a9b91520f27f6a31563f3e0716523f6b36fd29a35ff2fd0d2b01b9b9b9625b"}
Jul 17 17:29:57.983944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:57.983614    2561 scope.go:117] "RemoveContainer" containerID="bda8b11726a04c8c4b363a398b7b660bcd114da35677dd63ccf90cd01d891e5f"
Jul 17 17:29:57.983944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:57.983846    2561 scope.go:117] "RemoveContainer" containerID="55a9b91520f27f6a31563f3e0716523f6b36fd29a35ff2fd0d2b01b9b9b9625b"
Jul 17 17:29:57.984120 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:29:57.984100    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5_odh-ai-gateway-infra(08cb13d3-0639-4314-b979-cced28a87f30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" podUID="08cb13d3-0639-4314-b979-cced28a87f30"
Jul 17 17:29:58.006529 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:58.006507    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:29:59.124279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.124254    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:59.186182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.186153    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca\") pod \"08cb13d3-0639-4314-b979-cced28a87f30\" (UID: \"08cb13d3-0639-4314-b979-cced28a87f30\") "
Jul 17 17:29:59.186512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.186491    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08cb13d3-0639-4314-b979-cced28a87f30" (UID: "08cb13d3-0639-4314-b979-cced28a87f30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:29:59.287729 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.287689    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cb13d3-0639-4314-b979-cced28a87f30-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:29:59.995733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.995703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"
Jul 17 17:29:59.995733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.995729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5" event={"ID":"08cb13d3-0639-4314-b979-cced28a87f30","Type":"ContainerDied","Data":"52e2722a2a7825159ffcf19269fd818e46273f757758c9fdee00c76b2d5e25e5"}
Jul 17 17:29:59.995966 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:29:59.995783    2561 scope.go:117] "RemoveContainer" containerID="55a9b91520f27f6a31563f3e0716523f6b36fd29a35ff2fd0d2b01b9b9b9625b"
Jul 17 17:30:00.018424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.018399    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:30:00.026635 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.026613    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-tj5p5"]
Jul 17 17:30:00.144024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.143987    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144376    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144391    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144396    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144409    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144414    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144420    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144425    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144436    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144446    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04596f3f-d762-4764-844d-7a0555ac0cda" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144501    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144508    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144518    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.144778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.144524    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.148826 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.148803    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"]
Jul 17 17:30:00.148952 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.148913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"
Jul 17 17:30:00.149265 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.149248    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.149321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.149269    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.149367 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.149352    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3a74c7-24e5-4cfb-9111-0898639bbe97" containerName="revoke-keys"
Jul 17 17:30:00.149367 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.149362    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cb13d3-0639-4314-b979-cced28a87f30" containerName="revoke-keys"
Jul 17 17:30:00.152280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.152261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"
Jul 17 17:30:00.154445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.154413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:30:00.160535 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.160518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"
Jul 17 17:30:00.163899 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.163857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:00.166750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.166726    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"]
Jul 17 17:30:00.168880 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.168851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"
Jul 17 17:30:00.171032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.171014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"]
Jul 17 17:30:00.171358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.171346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"
Jul 17 17:30:00.187186 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.187077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"]
Jul 17 17:30:00.196895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.196850    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:30:00.197779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.197163    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" podUID="c197c441-ad92-485a-bb85-76c11377c967" containerName="maas-api" containerID="cri-o://8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c" gracePeriod=30
Jul 17 17:30:00.221407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.221382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"
Jul 17 17:30:00.225560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.225526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:00.346351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.346323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"]
Jul 17 17:30:00.349111 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:00.349068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea02c0d6_ff74_4d73_b26f_7c776579d69f.slice/crio-09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8 WatchSource:0}: Error finding container 09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8: Status 404 returned error can't find the container with id 09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8
Jul 17 17:30:00.378485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.378457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:00.385137 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:00.385107    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b66b5a7_1bea_426f_98ce_9c7ea4975419.slice/crio-ee4c2b3443856b69ad70819d4abac834d5f652eafed39a79e3bafc97debf500f WatchSource:0}: Error finding container ee4c2b3443856b69ad70819d4abac834d5f652eafed39a79e3bafc97debf500f: Status 404 returned error can't find the container with id ee4c2b3443856b69ad70819d4abac834d5f652eafed39a79e3bafc97debf500f
Jul 17 17:30:00.405459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.405439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"]
Jul 17 17:30:00.407647 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:00.407619    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824dd8ea_4cf6_48b8_9d76_793ee0661ffe.slice/crio-9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5 WatchSource:0}: Error finding container 9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5: Status 404 returned error can't find the container with id 9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5
Jul 17 17:30:00.466976 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.466955    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:30:00.506938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.506866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca\") pod \"c197c441-ad92-485a-bb85-76c11377c967\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") "
Jul 17 17:30:00.506938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.506902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls\") pod \"c197c441-ad92-485a-bb85-76c11377c967\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") "
Jul 17 17:30:00.506938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.506933    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c65ll\" (UniqueName: \"kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll\") pod \"c197c441-ad92-485a-bb85-76c11377c967\" (UID: \"c197c441-ad92-485a-bb85-76c11377c967\") "
Jul 17 17:30:00.507345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.507320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c197c441-ad92-485a-bb85-76c11377c967" (UID: "c197c441-ad92-485a-bb85-76c11377c967"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:30:00.509025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.508977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll" (OuterVolumeSpecName: "kube-api-access-c65ll") pod "c197c441-ad92-485a-bb85-76c11377c967" (UID: "c197c441-ad92-485a-bb85-76c11377c967"). InnerVolumeSpecName "kube-api-access-c65ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:30:00.509133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.509017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c197c441-ad92-485a-bb85-76c11377c967" (UID: "c197c441-ad92-485a-bb85-76c11377c967"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:30:00.514479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.514452    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08cb13d3-0639-4314-b979-cced28a87f30" path="/var/lib/kubelet/pods/08cb13d3-0639-4314-b979-cced28a87f30/volumes"
Jul 17 17:30:00.607884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.607844    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c197c441-ad92-485a-bb85-76c11377c967-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:00.607884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.607883    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c197c441-ad92-485a-bb85-76c11377c967-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:00.608103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:00.607899    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65ll\" (UniqueName: \"kubernetes.io/projected/c197c441-ad92-485a-bb85-76c11377c967-kube-api-access-c65ll\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:01.001904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.001871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq" event={"ID":"ea02c0d6-ff74-4d73-b26f-7c776579d69f","Type":"ContainerStarted","Data":"7d27fd0243d81121c39b2109e8def119eb78541fa1b724216846788fe9426c0d"}
Jul 17 17:30:01.002116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.001924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq" event={"ID":"ea02c0d6-ff74-4d73-b26f-7c776579d69f","Type":"ContainerStarted","Data":"09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8"}
Jul 17 17:30:01.003085 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.003063    2561 generic.go:356] "Generic (PLEG): container finished" podID="c197c441-ad92-485a-bb85-76c11377c967" containerID="8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c" exitCode=0
Jul 17 17:30:01.003207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.003099    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"
Jul 17 17:30:01.003207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.003140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" event={"ID":"c197c441-ad92-485a-bb85-76c11377c967","Type":"ContainerDied","Data":"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"}
Jul 17 17:30:01.003207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.003178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n" event={"ID":"c197c441-ad92-485a-bb85-76c11377c967","Type":"ContainerDied","Data":"c35fee7f515ff18002f3af733b5eb801732e701b79acafc1347755fb7aefacbf"}
Jul 17 17:30:01.003207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.003207    2561 scope.go:117] "RemoveContainer" containerID="8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"
Jul 17 17:30:01.004640 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.004616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerStarted","Data":"76adf674c8442ac440217e9dde1ed748925fffab515341fdc29816edabe892c9"}
Jul 17 17:30:01.004736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.004654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerStarted","Data":"ee4c2b3443856b69ad70819d4abac834d5f652eafed39a79e3bafc97debf500f"}
Jul 17 17:30:01.006388 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.006367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5" event={"ID":"824dd8ea-4cf6-48b8-9d76-793ee0661ffe","Type":"ContainerStarted","Data":"f6f4ec36171f67495a3a7c16783606dbc1ef7895f96fa1d1c9328ede0751fb93"}
Jul 17 17:30:01.006503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.006407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5" event={"ID":"824dd8ea-4cf6-48b8-9d76-793ee0661ffe","Type":"ContainerStarted","Data":"9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5"}
Jul 17 17:30:01.012130 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.012119    2561 scope.go:117] "RemoveContainer" containerID="8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"
Jul 17 17:30:01.012380 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:01.012362    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c\": container with ID starting with 8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c not found: ID does not exist" containerID="8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"
Jul 17 17:30:01.012445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.012391    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c"} err="failed to get container status \"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c\": rpc error: code = NotFound desc = could not find container \"8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c\": container with ID starting with 8910dcaf7a43dc307372366d109330f0e8a40a12703928b686048c855433bd6c not found: ID does not exist"
Jul 17 17:30:01.019089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.019028    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq" podStartSLOduration=1.019015517 podStartE2EDuration="1.019015517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:01.017170285 +0000 UTC m=+3835.002257844" watchObservedRunningTime="2026-07-17 17:30:01.019015517 +0000 UTC m=+3835.004103054"
Jul 17 17:30:01.029103 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.029081    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:30:01.035032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.035012    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ca540-588fd99f6d-qsh4n"]
Jul 17 17:30:01.046246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.046205    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" podStartSLOduration=1.046180615 podStartE2EDuration="1.046180615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:01.045851526 +0000 UTC m=+3835.030939064" watchObservedRunningTime="2026-07-17 17:30:01.046180615 +0000 UTC m=+3835.031268153"
Jul 17 17:30:01.059534 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:01.059494    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5" podStartSLOduration=1.059482122 podStartE2EDuration="1.059482122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:01.058425724 +0000 UTC m=+3835.043513305" watchObservedRunningTime="2026-07-17 17:30:01.059482122 +0000 UTC m=+3835.044569697"
Jul 17 17:30:02.013141 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.013106    2561 generic.go:356] "Generic (PLEG): container finished" podID="ea02c0d6-ff74-4d73-b26f-7c776579d69f" containerID="7d27fd0243d81121c39b2109e8def119eb78541fa1b724216846788fe9426c0d" exitCode=0
Jul 17 17:30:02.013584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.013197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq" event={"ID":"ea02c0d6-ff74-4d73-b26f-7c776579d69f","Type":"ContainerDied","Data":"7d27fd0243d81121c39b2109e8def119eb78541fa1b724216846788fe9426c0d"}
Jul 17 17:30:02.015302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.015273    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerID="76adf674c8442ac440217e9dde1ed748925fffab515341fdc29816edabe892c9" exitCode=7
Jul 17 17:30:02.015438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.015334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerDied","Data":"76adf674c8442ac440217e9dde1ed748925fffab515341fdc29816edabe892c9"}
Jul 17 17:30:02.015509 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.015483    2561 scope.go:117] "RemoveContainer" containerID="76adf674c8442ac440217e9dde1ed748925fffab515341fdc29816edabe892c9"
Jul 17 17:30:02.016909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.016887    2561 generic.go:356] "Generic (PLEG): container finished" podID="824dd8ea-4cf6-48b8-9d76-793ee0661ffe" containerID="f6f4ec36171f67495a3a7c16783606dbc1ef7895f96fa1d1c9328ede0751fb93" exitCode=0
Jul 17 17:30:02.017031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.016935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5" event={"ID":"824dd8ea-4cf6-48b8-9d76-793ee0661ffe","Type":"ContainerDied","Data":"f6f4ec36171f67495a3a7c16783606dbc1ef7895f96fa1d1c9328ede0751fb93"}
Jul 17 17:30:02.514659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:02.514625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c197c441-ad92-485a-bb85-76c11377c967" path="/var/lib/kubelet/pods/c197c441-ad92-485a-bb85-76c11377c967/volumes"
Jul 17 17:30:03.023392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.023366    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerID="d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67" exitCode=7
Jul 17 17:30:03.023837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.023451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerDied","Data":"d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67"}
Jul 17 17:30:03.023837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.023509    2561 scope.go:117] "RemoveContainer" containerID="76adf674c8442ac440217e9dde1ed748925fffab515341fdc29816edabe892c9"
Jul 17 17:30:03.023837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.023592    2561 scope.go:117] "RemoveContainer" containerID="d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67"
Jul 17 17:30:03.023980 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:03.023860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh_odh-ai-gateway-infra(2b66b5a7-1bea-426f-98ce-9c7ea4975419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419"
Jul 17 17:30:03.182923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.182903    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"
Jul 17 17:30:03.184945 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.184926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"
Jul 17 17:30:03.508518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.508438    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:30:03.508660 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:03.508626    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:30:03.696565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.696518    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" podUID="341c2d7e-76c5-407d-814b-e00b37879ea8" containerName="main" containerID="cri-o://07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9" gracePeriod=45
Jul 17 17:30:03.946939 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:03.946916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:30:04.029817 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.029789    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"
Jul 17 17:30:04.030270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.029829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq" event={"ID":"ea02c0d6-ff74-4d73-b26f-7c776579d69f","Type":"ContainerDied","Data":"09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8"}
Jul 17 17:30:04.030270 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.029881    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09bcb8959261171c02c9b69d40729b3626eb3b8d8ffecccf44a6e4b8e9e368f8"
Jul 17 17:30:04.031559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.031540    2561 scope.go:117] "RemoveContainer" containerID="d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67"
Jul 17 17:30:04.031825 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:04.031796    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh_odh-ai-gateway-infra(2b66b5a7-1bea-426f-98ce-9c7ea4975419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419"
Jul 17 17:30:04.032674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.032653    2561 generic.go:356] "Generic (PLEG): container finished" podID="341c2d7e-76c5-407d-814b-e00b37879ea8" containerID="07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9" exitCode=0
Jul 17 17:30:04.032771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.032689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"
Jul 17 17:30:04.032771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.032716    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" event={"ID":"341c2d7e-76c5-407d-814b-e00b37879ea8","Type":"ContainerDied","Data":"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"}
Jul 17 17:30:04.032771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.032746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk" event={"ID":"341c2d7e-76c5-407d-814b-e00b37879ea8","Type":"ContainerDied","Data":"2807600fa3d04142353e9bf9886475748b7dae7bc95777f58ff1855194feda28"}
Jul 17 17:30:04.032771 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.032766    2561 scope.go:117] "RemoveContainer" containerID="07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"
Jul 17 17:30:04.033959 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.033941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5" event={"ID":"824dd8ea-4cf6-48b8-9d76-793ee0661ffe","Type":"ContainerDied","Data":"9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5"}
Jul 17 17:30:04.034022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.033975    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738490-schh5"
Jul 17 17:30:04.034113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.033980    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9f5183f44df4a88deb9c9b7b7aa5bd97e310067bde004cde9c0a0385d79ed5"
Jul 17 17:30:04.035833 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.035813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.035939 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.035865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.035939 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.035906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.035973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxg7s\" (UniqueName: \"kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036099    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir\") pod \"341c2d7e-76c5-407d-814b-e00b37879ea8\" (UID: \"341c2d7e-76c5-407d-814b-e00b37879ea8\") "
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home" (OuterVolumeSpecName: "home") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache" (OuterVolumeSpecName: "model-cache") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036379    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.036418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036397    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.036948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.036440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.038451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.038424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm" (OuterVolumeSpecName: "dshm") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.038560 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.038534    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s" (OuterVolumeSpecName: "kube-api-access-fxg7s") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "kube-api-access-fxg7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:30:04.038658 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.038591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "341c2d7e-76c5-407d-814b-e00b37879ea8" (UID: "341c2d7e-76c5-407d-814b-e00b37879ea8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:30:04.043171 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.043155    2561 scope.go:117] "RemoveContainer" containerID="07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"
Jul 17 17:30:04.043399 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:04.043379    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9\": container with ID starting with 07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9 not found: ID does not exist" containerID="07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"
Jul 17 17:30:04.043453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.043410    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9"} err="failed to get container status \"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9\": rpc error: code = NotFound desc = could not find container \"07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9\": container with ID starting with 07ded3ffb6e5d92ad46cecca94ba4d96f3ea8af519a7d6cf8529ce4f50bb12c9 not found: ID does not exist"
Jul 17 17:30:04.136865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.136799    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c2d7e-76c5-407d-814b-e00b37879ea8-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.136865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.136824    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxg7s\" (UniqueName: \"kubernetes.io/projected/341c2d7e-76c5-407d-814b-e00b37879ea8-kube-api-access-fxg7s\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.136865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.136834    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.136865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.136843    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c2d7e-76c5-407d-814b-e00b37879ea8-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.353336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.353307    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:30:04.356718 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.356690    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7ca540/test-model-7ca540-kserve-5b99dd69df-hr2lk"]
Jul 17 17:30:04.515001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.514926    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341c2d7e-76c5-407d-814b-e00b37879ea8" path="/var/lib/kubelet/pods/341c2d7e-76c5-407d-814b-e00b37879ea8/volumes"
Jul 17 17:30:04.608309 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.608265    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" podUID="e36623b7-776e-4c85-ad26-63f7c057a64c" containerName="main" containerID="cri-o://8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827" gracePeriod=45
Jul 17 17:30:04.816864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.816831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:04.855943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.855919    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:30:04.943873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.943845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.943879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.943916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.943939    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lrgw\" (UniqueName: \"kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home\") pod \"e36623b7-776e-4c85-ad26-63f7c057a64c\" (UID: \"e36623b7-776e-4c85-ad26-63f7c057a64c\") "
Jul 17 17:30:04.944245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.944245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.944399 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944360    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.944399 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home" (OuterVolumeSpecName: "home") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:04.944399 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.944386    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:04.946067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.946035    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:30:04.946177 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.946153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw" (OuterVolumeSpecName: "kube-api-access-5lrgw") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "kube-api-access-5lrgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:30:04.946231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:04.946219    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm" (OuterVolumeSpecName: "dshm") pod "e36623b7-776e-4c85-ad26-63f7c057a64c" (UID: "e36623b7-776e-4c85-ad26-63f7c057a64c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:30:05.040840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.040757    2561 generic.go:356] "Generic (PLEG): container finished" podID="e36623b7-776e-4c85-ad26-63f7c057a64c" containerID="8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827" exitCode=0
Jul 17 17:30:05.040840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.040798    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" event={"ID":"e36623b7-776e-4c85-ad26-63f7c057a64c","Type":"ContainerDied","Data":"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"}
Jul 17 17:30:05.040840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.040801    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"
Jul 17 17:30:05.040840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.040823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9" event={"ID":"e36623b7-776e-4c85-ad26-63f7c057a64c","Type":"ContainerDied","Data":"032f63201c023fb5b3d8cdcab9a10760d5c82dc31e527ccc7cae3d16ac22ed65"}
Jul 17 17:30:05.040840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.040840    2561 scope.go:117] "RemoveContainer" containerID="8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"
Jul 17 17:30:05.045021 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.044983    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e36623b7-776e-4c85-ad26-63f7c057a64c-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:05.045021 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.045019    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:05.045206 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.045027    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrgw\" (UniqueName: \"kubernetes.io/projected/e36623b7-776e-4c85-ad26-63f7c057a64c-kube-api-access-5lrgw\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:05.045206 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.045036    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e36623b7-776e-4c85-ad26-63f7c057a64c-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:05.049293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.049276    2561 scope.go:117] "RemoveContainer" containerID="8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"
Jul 17 17:30:05.049517 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:05.049502    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827\": container with ID starting with 8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827 not found: ID does not exist" containerID="8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"
Jul 17 17:30:05.049564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.049525    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827"} err="failed to get container status \"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827\": rpc error: code = NotFound desc = could not find container \"8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827\": container with ID starting with 8ad61326dd6e5db90869618b13e1fa1e15180ccfd3e77bb299bd99df8b925827 not found: ID does not exist"
Jul 17 17:30:05.063871 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.063850    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:30:05.069129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.069109    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f748a7/test-model-f748a7-kserve-59cdd497ff-gkcs9"]
Jul 17 17:30:05.206298 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.206261    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:05.508524 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:05.508449    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:30:05.508669 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:05.508628    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:30:06.251184 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:06.251156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:30:06.515220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:06.515146    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36623b7-776e-4c85-ad26-63f7c057a64c" path="/var/lib/kubelet/pods/e36623b7-776e-4c85-ad26-63f7c057a64c/volumes"
Jul 17 17:30:11.002088 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.002012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:11.068695 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.068662    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:11.256804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.256707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:11.489275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.489246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:11.597333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597304    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"]
Jul 17 17:30:11.597651 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597639    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c197c441-ad92-485a-bb85-76c11377c967" containerName="maas-api"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c197c441-ad92-485a-bb85-76c11377c967" containerName="maas-api"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597669    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824dd8ea-4cf6-48b8-9d76-793ee0661ffe" containerName="cleanup"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597675    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="824dd8ea-4cf6-48b8-9d76-793ee0661ffe" containerName="cleanup"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597682    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341c2d7e-76c5-407d-814b-e00b37879ea8" containerName="main"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597688    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="341c2d7e-76c5-407d-814b-e00b37879ea8" containerName="main"
Jul 17 17:30:11.597697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597695    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea02c0d6-ff74-4d73-b26f-7c776579d69f" containerName="cleanup"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597700    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea02c0d6-ff74-4d73-b26f-7c776579d69f" containerName="cleanup"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597707    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36623b7-776e-4c85-ad26-63f7c057a64c" containerName="main"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597712    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36623b7-776e-4c85-ad26-63f7c057a64c" containerName="main"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="824dd8ea-4cf6-48b8-9d76-793ee0661ffe" containerName="cleanup"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597780    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea02c0d6-ff74-4d73-b26f-7c776579d69f" containerName="cleanup"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597789    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c197c441-ad92-485a-bb85-76c11377c967" containerName="maas-api"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597796    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="341c2d7e-76c5-407d-814b-e00b37879ea8" containerName="main"
Jul 17 17:30:11.597895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.597801    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36623b7-776e-4c85-ad26-63f7c057a64c" containerName="main"
Jul 17 17:30:11.602095 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.602076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:11.607309 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.607290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"]
Jul 17 17:30:11.699847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.699820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7\" (UID: \"69be4c99-33ed-4a98-8bc5-72a8a6dd1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:11.800236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.800213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7\" (UID: \"69be4c99-33ed-4a98-8bc5-72a8a6dd1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:11.800756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.800738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7\" (UID: \"69be4c99-33ed-4a98-8bc5-72a8a6dd1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:11.913068 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:11.912973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:12.174830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:12.174733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:12.242605 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:12.242584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"]
Jul 17 17:30:12.244484 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:12.244454    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69be4c99_33ed_4a98_8bc5_72a8a6dd1871.slice/crio-f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5 WatchSource:0}: Error finding container f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5: Status 404 returned error can't find the container with id f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5
Jul 17 17:30:12.244612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:12.244559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:13.088603 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:13.088533    2561 generic.go:356] "Generic (PLEG): container finished" podID="69be4c99-33ed-4a98-8bc5-72a8a6dd1871" containerID="0016c5f96877ec218df6d716eb4d96db45a5242917b813b01e4d4b993493bd55" exitCode=0
Jul 17 17:30:13.088727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:13.088612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7" event={"ID":"69be4c99-33ed-4a98-8bc5-72a8a6dd1871","Type":"ContainerDied","Data":"0016c5f96877ec218df6d716eb4d96db45a5242917b813b01e4d4b993493bd55"}
Jul 17 17:30:13.088727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:13.088663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7" event={"ID":"69be4c99-33ed-4a98-8bc5-72a8a6dd1871","Type":"ContainerStarted","Data":"f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5"}
Jul 17 17:30:14.220248 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:14.220226    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:14.318760 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:14.318730    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca\") pod \"69be4c99-33ed-4a98-8bc5-72a8a6dd1871\" (UID: \"69be4c99-33ed-4a98-8bc5-72a8a6dd1871\") "
Jul 17 17:30:14.319038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:14.319018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69be4c99-33ed-4a98-8bc5-72a8a6dd1871" (UID: "69be4c99-33ed-4a98-8bc5-72a8a6dd1871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:30:14.419858 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:14.419801    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69be4c99-33ed-4a98-8bc5-72a8a6dd1871-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:15.099180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:15.099137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7" event={"ID":"69be4c99-33ed-4a98-8bc5-72a8a6dd1871","Type":"ContainerDied","Data":"f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5"}
Jul 17 17:30:15.099180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:15.099161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"
Jul 17 17:30:15.099407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:15.099197    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7cd2b00a7b3d995ce6529003b72c7ca39c6bffee797326913b5867b4cbfbff5"
Jul 17 17:30:15.508680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:15.508619    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:30:15.509069 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:15.508765    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:30:16.438558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.438524    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:16.438868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.438855    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69be4c99-33ed-4a98-8bc5-72a8a6dd1871" containerName="revoke-keys"
Jul 17 17:30:16.438912 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.438869    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="69be4c99-33ed-4a98-8bc5-72a8a6dd1871" containerName="revoke-keys"
Jul 17 17:30:16.438951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.438939    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="69be4c99-33ed-4a98-8bc5-72a8a6dd1871" containerName="revoke-keys"
Jul 17 17:30:16.441860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.441844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:16.448077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.448037    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:16.535542 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.535520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9\" (UID: \"4b485636-78c1-4695-947f-9513fa5eb0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:16.635890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.635863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9\" (UID: \"4b485636-78c1-4695-947f-9513fa5eb0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:16.636474 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.636456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9\" (UID: \"4b485636-78c1-4695-947f-9513fa5eb0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:16.754139 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.754070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:16.904931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:16.904907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:16.906851 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:16.906821    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b485636_78c1_4695_947f_9513fa5eb0de.slice/crio-0fb560237465069c91dc3871d9640b376df3609d9199ab4ec6fe007eca01fab1 WatchSource:0}: Error finding container 0fb560237465069c91dc3871d9640b376df3609d9199ab4ec6fe007eca01fab1: Status 404 returned error can't find the container with id 0fb560237465069c91dc3871d9640b376df3609d9199ab4ec6fe007eca01fab1
Jul 17 17:30:17.109258 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:17.109224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" event={"ID":"4b485636-78c1-4695-947f-9513fa5eb0de","Type":"ContainerStarted","Data":"0fb560237465069c91dc3871d9640b376df3609d9199ab4ec6fe007eca01fab1"}
Jul 17 17:30:18.115093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:18.115040    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b485636-78c1-4695-947f-9513fa5eb0de" containerID="9326942b3c09826a972b4fbb8b502119cf56709dfc2312c07f0480bfe6227769" exitCode=7
Jul 17 17:30:18.115470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:18.115114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" event={"ID":"4b485636-78c1-4695-947f-9513fa5eb0de","Type":"ContainerDied","Data":"9326942b3c09826a972b4fbb8b502119cf56709dfc2312c07f0480bfe6227769"}
Jul 17 17:30:18.115470 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:18.115276    2561 scope.go:117] "RemoveContainer" containerID="9326942b3c09826a972b4fbb8b502119cf56709dfc2312c07f0480bfe6227769"
Jul 17 17:30:18.508174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:18.508152    2561 scope.go:117] "RemoveContainer" containerID="d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67"
Jul 17 17:30:19.122419 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:19.122386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerStarted","Data":"30ca491c420f2e7ce9c8d7e19f210b5966ae96845c3d434dee61c2111c833a1d"}
Jul 17 17:30:19.124169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:19.124146    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b485636-78c1-4695-947f-9513fa5eb0de" containerID="aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398" exitCode=7
Jul 17 17:30:19.124302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:19.124173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" event={"ID":"4b485636-78c1-4695-947f-9513fa5eb0de","Type":"ContainerDied","Data":"aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398"}
Jul 17 17:30:19.124302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:19.124240    2561 scope.go:117] "RemoveContainer" containerID="9326942b3c09826a972b4fbb8b502119cf56709dfc2312c07f0480bfe6227769"
Jul 17 17:30:19.124405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:19.124355    2561 scope.go:117] "RemoveContainer" containerID="aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398"
Jul 17 17:30:19.124564 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:19.124543    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9_odh-ai-gateway-infra(4b485636-78c1-4695-947f-9513fa5eb0de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" podUID="4b485636-78c1-4695-947f-9513fa5eb0de"
Jul 17 17:30:20.131102 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.131062    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerID="30ca491c420f2e7ce9c8d7e19f210b5966ae96845c3d434dee61c2111c833a1d" exitCode=7
Jul 17 17:30:20.131553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.131075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerDied","Data":"30ca491c420f2e7ce9c8d7e19f210b5966ae96845c3d434dee61c2111c833a1d"}
Jul 17 17:30:20.131553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.131232    2561 scope.go:117] "RemoveContainer" containerID="d8fb1c6d003e45fcebf38437cac0da0034312c50c4459130555df5c694274d67"
Jul 17 17:30:20.131553 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.131477    2561 scope.go:117] "RemoveContainer" containerID="30ca491c420f2e7ce9c8d7e19f210b5966ae96845c3d434dee61c2111c833a1d"
Jul 17 17:30:20.131743 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:20.131717    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh_odh-ai-gateway-infra(2b66b5a7-1bea-426f-98ce-9c7ea4975419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419"
Jul 17 17:30:20.133098 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.133079    2561 scope.go:117] "RemoveContainer" containerID="aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398"
Jul 17 17:30:20.133250 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:20.133232    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9_odh-ai-gateway-infra(4b485636-78c1-4695-947f-9513fa5eb0de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" podUID="4b485636-78c1-4695-947f-9513fa5eb0de"
Jul 17 17:30:20.160172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.160147    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:20.508166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:20.508101    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:30:20.508304 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:20.508282    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:30:21.277417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:21.277387    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"
Jul 17 17:30:22.146902 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.146866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh" event={"ID":"2b66b5a7-1bea-426f-98ce-9c7ea4975419","Type":"ContainerDied","Data":"ee4c2b3443856b69ad70819d4abac834d5f652eafed39a79e3bafc97debf500f"}
Jul 17 17:30:22.147091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.146929    2561 scope.go:117] "RemoveContainer" containerID="30ca491c420f2e7ce9c8d7e19f210b5966ae96845c3d434dee61c2111c833a1d"
Jul 17 17:30:22.147091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.146942    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"
Jul 17 17:30:22.147620 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.147592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:22.154437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.154412    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"]
Jul 17 17:30:22.156290 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.156271    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f748a7-29738490-gwktq"]
Jul 17 17:30:22.170837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.170816    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:22.176565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.176546    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a8e6750e-29738490-cw8sh"]
Jul 17 17:30:22.514441 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.514378    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" path="/var/lib/kubelet/pods/2b66b5a7-1bea-426f-98ce-9c7ea4975419/volumes"
Jul 17 17:30:22.514754 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:22.514625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea02c0d6-ff74-4d73-b26f-7c776579d69f" path="/var/lib/kubelet/pods/ea02c0d6-ff74-4d73-b26f-7c776579d69f/volumes"
Jul 17 17:30:23.859950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.859919    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:23.860321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860281    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860293    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860314    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860319    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860326    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860392    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.860459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.860401    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:23.863484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.863468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:23.870283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.870226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:23.998713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:23.998677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln\" (UID: \"79a7c093-3a8e-4207-9639-086e6caa38c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:24.099938 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:24.099913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln\" (UID: \"79a7c093-3a8e-4207-9639-086e6caa38c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:24.100541 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:24.100518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln\" (UID: \"79a7c093-3a8e-4207-9639-086e6caa38c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:24.174744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:24.174694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:24.505638 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:24.505563    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:24.508662 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:24.508635    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a7c093_3a8e_4207_9639_086e6caa38c1.slice/crio-939c71b6af77ba63fc44024a2f2205f2870d8a785db06245a97cba85df7b0e07 WatchSource:0}: Error finding container 939c71b6af77ba63fc44024a2f2205f2870d8a785db06245a97cba85df7b0e07: Status 404 returned error can't find the container with id 939c71b6af77ba63fc44024a2f2205f2870d8a785db06245a97cba85df7b0e07
Jul 17 17:30:25.159166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:25.159140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:30:25.164537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:25.164509    2561 generic.go:356] "Generic (PLEG): container finished" podID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerID="84741640f2abff55c95e9c06b4232028173863b2d9b030547a512477f573c00b" exitCode=7
Jul 17 17:30:25.164635 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:25.164594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" event={"ID":"79a7c093-3a8e-4207-9639-086e6caa38c1","Type":"ContainerDied","Data":"84741640f2abff55c95e9c06b4232028173863b2d9b030547a512477f573c00b"}
Jul 17 17:30:25.164677 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:25.164643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" event={"ID":"79a7c093-3a8e-4207-9639-086e6caa38c1","Type":"ContainerStarted","Data":"939c71b6af77ba63fc44024a2f2205f2870d8a785db06245a97cba85df7b0e07"}
Jul 17 17:30:25.164738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:25.164725    2561 scope.go:117] "RemoveContainer" containerID="84741640f2abff55c95e9c06b4232028173863b2d9b030547a512477f573c00b"
Jul 17 17:30:26.171025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.170935    2561 generic.go:356] "Generic (PLEG): container finished" podID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerID="61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089" exitCode=7
Jul 17 17:30:26.171025 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.171016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" event={"ID":"79a7c093-3a8e-4207-9639-086e6caa38c1","Type":"ContainerDied","Data":"61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089"}
Jul 17 17:30:26.171666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.171121    2561 scope.go:117] "RemoveContainer" containerID="84741640f2abff55c95e9c06b4232028173863b2d9b030547a512477f573c00b"
Jul 17 17:30:26.171666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.171342    2561 scope.go:117] "RemoveContainer" containerID="61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089"
Jul 17 17:30:26.171666 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:26.171542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln_odh-ai-gateway-infra(79a7c093-3a8e-4207-9639-086e6caa38c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1"
Jul 17 17:30:26.675913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.675878    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:26.676127 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.676107    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"]
Jul 17 17:30:26.680631 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:26.680609    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f748a7-d4nh7"]
Jul 17 17:30:27.113208 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:27.113176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:27.177406 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:27.177380    2561 scope.go:117] "RemoveContainer" containerID="61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089"
Jul 17 17:30:27.177799 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:27.177564    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln_odh-ai-gateway-infra(79a7c093-3a8e-4207-9639-086e6caa38c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1"
Jul 17 17:30:27.508551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:27.508471    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:30:27.508719 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:27.508699    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:30:28.514770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:28.514693    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69be4c99-33ed-4a98-8bc5-72a8a6dd1871" path="/var/lib/kubelet/pods/69be4c99-33ed-4a98-8bc5-72a8a6dd1871/volumes"
Jul 17 17:30:31.402124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.402080    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:31.402708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.402687    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b66b5a7-1bea-426f-98ce-9c7ea4975419" containerName="cleanup"
Jul 17 17:30:31.407205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.407185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:31.411975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.411952    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:31.466916 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.466892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27\" (UID: \"0e679ecd-9781-47e6-aa82-a2a767787538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:31.568220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.568194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27\" (UID: \"0e679ecd-9781-47e6-aa82-a2a767787538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:31.568742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.568721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27\" (UID: \"0e679ecd-9781-47e6-aa82-a2a767787538\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:31.719417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.719364    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:31.845442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:31.845400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:31.848778 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:31.848754    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e679ecd_9781_47e6_aa82_a2a767787538.slice/crio-3b440c1400a9983bbef10d5065cb26e531eda40528a6972b4ff42bbbca447410 WatchSource:0}: Error finding container 3b440c1400a9983bbef10d5065cb26e531eda40528a6972b4ff42bbbca447410: Status 404 returned error can't find the container with id 3b440c1400a9983bbef10d5065cb26e531eda40528a6972b4ff42bbbca447410
Jul 17 17:30:32.202851 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:32.202812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerStarted","Data":"0c6915338f0ffb8c787d79c783d2442f0731f6700e0b78b0ff3319252ed246e8"}
Jul 17 17:30:32.203064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:32.202891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerStarted","Data":"3b440c1400a9983bbef10d5065cb26e531eda40528a6972b4ff42bbbca447410"}
Jul 17 17:30:32.222363 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:32.222316    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" podStartSLOduration=1.222301985 podStartE2EDuration="1.222301985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:32.21959759 +0000 UTC m=+3866.204685148" watchObservedRunningTime="2026-07-17 17:30:32.222301985 +0000 UTC m=+3866.207389521"
Jul 17 17:30:32.508379 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:32.508350    2561 scope.go:117] "RemoveContainer" containerID="aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398"
Jul 17 17:30:33.015371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.015291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:33.113679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.113652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:33.209983 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.209941    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e679ecd-9781-47e6-aa82-a2a767787538" containerID="0c6915338f0ffb8c787d79c783d2442f0731f6700e0b78b0ff3319252ed246e8" exitCode=7
Jul 17 17:30:33.210190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.209982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerDied","Data":"0c6915338f0ffb8c787d79c783d2442f0731f6700e0b78b0ff3319252ed246e8"}
Jul 17 17:30:33.210268 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.210255    2561 scope.go:117] "RemoveContainer" containerID="0c6915338f0ffb8c787d79c783d2442f0731f6700e0b78b0ff3319252ed246e8"
Jul 17 17:30:33.211986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.211968    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b485636-78c1-4695-947f-9513fa5eb0de" containerID="37926a995db904a39c20097a157f24b11262ec76fa425f5e713c9fde78490dc4" exitCode=7
Jul 17 17:30:33.212112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.212069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" event={"ID":"4b485636-78c1-4695-947f-9513fa5eb0de","Type":"ContainerDied","Data":"37926a995db904a39c20097a157f24b11262ec76fa425f5e713c9fde78490dc4"}
Jul 17 17:30:33.212164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.212123    2561 scope.go:117] "RemoveContainer" containerID="aef2824a370b5e0ccba3f9022378d4df44e51090a76845f0e60ccc4e0cfb2398"
Jul 17 17:30:33.212552 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.212443    2561 scope.go:117] "RemoveContainer" containerID="37926a995db904a39c20097a157f24b11262ec76fa425f5e713c9fde78490dc4"
Jul 17 17:30:33.212668 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:33.212642    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9_odh-ai-gateway-infra(4b485636-78c1-4695-947f-9513fa5eb0de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" podUID="4b485636-78c1-4695-947f-9513fa5eb0de"
Jul 17 17:30:33.281289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.281253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:33.520300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.520275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:33.852224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:33.852189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:34.221372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.221281    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e679ecd-9781-47e6-aa82-a2a767787538" containerID="56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f" exitCode=7
Jul 17 17:30:34.221545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.221369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerDied","Data":"56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f"}
Jul 17 17:30:34.221545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.221432    2561 scope.go:117] "RemoveContainer" containerID="0c6915338f0ffb8c787d79c783d2442f0731f6700e0b78b0ff3319252ed246e8"
Jul 17 17:30:34.221691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.221637    2561 scope.go:117] "RemoveContainer" containerID="56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f"
Jul 17 17:30:34.221922 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:34.221894    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27_odh-ai-gateway-infra(0e679ecd-9781-47e6-aa82-a2a767787538)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" podUID="0e679ecd-9781-47e6-aa82-a2a767787538"
Jul 17 17:30:34.248844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.248812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:34.262142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.262117    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:34.387198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.387171    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:34.497966 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.497889    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca\") pod \"4b485636-78c1-4695-947f-9513fa5eb0de\" (UID: \"4b485636-78c1-4695-947f-9513fa5eb0de\") "
Jul 17 17:30:34.498254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.498232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b485636-78c1-4695-947f-9513fa5eb0de" (UID: "4b485636-78c1-4695-947f-9513fa5eb0de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:30:34.508326 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.508309    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:30:34.508488 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:34.508469    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:30:34.598884 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.598844    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b485636-78c1-4695-947f-9513fa5eb0de-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:34.793181 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:34.793149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:35.230652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.230563    2561 scope.go:117] "RemoveContainer" containerID="56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f"
Jul 17 17:30:35.230837 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:35.230812    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27_odh-ai-gateway-infra(0e679ecd-9781-47e6-aa82-a2a767787538)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" podUID="0e679ecd-9781-47e6-aa82-a2a767787538"
Jul 17 17:30:35.231504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.231484    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"
Jul 17 17:30:35.231607 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.231483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9" event={"ID":"4b485636-78c1-4695-947f-9513fa5eb0de","Type":"ContainerDied","Data":"0fb560237465069c91dc3871d9640b376df3609d9199ab4ec6fe007eca01fab1"}
Jul 17 17:30:35.231655 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.231606    2561 scope.go:117] "RemoveContainer" containerID="37926a995db904a39c20097a157f24b11262ec76fa425f5e713c9fde78490dc4"
Jul 17 17:30:35.255671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.255645    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:35.261865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:35.261841    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6pvf9"]
Jul 17 17:30:36.514230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:36.514196    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" path="/var/lib/kubelet/pods/4b485636-78c1-4695-947f-9513fa5eb0de/volumes"
Jul 17 17:30:37.546901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.546868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:37.630335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630302    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:30:37.630702 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630688    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.630781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630703    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.630781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630721    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.630781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630727    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.630781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630734    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.630781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630740    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.631034 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630814    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.631034 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.630827    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:37.632891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.632863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:37.635146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.635129    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.637403 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.637384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-876752-serving-cert\""
Jul 17 17:30:37.647129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.647102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:30:37.708587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.708552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:37.729458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.729427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2j2m\" (UniqueName: \"kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.729601 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.729461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.729601 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.729480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.807246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.807165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:37.830559 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.830530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2j2m\" (UniqueName: \"kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.830649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.830567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.830649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.830587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.831296 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.831273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.832878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.832861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.838255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.838234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2j2m\" (UniqueName: \"kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m\") pod \"maas-api-e2e-api-a-876752-79658c58dd-96d6r\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:37.945926 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:37.945895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:38.076787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:38.076759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:30:38.078398 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:38.078371    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87cf333_b8a4_4d55_aa79_9ebfd6381d16.slice/crio-8c923137de1ccb0ce370208fc70ecaa791d5014977f158bb0a8fc0da1577640b WatchSource:0}: Error finding container 8c923137de1ccb0ce370208fc70ecaa791d5014977f158bb0a8fc0da1577640b: Status 404 returned error can't find the container with id 8c923137de1ccb0ce370208fc70ecaa791d5014977f158bb0a8fc0da1577640b
Jul 17 17:30:38.127482 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:38.127451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:38.228034 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:38.228006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:38.248751 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:38.248722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" event={"ID":"e87cf333-b8a4-4d55-aa79-9ebfd6381d16","Type":"ContainerStarted","Data":"8c923137de1ccb0ce370208fc70ecaa791d5014977f158bb0a8fc0da1577640b"}
Jul 17 17:30:39.255420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:39.255387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" event={"ID":"e87cf333-b8a4-4d55-aa79-9ebfd6381d16","Type":"ContainerStarted","Data":"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"}
Jul 17 17:30:39.255818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:39.255507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:39.275510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:39.275464    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" podStartSLOduration=1.860142269 podStartE2EDuration="2.275448906s" totalImagesPullingTime="415.306637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:30:39.273181526 +0000 UTC m=+3873.258269066" watchObservedRunningTime="2026-07-17 17:30:39.275448906 +0000 UTC m=+3873.260536485"
Jul 17 17:30:40.508360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:40.508328    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:30:40.508747 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:40.508410    2561 scope.go:117] "RemoveContainer" containerID="61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089"
Jul 17 17:30:40.508747 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:40.508517    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:30:41.266499 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:41.266465    2561 generic.go:356] "Generic (PLEG): container finished" podID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerID="78f6c36132804609a55a33bab29f8b4233fe94ecf96983fa17afb54a800eeeee" exitCode=7
Jul 17 17:30:41.266675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:41.266535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" event={"ID":"79a7c093-3a8e-4207-9639-086e6caa38c1","Type":"ContainerDied","Data":"78f6c36132804609a55a33bab29f8b4233fe94ecf96983fa17afb54a800eeeee"}
Jul 17 17:30:41.266675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:41.266620    2561 scope.go:117] "RemoveContainer" containerID="61dca565154a79bd55e6460a9a3a276144a7c9abe3b7ebd9a94f8446cee80089"
Jul 17 17:30:41.266937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:41.266920    2561 scope.go:117] "RemoveContainer" containerID="78f6c36132804609a55a33bab29f8b4233fe94ecf96983fa17afb54a800eeeee"
Jul 17 17:30:41.267152 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:41.267133    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln_odh-ai-gateway-infra(79a7c093-3a8e-4207-9639-086e6caa38c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1"
Jul 17 17:30:42.064112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.064081    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:42.291091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.291058    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:42.424574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.424548    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:42.515293 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.515263    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:42.576817 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.576785    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca\") pod \"79a7c093-3a8e-4207-9639-086e6caa38c1\" (UID: \"79a7c093-3a8e-4207-9639-086e6caa38c1\") "
Jul 17 17:30:42.577194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.577170    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79a7c093-3a8e-4207-9639-086e6caa38c1" (UID: "79a7c093-3a8e-4207-9639-086e6caa38c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:30:42.677594 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.677512    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a7c093-3a8e-4207-9639-086e6caa38c1-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:42.827136 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:42.827090    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:43.224162 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.224134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:43.270265 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.270232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:30:43.279731 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.279705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"
Jul 17 17:30:43.279731 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.279724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln" event={"ID":"79a7c093-3a8e-4207-9639-086e6caa38c1","Type":"ContainerDied","Data":"939c71b6af77ba63fc44024a2f2205f2870d8a785db06245a97cba85df7b0e07"}
Jul 17 17:30:43.279965 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.279786    2561 scope.go:117] "RemoveContainer" containerID="78f6c36132804609a55a33bab29f8b4233fe94ecf96983fa17afb54a800eeeee"
Jul 17 17:30:43.308860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.308834    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:43.319373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:43.319351    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-ztqln"]
Jul 17 17:30:44.513744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:44.513713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" path="/var/lib/kubelet/pods/79a7c093-3a8e-4207-9639-086e6caa38c1/volumes"
Jul 17 17:30:45.265862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:45.265837    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:30:45.623199 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:45.623164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:45.727582 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:45.727554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:45.825709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:45.825675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:46.089248 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:46.089211    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:46.392782 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:46.392674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:46.491142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:46.491097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:46.615925 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:46.615889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:46.882616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:46.882582    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:47.508412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:47.508386    2561 scope.go:117] "RemoveContainer" containerID="56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f"
Jul 17 17:30:48.306931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:48.306906    2561 generic.go:356] "Generic (PLEG): container finished" podID="0e679ecd-9781-47e6-aa82-a2a767787538" containerID="c8ea9ab17ab27512184057e92a8dddc3c116294e5ad1d1bfcabd72fe09440838" exitCode=7
Jul 17 17:30:48.307291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:48.306969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerDied","Data":"c8ea9ab17ab27512184057e92a8dddc3c116294e5ad1d1bfcabd72fe09440838"}
Jul 17 17:30:48.307291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:48.307012    2561 scope.go:117] "RemoveContainer" containerID="56adaa86270a1f1e89778b2e9be5c6530ade3c41a6379b50f6bbfd07233dbf7f"
Jul 17 17:30:48.307415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:48.307303    2561 scope.go:117] "RemoveContainer" containerID="c8ea9ab17ab27512184057e92a8dddc3c116294e5ad1d1bfcabd72fe09440838"
Jul 17 17:30:48.307516 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:48.307494    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27_odh-ai-gateway-infra(0e679ecd-9781-47e6-aa82-a2a767787538)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" podUID="0e679ecd-9781-47e6-aa82-a2a767787538"
Jul 17 17:30:49.335202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.335166    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:49.465925 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.465904    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:49.508719 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.508687    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:30:49.508963 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:49.508933    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:30:49.533985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.533959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca\") pod \"0e679ecd-9781-47e6-aa82-a2a767787538\" (UID: \"0e679ecd-9781-47e6-aa82-a2a767787538\") "
Jul 17 17:30:49.534304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.534282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e679ecd-9781-47e6-aa82-a2a767787538" (UID: "0e679ecd-9781-47e6-aa82-a2a767787538"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:30:49.635380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:49.635329    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e679ecd-9781-47e6-aa82-a2a767787538-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:30:50.318258 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.318229    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"
Jul 17 17:30:50.318439 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.318247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27" event={"ID":"0e679ecd-9781-47e6-aa82-a2a767787538","Type":"ContainerDied","Data":"3b440c1400a9983bbef10d5065cb26e531eda40528a6972b4ff42bbbca447410"}
Jul 17 17:30:50.318439 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.318322    2561 scope.go:117] "RemoveContainer" containerID="c8ea9ab17ab27512184057e92a8dddc3c116294e5ad1d1bfcabd72fe09440838"
Jul 17 17:30:50.328525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.328496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:50.360661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.360636    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:50.365766 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.365746    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-6ns27"]
Jul 17 17:30:50.513731 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:50.513707    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" path="/var/lib/kubelet/pods/0e679ecd-9781-47e6-aa82-a2a767787538/volumes"
Jul 17 17:30:54.514443 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:54.514420    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:30:54.514823 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:30:54.514568    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:30:58.227159 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227125    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"]
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227616    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227636    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227649    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227658    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227685    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227694    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.227709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227712    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227720    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227738    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.227748    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228390    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.228492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228414    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.228492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228424    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228433    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a7c093-3a8e-4207-9639-086e6caa38c1" containerName="revoke-keys"
Jul 17 17:30:58.228492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228451    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:30:58.228492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.228460    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b485636-78c1-4695-947f-9513fa5eb0de" containerName="revoke-keys"
Jul 17 17:30:58.232503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.232481    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:30:58.235944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.235829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:30:58.238909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.238883    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"]
Jul 17 17:30:58.303194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.303168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-876752-hgr8f\" (UID: \"a8d8250f-5f49-4bad-a6a5-72a0c44967e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:30:58.403921 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.403895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-876752-hgr8f\" (UID: \"a8d8250f-5f49-4bad-a6a5-72a0c44967e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:30:58.404510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.404490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-876752-hgr8f\" (UID: \"a8d8250f-5f49-4bad-a6a5-72a0c44967e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:30:58.545358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.545334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:30:58.673705 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.673677    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"]
Jul 17 17:30:58.675542 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:30:58.675514    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d8250f_5f49_4bad_a6a5_72a0c44967e9.slice/crio-44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2 WatchSource:0}: Error finding container 44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2: Status 404 returned error can't find the container with id 44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2
Jul 17 17:30:58.738283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.738258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:58.840693 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:58.840631    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:30:59.366578 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:59.366552    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8d8250f-5f49-4bad-a6a5-72a0c44967e9" containerID="1695bc3caf6ef5ed5468a962888b5f4b7e98d2ccdc2ec605978a4a35f7b7717a" exitCode=0
Jul 17 17:30:59.366901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:59.366607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f" event={"ID":"a8d8250f-5f49-4bad-a6a5-72a0c44967e9","Type":"ContainerDied","Data":"1695bc3caf6ef5ed5468a962888b5f4b7e98d2ccdc2ec605978a4a35f7b7717a"}
Jul 17 17:30:59.366901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:30:59.366643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f" event={"ID":"a8d8250f-5f49-4bad-a6a5-72a0c44967e9","Type":"ContainerStarted","Data":"44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2"}
Jul 17 17:31:00.496841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:00.496817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:31:00.522216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:00.522191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca\") pod \"a8d8250f-5f49-4bad-a6a5-72a0c44967e9\" (UID: \"a8d8250f-5f49-4bad-a6a5-72a0c44967e9\") "
Jul 17 17:31:00.522532 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:00.522512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8d8250f-5f49-4bad-a6a5-72a0c44967e9" (UID: "a8d8250f-5f49-4bad-a6a5-72a0c44967e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:00.623839 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:00.623762    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d8250f-5f49-4bad-a6a5-72a0c44967e9-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:01.074488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:01.074448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:31:01.378097 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:01.378005    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"
Jul 17 17:31:01.378097 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:01.378057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f" event={"ID":"a8d8250f-5f49-4bad-a6a5-72a0c44967e9","Type":"ContainerDied","Data":"44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2"}
Jul 17 17:31:01.378297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:01.378106    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44aeb29bc0adfe9b7740a71b95030c149830bfedb53274d6d96b0181e41850f2"
Jul 17 17:31:03.508662 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:03.508632    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:31:03.509075 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:03.508802    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:31:06.478642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.478605    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:06.479003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.478954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:31:06.479003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.478964    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:31:06.479003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.478974    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d8250f-5f49-4bad-a6a5-72a0c44967e9" containerName="revoke-keys"
Jul 17 17:31:06.479003 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.478980    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d8250f-5f49-4bad-a6a5-72a0c44967e9" containerName="revoke-keys"
Jul 17 17:31:06.479157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.479067    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d8250f-5f49-4bad-a6a5-72a0c44967e9" containerName="revoke-keys"
Jul 17 17:31:06.479157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.479075    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e679ecd-9781-47e6-aa82-a2a767787538" containerName="revoke-keys"
Jul 17 17:31:06.482256 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.482238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:06.484291 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.484275    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:31:06.489404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.489088    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:06.577387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.577355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4\" (UID: \"d9402343-4b6b-4dca-b64a-dd958fbb669b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:06.677901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.677865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4\" (UID: \"d9402343-4b6b-4dca-b64a-dd958fbb669b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:06.678585 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.678565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4\" (UID: \"d9402343-4b6b-4dca-b64a-dd958fbb669b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:06.691320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.691294    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:31:06.700814 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.700798    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:31:06.793620 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.793593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:06.924451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:06.924425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:06.926979 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:06.926950    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9402343_4b6b_4dca_b64a_dd958fbb669b.slice/crio-2a4449dd63429ca76c46e32523dd1e1a3d53b798d4fc2f117309eb8e313f1bed WatchSource:0}: Error finding container 2a4449dd63429ca76c46e32523dd1e1a3d53b798d4fc2f117309eb8e313f1bed: Status 404 returned error can't find the container with id 2a4449dd63429ca76c46e32523dd1e1a3d53b798d4fc2f117309eb8e313f1bed
Jul 17 17:31:07.162883 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:07.162854    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:07.267518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:07.267491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:07.408523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:07.408442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerStarted","Data":"1d25c60522b159267379fbd14cae58f9fa9574abd5f80929b424d84be5d84932"}
Jul 17 17:31:07.408645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:07.408534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerStarted","Data":"2a4449dd63429ca76c46e32523dd1e1a3d53b798d4fc2f117309eb8e313f1bed"}
Jul 17 17:31:07.424713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:07.424670    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" podStartSLOduration=1.424656407 podStartE2EDuration="1.424656407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:07.423574266 +0000 UTC m=+3901.408661803" watchObservedRunningTime="2026-07-17 17:31:07.424656407 +0000 UTC m=+3901.409743943"
Jul 17 17:31:08.414780 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:08.414749    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerID="1d25c60522b159267379fbd14cae58f9fa9574abd5f80929b424d84be5d84932" exitCode=7
Jul 17 17:31:08.415216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:08.414832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerDied","Data":"1d25c60522b159267379fbd14cae58f9fa9574abd5f80929b424d84be5d84932"}
Jul 17 17:31:08.415216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:08.414947    2561 scope.go:117] "RemoveContainer" containerID="1d25c60522b159267379fbd14cae58f9fa9574abd5f80929b424d84be5d84932"
Jul 17 17:31:08.888645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:08.888614    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:08.986414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:08.986394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:09.421853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:09.421825    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerID="4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8" exitCode=7
Jul 17 17:31:09.422312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:09.421916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerDied","Data":"4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8"}
Jul 17 17:31:09.422312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:09.422008    2561 scope.go:117] "RemoveContainer" containerID="1d25c60522b159267379fbd14cae58f9fa9574abd5f80929b424d84be5d84932"
Jul 17 17:31:09.422312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:09.422247    2561 scope.go:117] "RemoveContainer" containerID="4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8"
Jul 17 17:31:09.422497 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:09.422472    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4_odh-ai-gateway-infra(d9402343-4b6b-4dca-b64a-dd958fbb669b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b"
Jul 17 17:31:09.508369 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:09.508351    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:31:10.428319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:10.428288    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9" exitCode=1
Jul 17 17:31:10.428743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:10.428361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"}
Jul 17 17:31:10.428743 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:10.428414    2561 scope.go:117] "RemoveContainer" containerID="41cb1381d5a324ad96593a82569c336385b67c63a021832e89404ed0d3806255"
Jul 17 17:31:10.428868 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:10.428744    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:10.428986 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:10.428964    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:10.430193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:10.430162    2561 scope.go:117] "RemoveContainer" containerID="4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8"
Jul 17 17:31:10.430354 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:10.430338    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4_odh-ai-gateway-infra(d9402343-4b6b-4dca-b64a-dd958fbb669b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b"
Jul 17 17:31:12.067756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:12.067722    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:31:12.068178 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:12.068084    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:12.068271 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:12.068250    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:13.903358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:13.903323    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:13.908193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:13.908171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:13.918848 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:13.918825    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:14.045486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.045446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5\" (UID: \"114397bd-1be7-4c26-a243-21baa104c0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:14.131973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.124502    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:31:14.131973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.127409    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" podUID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" containerName="maas-api" containerID="cri-o://39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26" gracePeriod=30
Jul 17 17:31:14.146531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.146501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5\" (UID: \"114397bd-1be7-4c26-a243-21baa104c0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:14.147092 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.147039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5\" (UID: \"114397bd-1be7-4c26-a243-21baa104c0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:14.155087 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.155018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:14.221892 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.221857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:14.384753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.384728    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:31:14.448093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.447991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls\") pod \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") "
Jul 17 17:31:14.448093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.448072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2j2m\" (UniqueName: \"kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m\") pod \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") "
Jul 17 17:31:14.448329 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.448151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca\") pod \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\" (UID: \"e87cf333-b8a4-4d55-aa79-9ebfd6381d16\") "
Jul 17 17:31:14.448561 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.448531    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e87cf333-b8a4-4d55-aa79-9ebfd6381d16" (UID: "e87cf333-b8a4-4d55-aa79-9ebfd6381d16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:14.450166 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.450139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m" (OuterVolumeSpecName: "kube-api-access-l2j2m") pod "e87cf333-b8a4-4d55-aa79-9ebfd6381d16" (UID: "e87cf333-b8a4-4d55-aa79-9ebfd6381d16"). InnerVolumeSpecName "kube-api-access-l2j2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:31:14.450453 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.450432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e87cf333-b8a4-4d55-aa79-9ebfd6381d16" (UID: "e87cf333-b8a4-4d55-aa79-9ebfd6381d16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:31:14.453892 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.453869    2561 generic.go:356] "Generic (PLEG): container finished" podID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" containerID="39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26" exitCode=0
Jul 17 17:31:14.453987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.453911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"
Jul 17 17:31:14.453987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.453939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" event={"ID":"e87cf333-b8a4-4d55-aa79-9ebfd6381d16","Type":"ContainerDied","Data":"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"}
Jul 17 17:31:14.453987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.453978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r" event={"ID":"e87cf333-b8a4-4d55-aa79-9ebfd6381d16","Type":"ContainerDied","Data":"8c923137de1ccb0ce370208fc70ecaa791d5014977f158bb0a8fc0da1577640b"}
Jul 17 17:31:14.454117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.453998    2561 scope.go:117] "RemoveContainer" containerID="39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"
Jul 17 17:31:14.462917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.462900    2561 scope.go:117] "RemoveContainer" containerID="39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"
Jul 17 17:31:14.463176 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:14.463158    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26\": container with ID starting with 39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26 not found: ID does not exist" containerID="39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"
Jul 17 17:31:14.463240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.463185    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26"} err="failed to get container status \"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26\": rpc error: code = NotFound desc = could not find container \"39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26\": container with ID starting with 39ee6121069bf94c047146cc7f04c7bf497def8dd389db456ed1784b04e0db26 not found: ID does not exist"
Jul 17 17:31:14.474910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.474889    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:31:14.479317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.479297    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-876752-79658c58dd-96d6r"]
Jul 17 17:31:14.513759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.513727    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" path="/var/lib/kubelet/pods/e87cf333-b8a4-4d55-aa79-9ebfd6381d16/volumes"
Jul 17 17:31:14.549359 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.549335    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:14.549359 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.549357    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:14.549495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.549367    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2j2m\" (UniqueName: \"kubernetes.io/projected/e87cf333-b8a4-4d55-aa79-9ebfd6381d16-kube-api-access-l2j2m\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:14.578451 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.578428    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:14.580319 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:14.580293    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114397bd_1be7_4c26_a243_21baa104c0d9.slice/crio-3f0033d76156cce0ad07c952af17a86e8820efe43b7dc807e7418849db15edae WatchSource:0}: Error finding container 3f0033d76156cce0ad07c952af17a86e8820efe43b7dc807e7418849db15edae: Status 404 returned error can't find the container with id 3f0033d76156cce0ad07c952af17a86e8820efe43b7dc807e7418849db15edae
Jul 17 17:31:14.969020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.968986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:31:14.969415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:14.969398    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:14.969583 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:14.969564    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:15.461667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:15.461583    2561 generic.go:356] "Generic (PLEG): container finished" podID="114397bd-1be7-4c26-a243-21baa104c0d9" containerID="fd0d0393e842b5e5ee8ea55ea08bef33fd9fea9bf354ca94eba179bf1d03fdd3" exitCode=7
Jul 17 17:31:15.461667 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:15.461650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" event={"ID":"114397bd-1be7-4c26-a243-21baa104c0d9","Type":"ContainerDied","Data":"fd0d0393e842b5e5ee8ea55ea08bef33fd9fea9bf354ca94eba179bf1d03fdd3"}
Jul 17 17:31:15.461874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:15.461690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" event={"ID":"114397bd-1be7-4c26-a243-21baa104c0d9","Type":"ContainerStarted","Data":"3f0033d76156cce0ad07c952af17a86e8820efe43b7dc807e7418849db15edae"}
Jul 17 17:31:15.461915 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:15.461893    2561 scope.go:117] "RemoveContainer" containerID="fd0d0393e842b5e5ee8ea55ea08bef33fd9fea9bf354ca94eba179bf1d03fdd3"
Jul 17 17:31:16.033201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.033175    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:16.033555 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.033543    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" containerName="maas-api"
Jul 17 17:31:16.033555 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.033556    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" containerName="maas-api"
Jul 17 17:31:16.033637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.033625    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e87cf333-b8a4-4d55-aa79-9ebfd6381d16" containerName="maas-api"
Jul 17 17:31:16.036508 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.036490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:16.042917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.042887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:16.163721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.163680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skcb8\" (UID: \"926db0f2-95bc-44d2-85db-816c00510341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:16.264757 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.264716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skcb8\" (UID: \"926db0f2-95bc-44d2-85db-816c00510341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:16.265357 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.265338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-skcb8\" (UID: \"926db0f2-95bc-44d2-85db-816c00510341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:16.386302 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.386262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:16.469932 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.469901    2561 generic.go:356] "Generic (PLEG): container finished" podID="114397bd-1be7-4c26-a243-21baa104c0d9" containerID="03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc" exitCode=7
Jul 17 17:31:16.470112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.469976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" event={"ID":"114397bd-1be7-4c26-a243-21baa104c0d9","Type":"ContainerDied","Data":"03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc"}
Jul 17 17:31:16.470112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.470032    2561 scope.go:117] "RemoveContainer" containerID="fd0d0393e842b5e5ee8ea55ea08bef33fd9fea9bf354ca94eba179bf1d03fdd3"
Jul 17 17:31:16.470279 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.470261    2561 scope.go:117] "RemoveContainer" containerID="03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc"
Jul 17 17:31:16.470510 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:16.470485    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5_odh-ai-gateway-infra(114397bd-1be7-4c26-a243-21baa104c0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" podUID="114397bd-1be7-4c26-a243-21baa104c0d9"
Jul 17 17:31:16.527575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:16.527546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:16.529603 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:16.529573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926db0f2_95bc_44d2_85db_816c00510341.slice/crio-f08d5c65f8fd6c2e9aea4c0374f1f94459b72857dd30fc9db830bebc574730df WatchSource:0}: Error finding container f08d5c65f8fd6c2e9aea4c0374f1f94459b72857dd30fc9db830bebc574730df: Status 404 returned error can't find the container with id f08d5c65f8fd6c2e9aea4c0374f1f94459b72857dd30fc9db830bebc574730df
Jul 17 17:31:17.476861 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.476769    2561 generic.go:356] "Generic (PLEG): container finished" podID="926db0f2-95bc-44d2-85db-816c00510341" containerID="2d1db80207649f82c19209edffa318dc3768484f11533f118ebbc10efbca7294" exitCode=7
Jul 17 17:31:17.477344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.476860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" event={"ID":"926db0f2-95bc-44d2-85db-816c00510341","Type":"ContainerDied","Data":"2d1db80207649f82c19209edffa318dc3768484f11533f118ebbc10efbca7294"}
Jul 17 17:31:17.477344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.476924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" event={"ID":"926db0f2-95bc-44d2-85db-816c00510341","Type":"ContainerStarted","Data":"f08d5c65f8fd6c2e9aea4c0374f1f94459b72857dd30fc9db830bebc574730df"}
Jul 17 17:31:17.477344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.477067    2561 scope.go:117] "RemoveContainer" containerID="2d1db80207649f82c19209edffa318dc3768484f11533f118ebbc10efbca7294"
Jul 17 17:31:17.478727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.478710    2561 scope.go:117] "RemoveContainer" containerID="03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc"
Jul 17 17:31:17.478893 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:17.478877    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5_odh-ai-gateway-infra(114397bd-1be7-4c26-a243-21baa104c0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" podUID="114397bd-1be7-4c26-a243-21baa104c0d9"
Jul 17 17:31:17.509037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:17.509013    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:31:17.509251 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:17.509229    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:31:18.313878 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.313842    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"]
Jul 17 17:31:18.316763 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.316737    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-876752-hgr8f"]
Jul 17 17:31:18.316919 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.316896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:18.485976 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.485893    2561 generic.go:356] "Generic (PLEG): container finished" podID="926db0f2-95bc-44d2-85db-816c00510341" containerID="c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497" exitCode=7
Jul 17 17:31:18.486375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.485982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" event={"ID":"926db0f2-95bc-44d2-85db-816c00510341","Type":"ContainerDied","Data":"c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497"}
Jul 17 17:31:18.486375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.486082    2561 scope.go:117] "RemoveContainer" containerID="2d1db80207649f82c19209edffa318dc3768484f11533f118ebbc10efbca7294"
Jul 17 17:31:18.486375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.486298    2561 scope.go:117] "RemoveContainer" containerID="c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497"
Jul 17 17:31:18.486555 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:18.486531    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skcb8_odh-ai-gateway-infra(926db0f2-95bc-44d2-85db-816c00510341)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" podUID="926db0f2-95bc-44d2-85db-816c00510341"
Jul 17 17:31:18.514195 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.514172    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d8250f-5f49-4bad-a6a5-72a0c44967e9" path="/var/lib/kubelet/pods/a8d8250f-5f49-4bad-a6a5-72a0c44967e9/volumes"
Jul 17 17:31:18.557631 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.557605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:31:18.702680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:18.702647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:19.492934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:19.492904    2561 scope.go:117] "RemoveContainer" containerID="c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497"
Jul 17 17:31:19.493370 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:19.493121    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skcb8_odh-ai-gateway-infra(926db0f2-95bc-44d2-85db-816c00510341)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" podUID="926db0f2-95bc-44d2-85db-816c00510341"
Jul 17 17:31:21.443497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.443456    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:21.448445 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.448419    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:21.453929 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.453899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:21.611519 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.611468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2\" (UID: \"322391a2-9c78-4204-8082-40dfed85c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:21.712612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.712515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2\" (UID: \"322391a2-9c78-4204-8082-40dfed85c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:21.713250 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.713224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2\" (UID: \"322391a2-9c78-4204-8082-40dfed85c49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:21.759515 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.759482    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:21.891593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:21.891557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:21.892862 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:21.892830    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322391a2_9c78_4204_8082_40dfed85c49e.slice/crio-7f5764914a10bb7b69f5ff1b7af457d71a44999040818ce4e0d52f1b9e1dbad6 WatchSource:0}: Error finding container 7f5764914a10bb7b69f5ff1b7af457d71a44999040818ce4e0d52f1b9e1dbad6: Status 404 returned error can't find the container with id 7f5764914a10bb7b69f5ff1b7af457d71a44999040818ce4e0d52f1b9e1dbad6
Jul 17 17:31:22.514069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:22.513962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerStarted","Data":"598af154d6e0c704d7b28e692d6343dfadb57b67292b45b3e9dc8698b8ac27a7"}
Jul 17 17:31:22.514069 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:22.514011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerStarted","Data":"7f5764914a10bb7b69f5ff1b7af457d71a44999040818ce4e0d52f1b9e1dbad6"}
Jul 17 17:31:22.527104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:22.527031    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" podStartSLOduration=1.527013113 podStartE2EDuration="1.527013113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:22.524772286 +0000 UTC m=+3916.509859824" watchObservedRunningTime="2026-07-17 17:31:22.527013113 +0000 UTC m=+3916.512100654"
Jul 17 17:31:23.060438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.060402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:23.159248 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.159205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:23.509108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.508999    2561 scope.go:117] "RemoveContainer" containerID="4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8"
Jul 17 17:31:23.515941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.515915    2561 generic.go:356] "Generic (PLEG): container finished" podID="322391a2-9c78-4204-8082-40dfed85c49e" containerID="598af154d6e0c704d7b28e692d6343dfadb57b67292b45b3e9dc8698b8ac27a7" exitCode=7
Jul 17 17:31:23.516333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.515974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerDied","Data":"598af154d6e0c704d7b28e692d6343dfadb57b67292b45b3e9dc8698b8ac27a7"}
Jul 17 17:31:23.516333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:23.516138    2561 scope.go:117] "RemoveContainer" containerID="598af154d6e0c704d7b28e692d6343dfadb57b67292b45b3e9dc8698b8ac27a7"
Jul 17 17:31:24.522533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.522453    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerID="10123b455a9b1e6dce4c40f83c89044b41e6136ec532aef0b86ba0a18c5f53bb" exitCode=7
Jul 17 17:31:24.522533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.522520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerDied","Data":"10123b455a9b1e6dce4c40f83c89044b41e6136ec532aef0b86ba0a18c5f53bb"}
Jul 17 17:31:24.523065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.522571    2561 scope.go:117] "RemoveContainer" containerID="4265f94977520163c85cbb5b2d1be1002f7e5432da1b037dd3a563f29a652ed8"
Jul 17 17:31:24.523065 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.522822    2561 scope.go:117] "RemoveContainer" containerID="10123b455a9b1e6dce4c40f83c89044b41e6136ec532aef0b86ba0a18c5f53bb"
Jul 17 17:31:24.523203 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:24.523085    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4_odh-ai-gateway-infra(d9402343-4b6b-4dca-b64a-dd958fbb669b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b"
Jul 17 17:31:24.524797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.524778    2561 generic.go:356] "Generic (PLEG): container finished" podID="322391a2-9c78-4204-8082-40dfed85c49e" containerID="f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e" exitCode=7
Jul 17 17:31:24.524896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.524837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerDied","Data":"f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e"}
Jul 17 17:31:24.525023 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.525011    2561 scope.go:117] "RemoveContainer" containerID="f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e"
Jul 17 17:31:24.525231 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:24.525173    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2_odh-ai-gateway-infra(322391a2-9c78-4204-8082-40dfed85c49e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" podUID="322391a2-9c78-4204-8082-40dfed85c49e"
Jul 17 17:31:24.534836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.534819    2561 scope.go:117] "RemoveContainer" containerID="598af154d6e0c704d7b28e692d6343dfadb57b67292b45b3e9dc8698b8ac27a7"
Jul 17 17:31:24.817806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.817764    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:24.890128 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:24.890095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:25.104180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.104103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:25.316890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.316848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:25.434412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.434336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"]
Jul 17 17:31:25.438864 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.438847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:25.444805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.444780    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"]
Jul 17 17:31:25.531546 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.531516    2561 scope.go:117] "RemoveContainer" containerID="f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e"
Jul 17 17:31:25.531958 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:25.531723    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2_odh-ai-gateway-infra(322391a2-9c78-4204-8082-40dfed85c49e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" podUID="322391a2-9c78-4204-8082-40dfed85c49e"
Jul 17 17:31:25.547414 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.547385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47d559-2pq7x\" (UID: \"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:25.551245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.551222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:25.648807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.648774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47d559-2pq7x\" (UID: \"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:25.649592 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.649569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47d559-2pq7x\" (UID: \"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:25.691551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.691495    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:25.751986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.751957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:25.851540 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.851510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca\") pod \"d9402343-4b6b-4dca-b64a-dd958fbb669b\" (UID: \"d9402343-4b6b-4dca-b64a-dd958fbb669b\") "
Jul 17 17:31:25.851917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.851887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9402343-4b6b-4dca-b64a-dd958fbb669b" (UID: "d9402343-4b6b-4dca-b64a-dd958fbb669b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:25.879372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.879351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"]
Jul 17 17:31:25.882019 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:25.881992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d04bb89_3f7f_4f9e_84b1_d10b27771eb0.slice/crio-2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb WatchSource:0}: Error finding container 2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb: Status 404 returned error can't find the container with id 2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb
Jul 17 17:31:25.949532 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.949470    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:25.952669 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:25.952648    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9402343-4b6b-4dca-b64a-dd958fbb669b-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:26.049487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.049459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:26.539418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.539391    2561 generic.go:356] "Generic (PLEG): container finished" podID="0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" containerID="62eff17b12be524f96cea57f3ab2733138616737333dc4d538b1ddb3ada393cf" exitCode=0
Jul 17 17:31:26.539789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.539474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x" event={"ID":"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0","Type":"ContainerDied","Data":"62eff17b12be524f96cea57f3ab2733138616737333dc4d538b1ddb3ada393cf"}
Jul 17 17:31:26.539789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.539537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x" event={"ID":"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0","Type":"ContainerStarted","Data":"2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb"}
Jul 17 17:31:26.540642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.540624    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"
Jul 17 17:31:26.540642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.540634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4" event={"ID":"d9402343-4b6b-4dca-b64a-dd958fbb669b","Type":"ContainerDied","Data":"2a4449dd63429ca76c46e32523dd1e1a3d53b798d4fc2f117309eb8e313f1bed"}
Jul 17 17:31:26.540803 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.540678    2561 scope.go:117] "RemoveContainer" containerID="10123b455a9b1e6dce4c40f83c89044b41e6136ec532aef0b86ba0a18c5f53bb"
Jul 17 17:31:26.572728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.572706    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:26.576515 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:26.576493    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-hbtm4"]
Jul 17 17:31:27.508230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:27.508200    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:27.508412 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:27.508350    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:27.675568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:27.675545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:27.769110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:27.769011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca\") pod \"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0\" (UID: \"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0\") "
Jul 17 17:31:27.769337 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:27.769314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" (UID: "0d04bb89-3f7f-4f9e-84b1-d10b27771eb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:27.870584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:27.870555    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:28.514888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:28.514851    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" path="/var/lib/kubelet/pods/d9402343-4b6b-4dca-b64a-dd958fbb669b/volumes"
Jul 17 17:31:28.551494 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:28.551466    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"
Jul 17 17:31:28.551615 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:28.551462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x" event={"ID":"0d04bb89-3f7f-4f9e-84b1-d10b27771eb0","Type":"ContainerDied","Data":"2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb"}
Jul 17 17:31:28.551615 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:28.551587    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0f3eeb545c2d57dd3741780b2ca7c7aecd9a20e8ac2298f74e2c25988474bb"
Jul 17 17:31:31.508157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:31.508129    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:31:31.508671 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:31.508302    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:31:31.508671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:31.508328    2561 scope.go:117] "RemoveContainer" containerID="03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc"
Jul 17 17:31:32.571557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:32.571520    2561 generic.go:356] "Generic (PLEG): container finished" podID="114397bd-1be7-4c26-a243-21baa104c0d9" containerID="560781c9090351fb1894da390f874e41deee1dbbc45bcd4ca54b8cc9544d4432" exitCode=7
Jul 17 17:31:32.572036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:32.571596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" event={"ID":"114397bd-1be7-4c26-a243-21baa104c0d9","Type":"ContainerDied","Data":"560781c9090351fb1894da390f874e41deee1dbbc45bcd4ca54b8cc9544d4432"}
Jul 17 17:31:32.572036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:32.571659    2561 scope.go:117] "RemoveContainer" containerID="03e16c39364b4688183a1b9b463e1bc2aa5698dd0d9cc55cc7bddb91baaf8cbc"
Jul 17 17:31:32.572036 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:32.572012    2561 scope.go:117] "RemoveContainer" containerID="560781c9090351fb1894da390f874e41deee1dbbc45bcd4ca54b8cc9544d4432"
Jul 17 17:31:32.572290 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:32.572268    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5_odh-ai-gateway-infra(114397bd-1be7-4c26-a243-21baa104c0d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" podUID="114397bd-1be7-4c26-a243-21baa104c0d9"
Jul 17 17:31:33.508352 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.508319    2561 scope.go:117] "RemoveContainer" containerID="c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497"
Jul 17 17:31:33.598421 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.598387    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:33.736775 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.736752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:33.828824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.828776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca\") pod \"114397bd-1be7-4c26-a243-21baa104c0d9\" (UID: \"114397bd-1be7-4c26-a243-21baa104c0d9\") "
Jul 17 17:31:33.829167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.829143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "114397bd-1be7-4c26-a243-21baa104c0d9" (UID: "114397bd-1be7-4c26-a243-21baa104c0d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:33.929863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:33.929822    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/114397bd-1be7-4c26-a243-21baa104c0d9-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:34.585094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.585036    2561 generic.go:356] "Generic (PLEG): container finished" podID="926db0f2-95bc-44d2-85db-816c00510341" containerID="655ea80bec912f3af8b8375fdb7f58bdfc272eb84e05d98bd1783c1d9b19734b" exitCode=7
Jul 17 17:31:34.585277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.585111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" event={"ID":"926db0f2-95bc-44d2-85db-816c00510341","Type":"ContainerDied","Data":"655ea80bec912f3af8b8375fdb7f58bdfc272eb84e05d98bd1783c1d9b19734b"}
Jul 17 17:31:34.585277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.585175    2561 scope.go:117] "RemoveContainer" containerID="c2b984866048eaab7a3caae876dfb5f9e29add0e814b5f513024f9919c9f1497"
Jul 17 17:31:34.585517 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.585488    2561 scope.go:117] "RemoveContainer" containerID="655ea80bec912f3af8b8375fdb7f58bdfc272eb84e05d98bd1783c1d9b19734b"
Jul 17 17:31:34.585760 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:34.585735    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-skcb8_odh-ai-gateway-infra(926db0f2-95bc-44d2-85db-816c00510341)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" podUID="926db0f2-95bc-44d2-85db-816c00510341"
Jul 17 17:31:34.586461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.586438    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"
Jul 17 17:31:34.586552 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.586469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5" event={"ID":"114397bd-1be7-4c26-a243-21baa104c0d9","Type":"ContainerDied","Data":"3f0033d76156cce0ad07c952af17a86e8820efe43b7dc807e7418849db15edae"}
Jul 17 17:31:34.596420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.596393    2561 scope.go:117] "RemoveContainer" containerID="560781c9090351fb1894da390f874e41deee1dbbc45bcd4ca54b8cc9544d4432"
Jul 17 17:31:34.615229 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.615201    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:34.620813 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:34.620786    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-vgnf5"]
Jul 17 17:31:35.609967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:35.609928    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:35.746459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:35.746434    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:35.847375 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:35.847336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca\") pod \"926db0f2-95bc-44d2-85db-816c00510341\" (UID: \"926db0f2-95bc-44d2-85db-816c00510341\") "
Jul 17 17:31:35.847709 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:35.847687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "926db0f2-95bc-44d2-85db-816c00510341" (UID: "926db0f2-95bc-44d2-85db-816c00510341"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:35.948148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:35.948036    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926db0f2-95bc-44d2-85db-816c00510341-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:36.514224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.514193    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" path="/var/lib/kubelet/pods/114397bd-1be7-4c26-a243-21baa104c0d9/volumes"
Jul 17 17:31:36.601942 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.601913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"
Jul 17 17:31:36.601942 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.601919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8" event={"ID":"926db0f2-95bc-44d2-85db-816c00510341","Type":"ContainerDied","Data":"f08d5c65f8fd6c2e9aea4c0374f1f94459b72857dd30fc9db830bebc574730df"}
Jul 17 17:31:36.602227 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.601990    2561 scope.go:117] "RemoveContainer" containerID="655ea80bec912f3af8b8375fdb7f58bdfc272eb84e05d98bd1783c1d9b19734b"
Jul 17 17:31:36.618641 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.618605    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:36.623538 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:36.623514    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-skcb8"]
Jul 17 17:31:38.509030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:38.508993    2561 scope.go:117] "RemoveContainer" containerID="f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e"
Jul 17 17:31:38.514518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:38.514493    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926db0f2-95bc-44d2-85db-816c00510341" path="/var/lib/kubelet/pods/926db0f2-95bc-44d2-85db-816c00510341/volumes"
Jul 17 17:31:39.508748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:39.508654    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:39.508907 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:39.508857    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:39.619283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:39.619249    2561 generic.go:356] "Generic (PLEG): container finished" podID="322391a2-9c78-4204-8082-40dfed85c49e" containerID="00b31c9c514d466cece2555f67973898c81e43b8926ea0b73d69a13999849b3f" exitCode=7
Jul 17 17:31:39.619690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:39.619322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerDied","Data":"00b31c9c514d466cece2555f67973898c81e43b8926ea0b73d69a13999849b3f"}
Jul 17 17:31:39.619690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:39.619378    2561 scope.go:117] "RemoveContainer" containerID="f0ba5f6d72eb3522ea706797560f25da5b73d76380426c629c7e2e7cbf41aa5e"
Jul 17 17:31:39.619690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:39.619681    2561 scope.go:117] "RemoveContainer" containerID="00b31c9c514d466cece2555f67973898c81e43b8926ea0b73d69a13999849b3f"
Jul 17 17:31:39.619929 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:39.619911    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2_odh-ai-gateway-infra(322391a2-9c78-4204-8082-40dfed85c49e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" podUID="322391a2-9c78-4204-8082-40dfed85c49e"
Jul 17 17:31:40.643601 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:40.643569    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:40.772021 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:40.772001    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:40.791495 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:40.791469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca\") pod \"322391a2-9c78-4204-8082-40dfed85c49e\" (UID: \"322391a2-9c78-4204-8082-40dfed85c49e\") "
Jul 17 17:31:40.791778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:40.791756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "322391a2-9c78-4204-8082-40dfed85c49e" (UID: "322391a2-9c78-4204-8082-40dfed85c49e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:31:40.893020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:40.892978    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/322391a2-9c78-4204-8082-40dfed85c49e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:31:41.055165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.055138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:41.632327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.632304    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"
Jul 17 17:31:41.632537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.632295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2" event={"ID":"322391a2-9c78-4204-8082-40dfed85c49e","Type":"ContainerDied","Data":"7f5764914a10bb7b69f5ff1b7af457d71a44999040818ce4e0d52f1b9e1dbad6"}
Jul 17 17:31:41.632537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.632445    2561 scope.go:117] "RemoveContainer" containerID="00b31c9c514d466cece2555f67973898c81e43b8926ea0b73d69a13999849b3f"
Jul 17 17:31:41.654901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.654875    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:41.658682 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:41.658661    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-9jhl2"]
Jul 17 17:31:42.513245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:42.513214    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322391a2-9c78-4204-8082-40dfed85c49e" path="/var/lib/kubelet/pods/322391a2-9c78-4204-8082-40dfed85c49e/volumes"
Jul 17 17:31:45.508492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:45.508457    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:31:45.508940 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:45.508635    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:31:45.513173 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:45.513140    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"]
Jul 17 17:31:45.515793 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:45.515769    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47d559-2pq7x"]
Jul 17 17:31:45.534513 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:45.534486    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:45.930827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:45.930791    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:46.515056 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:46.515011    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" path="/var/lib/kubelet/pods/0d04bb89-3f7f-4f9e-84b1-d10b27771eb0/volumes"
Jul 17 17:31:46.666721 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:46.666682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:31:51.633533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:51.633499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:51.866157 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:51.866123    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:51.965685 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:51.965604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:52.131010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:52.130977    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:52.455033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:52.454996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:52.747494 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:52.747411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:53.190818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:53.190786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:53.508371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:53.508283    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:31:53.508527 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:53.508484    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:31:56.067032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.067003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:56.151913 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.151875    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:56.158916 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.158891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:31:56.159277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159263    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159278    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159288    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159293    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159303    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159308    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159314    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159320    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159333    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159338    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159351    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159356    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159369    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159374    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159382    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159387    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159395    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" containerName="revoke-keys"
Jul 17 17:31:56.159390 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159400    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159462    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159472    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d04bb89-3f7f-4f9e-84b1-d10b27771eb0" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159492    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159498    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159504    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159511    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159521    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159527    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:31:56.159975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.159532    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:31:56.162417 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.162403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.164765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.164740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3d4103-serving-cert\""
Jul 17 17:31:56.173396 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.173376    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:31:56.223230 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.223206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:56.234941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.234914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.235033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.234945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.235033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.234984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8skz\" (UniqueName: \"kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.335608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.335536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.335608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.335570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.335772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.335606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8skz\" (UniqueName: \"kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.336205 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.336188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.337985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.337968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.346957 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.346931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8skz\" (UniqueName: \"kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz\") pod \"maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.391778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.391748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:56.472623 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.472552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:56.607612 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.607535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:31:56.610907 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:31:56.610881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0fe5e6_70e6_4490_87ac_1ddc49b1eb2e.slice/crio-c4c445811c0bccc71eed42a0356f3b94c909220471f5ddd23d8c0a02702b678e WatchSource:0}: Error finding container c4c445811c0bccc71eed42a0356f3b94c909220471f5ddd23d8c0a02702b678e: Status 404 returned error can't find the container with id c4c445811c0bccc71eed42a0356f3b94c909220471f5ddd23d8c0a02702b678e
Jul 17 17:31:56.720581 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.720547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" event={"ID":"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e","Type":"ContainerStarted","Data":"c4c445811c0bccc71eed42a0356f3b94c909220471f5ddd23d8c0a02702b678e"}
Jul 17 17:31:56.839493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.839460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:56.944806 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:56.944730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:31:57.727076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:57.726970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" event={"ID":"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e","Type":"ContainerStarted","Data":"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"}
Jul 17 17:31:57.727618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:57.727580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:31:57.749748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:57.749699    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" podStartSLOduration=1.3343996200000001 podStartE2EDuration="1.749685016s" totalImagesPullingTime="415.285396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:31:57.74816892 +0000 UTC m=+3951.733256460" watchObservedRunningTime="2026-07-17 17:31:57.749685016 +0000 UTC m=+3951.734772554"
Jul 17 17:31:58.508459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:31:58.508377    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:31:58.508655 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:31:58.508560    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:32:00.537446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:00.537399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:00.868203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:00.868102    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:02.243159 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:02.243119    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:32:02.490216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:02.490179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:03.738089 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:03.738037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:32:04.158432 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.158403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:04.263581 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263545    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:32:04.263967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263952    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263968    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="926db0f2-95bc-44d2-85db-816c00510341" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263977    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263982    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263991    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.263996    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.264002    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:32:04.264031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.264007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:32:04.264277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.264079    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="114397bd-1be7-4c26-a243-21baa104c0d9" containerName="revoke-keys"
Jul 17 17:32:04.264277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.264090    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="322391a2-9c78-4204-8082-40dfed85c49e" containerName="revoke-keys"
Jul 17 17:32:04.264277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.264100    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9402343-4b6b-4dca-b64a-dd958fbb669b" containerName="revoke-keys"
Jul 17 17:32:04.267297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.267274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.269810 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.269775    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:04.270970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.270950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4f5e7b-serving-cert\""
Jul 17 17:32:04.279038 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.279014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:32:04.361574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.361541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:04.409809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.409730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.409809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.409767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scm8q\" (UniqueName: \"kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.409979 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.409869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.510511 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.510479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.510738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.510519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scm8q\" (UniqueName: \"kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.510738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.510575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.510738 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:04.510625    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-serving-cert: secret "maas-api-e2e-shared-b-4f5e7b-serving-cert" not found
Jul 17 17:32:04.510738 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:04.510721    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls podName:09655ec8-cbb7-4c6c-87ab-72529b240e02 nodeName:}" failed. No retries permitted until 2026-07-17 17:32:05.01070055 +0000 UTC m=+3958.995788074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls") pod "maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" (UID: "09655ec8-cbb7-4c6c-87ab-72529b240e02") : secret "maas-api-e2e-shared-b-4f5e7b-serving-cert" not found
Jul 17 17:32:04.511304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.511286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.533565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.533532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scm8q\" (UniqueName: \"kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:04.822300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.822266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:04.992403 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:04.992368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:05.016528 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.016484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:05.018853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.018821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:05.092962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.092870    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:05.178078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.178031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:05.314027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.313994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:32:05.315849 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:05.315816    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09655ec8_cbb7_4c6c_87ab_72529b240e02.slice/crio-9522ea39a9cbde59cb1599c9ab6e9601ecefd21b044321c90833404ebe725e4f WatchSource:0}: Error finding container 9522ea39a9cbde59cb1599c9ab6e9601ecefd21b044321c90833404ebe725e4f: Status 404 returned error can't find the container with id 9522ea39a9cbde59cb1599c9ab6e9601ecefd21b044321c90833404ebe725e4f
Jul 17 17:32:05.768904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.768868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" event={"ID":"09655ec8-cbb7-4c6c-87ab-72529b240e02","Type":"ContainerStarted","Data":"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"}
Jul 17 17:32:05.769113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.768917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" event={"ID":"09655ec8-cbb7-4c6c-87ab-72529b240e02","Type":"ContainerStarted","Data":"9522ea39a9cbde59cb1599c9ab6e9601ecefd21b044321c90833404ebe725e4f"}
Jul 17 17:32:05.769113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.769007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:05.785966 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:05.785914    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" podStartSLOduration=1.408439096 podStartE2EDuration="1.785896946s" totalImagesPullingTime="377.45785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:05.784689187 +0000 UTC m=+3959.769776749" watchObservedRunningTime="2026-07-17 17:32:05.785896946 +0000 UTC m=+3959.770984484"
Jul 17 17:32:06.529657 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.529621    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:06.533215 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.533192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:06.535398 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.535373    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:32:06.539502 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.539470    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:06.631372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.631340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl\" (UID: \"27de04b9-16a4-42dc-b56f-923d209a9c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:06.732195 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.732166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl\" (UID: \"27de04b9-16a4-42dc-b56f-923d209a9c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:06.732757 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.732737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl\" (UID: \"27de04b9-16a4-42dc-b56f-923d209a9c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:06.845909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:06.845834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:07.000235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.000211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:07.002206 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:07.002184    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27de04b9_16a4_42dc_b56f_923d209a9c05.slice/crio-794c6615f2f47376a5ded82a1ac55ded537f173aa02bec2f58dc15ec2728bf67 WatchSource:0}: Error finding container 794c6615f2f47376a5ded82a1ac55ded537f173aa02bec2f58dc15ec2728bf67: Status 404 returned error can't find the container with id 794c6615f2f47376a5ded82a1ac55ded537f173aa02bec2f58dc15ec2728bf67
Jul 17 17:32:07.508214 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.508124    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:32:07.508398 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:07.508365    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:32:07.780735 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.780697    2561 generic.go:356] "Generic (PLEG): container finished" podID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerID="68160580290a03fcf94022d467a6b86a14dfbc888e374712af1f1ecf7aa59843" exitCode=7
Jul 17 17:32:07.781187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.780784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerDied","Data":"68160580290a03fcf94022d467a6b86a14dfbc888e374712af1f1ecf7aa59843"}
Jul 17 17:32:07.781187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.780873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerStarted","Data":"794c6615f2f47376a5ded82a1ac55ded537f173aa02bec2f58dc15ec2728bf67"}
Jul 17 17:32:07.781187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:07.780972    2561 scope.go:117] "RemoveContainer" containerID="68160580290a03fcf94022d467a6b86a14dfbc888e374712af1f1ecf7aa59843"
Jul 17 17:32:08.004256 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.004226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:08.623112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.623077    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:08.787908 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.787863    2561 generic.go:356] "Generic (PLEG): container finished" podID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerID="ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a" exitCode=7
Jul 17 17:32:08.788372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.787953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerDied","Data":"ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a"}
Jul 17 17:32:08.788372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.788022    2561 scope.go:117] "RemoveContainer" containerID="68160580290a03fcf94022d467a6b86a14dfbc888e374712af1f1ecf7aa59843"
Jul 17 17:32:08.788372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:08.788159    2561 scope.go:117] "RemoveContainer" containerID="ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a"
Jul 17 17:32:08.788519 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:08.788377    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl_odh-ai-gateway-infra(27de04b9-16a4-42dc-b56f-923d209a9c05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05"
Jul 17 17:32:09.796152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:09.796119    2561 scope.go:117] "RemoveContainer" containerID="ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a"
Jul 17 17:32:09.796536 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:09.796329    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl_odh-ai-gateway-infra(27de04b9-16a4-42dc-b56f-923d209a9c05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05"
Jul 17 17:32:11.779879 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:11.779849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:32:12.508282 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:12.508249    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:32:12.508470 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:12.508434    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:32:16.079187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.079153    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:16.083576 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.083556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:16.110457 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.110429    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:16.218786 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.218749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-x5hmr\" (UID: \"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:16.320245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.320207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-x5hmr\" (UID: \"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:16.320740 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.320718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-x5hmr\" (UID: \"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:16.393396 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.393308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:16.524450 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.524419    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:16.527245 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:16.527215    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd164db32_acc2_4bcf_8dd7_8f78eb8cc8fb.slice/crio-2a470a0cb5559f1d9726fd5189cf8175eb19783767202acc30eeecc5b4012039 WatchSource:0}: Error finding container 2a470a0cb5559f1d9726fd5189cf8175eb19783767202acc30eeecc5b4012039: Status 404 returned error can't find the container with id 2a470a0cb5559f1d9726fd5189cf8175eb19783767202acc30eeecc5b4012039
Jul 17 17:32:16.833575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.833542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerStarted","Data":"9a4e2e1db7844f971a140b4fb965de93ea3bfa197dd3ce626b879d7b996d248f"}
Jul 17 17:32:16.833734 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.833604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerStarted","Data":"2a470a0cb5559f1d9726fd5189cf8175eb19783767202acc30eeecc5b4012039"}
Jul 17 17:32:16.849380 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:16.849328    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" podStartSLOduration=0.849311379 podStartE2EDuration="849.311379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:16.848382798 +0000 UTC m=+3970.833470355" watchObservedRunningTime="2026-07-17 17:32:16.849311379 +0000 UTC m=+3970.834398935"
Jul 17 17:32:17.840315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:17.840279    2561 generic.go:356] "Generic (PLEG): container finished" podID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerID="9a4e2e1db7844f971a140b4fb965de93ea3bfa197dd3ce626b879d7b996d248f" exitCode=7
Jul 17 17:32:17.840732 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:17.840361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerDied","Data":"9a4e2e1db7844f971a140b4fb965de93ea3bfa197dd3ce626b879d7b996d248f"}
Jul 17 17:32:17.840732 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:17.840559    2561 scope.go:117] "RemoveContainer" containerID="9a4e2e1db7844f971a140b4fb965de93ea3bfa197dd3ce626b879d7b996d248f"
Jul 17 17:32:17.870073 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:17.870027    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:17.964360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:17.964327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:18.855029 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:18.854993    2561 generic.go:356] "Generic (PLEG): container finished" podID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerID="650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8" exitCode=7
Jul 17 17:32:18.855550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:18.855080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerDied","Data":"650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8"}
Jul 17 17:32:18.855550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:18.855184    2561 scope.go:117] "RemoveContainer" containerID="9a4e2e1db7844f971a140b4fb965de93ea3bfa197dd3ce626b879d7b996d248f"
Jul 17 17:32:18.855550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:18.855402    2561 scope.go:117] "RemoveContainer" containerID="650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8"
Jul 17 17:32:18.855680 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:18.855628    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-x5hmr_odh-ai-gateway-infra(d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb"
Jul 17 17:32:19.369229 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:19.369179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:32:19.861654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:19.861626    2561 scope.go:117] "RemoveContainer" containerID="650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8"
Jul 17 17:32:19.862073 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:19.861813    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-x5hmr_odh-ai-gateway-infra(d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb"
Jul 17 17:32:20.283076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:20.283026    2561 scope.go:117] "RemoveContainer" containerID="c724f7fbcce6ffc978e311d6e63a9fda95b208f0b9d4cbce12739a0950416294"
Jul 17 17:32:21.490039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.490005    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:21.495295 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.495275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:21.502691 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.502598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:21.566492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.566453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx\" (UID: \"53570f15-6a93-4e63-b8b0-04f5c706bae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:21.667787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.667745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx\" (UID: \"53570f15-6a93-4e63-b8b0-04f5c706bae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:21.668478 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.668449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx\" (UID: \"53570f15-6a93-4e63-b8b0-04f5c706bae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:21.806888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.806852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:21.941012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:21.940985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:21.942713 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:21.942683    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53570f15_6a93_4e63_b8b0_04f5c706bae6.slice/crio-d71a73f1f7f0b363176fe2d37858aa2ba86c3fa5b3cd2147cfacb4b216d8621c WatchSource:0}: Error finding container d71a73f1f7f0b363176fe2d37858aa2ba86c3fa5b3cd2147cfacb4b216d8621c: Status 404 returned error can't find the container with id d71a73f1f7f0b363176fe2d37858aa2ba86c3fa5b3cd2147cfacb4b216d8621c
Jul 17 17:32:22.463622 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.463586    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:32:22.467053 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.467025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.475060 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.475019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3d4103\"/\"default-dockercfg-plhjz\""
Jul 17 17:32:22.475235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.475029    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3d4103\"/\"kube-root-ca.crt\""
Jul 17 17:32:22.475235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.475123    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3d4103\"/\"openshift-service-ca.crt\""
Jul 17 17:32:22.475235 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.475036    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3d4103\"/\"auth-test-model-3d4103-kserve-self-signed-certs\""
Jul 17 17:32:22.486005 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.485984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:32:22.501433 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.501403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:22.508727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.508702    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:32:22.508919 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:22.508894    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:32:22.575860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.575828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.575860 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.575863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.576072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.575884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.576072 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.575941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.576148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.576067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.576182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.576149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxs2r\" (UniqueName: \"kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677464 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs2r\" (UniqueName: \"kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.677759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.678066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.678066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.678066 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.677972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.679831 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.679808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.680083 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.680064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.686297 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.686279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs2r\" (UniqueName: \"kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r\") pod \"auth-test-model-3d4103-kserve-7b8c694467-8p279\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") " pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.775967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.775940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:22.880678 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.880643    2561 generic.go:356] "Generic (PLEG): container finished" podID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerID="d8932f6ff9c957b84eab1797e5b1e4d22c60c7f8cef4f21e2666e558ccd1a8d7" exitCode=7
Jul 17 17:32:22.880867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.880725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerDied","Data":"d8932f6ff9c957b84eab1797e5b1e4d22c60c7f8cef4f21e2666e558ccd1a8d7"}
Jul 17 17:32:22.880867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.880799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerStarted","Data":"d71a73f1f7f0b363176fe2d37858aa2ba86c3fa5b3cd2147cfacb4b216d8621c"}
Jul 17 17:32:22.881019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.880907    2561 scope.go:117] "RemoveContainer" containerID="d8932f6ff9c957b84eab1797e5b1e4d22c60c7f8cef4f21e2666e558ccd1a8d7"
Jul 17 17:32:22.913317 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:22.913292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:32:22.923162 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:22.923125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5023241c_04b4_40c7_ba2a_5facccfbc1c1.slice/crio-204ab72eefeac00071c9f030ec4e52f42c8ba1c07a998b7ee199b7fb6b50b08f WatchSource:0}: Error finding container 204ab72eefeac00071c9f030ec4e52f42c8ba1c07a998b7ee199b7fb6b50b08f: Status 404 returned error can't find the container with id 204ab72eefeac00071c9f030ec4e52f42c8ba1c07a998b7ee199b7fb6b50b08f
Jul 17 17:32:23.888145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.888092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" event={"ID":"5023241c-04b4-40c7-ba2a-5facccfbc1c1","Type":"ContainerStarted","Data":"b4ad242d6d312b4590cba168d42762c2cd04fa2df5b4bb5244f60b4aabcb845a"}
Jul 17 17:32:23.888145 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.888142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" event={"ID":"5023241c-04b4-40c7-ba2a-5facccfbc1c1","Type":"ContainerStarted","Data":"204ab72eefeac00071c9f030ec4e52f42c8ba1c07a998b7ee199b7fb6b50b08f"}
Jul 17 17:32:23.889972 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.889948    2561 generic.go:356] "Generic (PLEG): container finished" podID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerID="93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6" exitCode=7
Jul 17 17:32:23.890091 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.890017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerDied","Data":"93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6"}
Jul 17 17:32:23.890141 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.890104    2561 scope.go:117] "RemoveContainer" containerID="d8932f6ff9c957b84eab1797e5b1e4d22c60c7f8cef4f21e2666e558ccd1a8d7"
Jul 17 17:32:23.890257 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.890237    2561 scope.go:117] "RemoveContainer" containerID="93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6"
Jul 17 17:32:23.890446 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:23.890427    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx_odh-ai-gateway-infra(53570f15-6a93-4e63-b8b0-04f5c706bae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6"
Jul 17 17:32:23.919737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:23.919688    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" podStartSLOduration=1.919673245 podStartE2EDuration="1.919673245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:23.904112727 +0000 UTC m=+3977.889200266" watchObservedRunningTime="2026-07-17 17:32:23.919673245 +0000 UTC m=+3977.904760780"
Jul 17 17:32:24.509064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:24.509009    2561 scope.go:117] "RemoveContainer" containerID="ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a"
Jul 17 17:32:24.896873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:24.896844    2561 scope.go:117] "RemoveContainer" containerID="93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6"
Jul 17 17:32:24.897355 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:24.897115    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx_odh-ai-gateway-infra(53570f15-6a93-4e63-b8b0-04f5c706bae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6"
Jul 17 17:32:24.898562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:24.898539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerStarted","Data":"8fdde961643cbc8edb2b5125f210feae0609286b610ef0ce7a980ad253989165"}
Jul 17 17:32:24.929843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:24.929794    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" podStartSLOduration=18.92977817 podStartE2EDuration="18.92977817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:24.927739852 +0000 UTC m=+3978.912827381" watchObservedRunningTime="2026-07-17 17:32:24.92977817 +0000 UTC m=+3978.914865750"
Jul 17 17:32:25.508624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.508590    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:32:25.508800 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:25.508779    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:32:25.905874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.905839    2561 generic.go:356] "Generic (PLEG): container finished" podID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerID="8fdde961643cbc8edb2b5125f210feae0609286b610ef0ce7a980ad253989165" exitCode=7
Jul 17 17:32:25.906378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.905915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerDied","Data":"8fdde961643cbc8edb2b5125f210feae0609286b610ef0ce7a980ad253989165"}
Jul 17 17:32:25.906378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.905992    2561 scope.go:117] "RemoveContainer" containerID="ac8cc17c5f9d2353b40e5b65f48e43dd091f4b305561170be74d00361b49598a"
Jul 17 17:32:25.906378 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.906360    2561 scope.go:117] "RemoveContainer" containerID="8fdde961643cbc8edb2b5125f210feae0609286b610ef0ce7a980ad253989165"
Jul 17 17:32:25.906628 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:25.906611    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl_odh-ai-gateway-infra(27de04b9-16a4-42dc-b56f-923d209a9c05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05"
Jul 17 17:32:25.937536 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:25.937506    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:26.706420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:26.706384    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:26.808523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:26.808480    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:27.040330 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.040304    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:27.121014 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.120979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca\") pod \"27de04b9-16a4-42dc-b56f-923d209a9c05\" (UID: \"27de04b9-16a4-42dc-b56f-923d209a9c05\") "
Jul 17 17:32:27.121376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.121353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27de04b9-16a4-42dc-b56f-923d209a9c05" (UID: "27de04b9-16a4-42dc-b56f-923d209a9c05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:32:27.222739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.222655    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27de04b9-16a4-42dc-b56f-923d209a9c05-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:32:27.919193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.919163    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"
Jul 17 17:32:27.919193 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.919172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl" event={"ID":"27de04b9-16a4-42dc-b56f-923d209a9c05","Type":"ContainerDied","Data":"794c6615f2f47376a5ded82a1ac55ded537f173aa02bec2f58dc15ec2728bf67"}
Jul 17 17:32:27.919438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.919240    2561 scope.go:117] "RemoveContainer" containerID="8fdde961643cbc8edb2b5125f210feae0609286b610ef0ce7a980ad253989165"
Jul 17 17:32:27.941920 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.941893    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:27.945814 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:27.945793    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-js9tl"]
Jul 17 17:32:28.513894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:28.513861    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" path="/var/lib/kubelet/pods/27de04b9-16a4-42dc-b56f-923d209a9c05/volumes"
Jul 17 17:32:32.776276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:32.776243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:32.776677 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:32.776588    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:32.788897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:32.788872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:32.957780 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:32.957745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:32:33.158807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:33.158726    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:32:34.509027 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:34.508989    2561 scope.go:117] "RemoveContainer" containerID="650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8"
Jul 17 17:32:34.957546 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:34.957508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerStarted","Data":"98f3e20ea090c178e0e6ebf8b93033997d5946581d8e40d38e6e31d7d5e91807"}
Jul 17 17:32:35.964510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:35.964473    2561 generic.go:356] "Generic (PLEG): container finished" podID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerID="98f3e20ea090c178e0e6ebf8b93033997d5946581d8e40d38e6e31d7d5e91807" exitCode=7
Jul 17 17:32:35.964945 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:35.964544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerDied","Data":"98f3e20ea090c178e0e6ebf8b93033997d5946581d8e40d38e6e31d7d5e91807"}
Jul 17 17:32:35.964945 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:35.964606    2561 scope.go:117] "RemoveContainer" containerID="650496b83d1e4c734c9007f0bf904b4d7557337b5204a866ffa4ba682c1de2d8"
Jul 17 17:32:35.964945 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:35.964830    2561 scope.go:117] "RemoveContainer" containerID="98f3e20ea090c178e0e6ebf8b93033997d5946581d8e40d38e6e31d7d5e91807"
Jul 17 17:32:35.965134 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:35.965111    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-x5hmr_odh-ai-gateway-infra(d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb"
Jul 17 17:32:35.984755 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:35.984723    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:36.515437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:36.515402    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:32:36.515762 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:36.515718    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:32:37.126234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.126198    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:37.217613 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.217579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca\") pod \"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb\" (UID: \"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb\") "
Jul 17 17:32:37.217930 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.217906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" (UID: "d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:32:37.319307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.319266    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:32:37.978035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.978006    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"
Jul 17 17:32:37.978254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.978062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr" event={"ID":"d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb","Type":"ContainerDied","Data":"2a470a0cb5559f1d9726fd5189cf8175eb19783767202acc30eeecc5b4012039"}
Jul 17 17:32:37.978254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:37.978123    2561 scope.go:117] "RemoveContainer" containerID="98f3e20ea090c178e0e6ebf8b93033997d5946581d8e40d38e6e31d7d5e91807"
Jul 17 17:32:38.001054 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:38.001003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:38.005935 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:38.005909    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-x5hmr"]
Jul 17 17:32:38.514374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:38.514340    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" path="/var/lib/kubelet/pods/d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb/volumes"
Jul 17 17:32:39.508172 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:39.508143    2561 scope.go:117] "RemoveContainer" containerID="93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6"
Jul 17 17:32:39.991194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:39.991160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerStarted","Data":"9fb9d03b4b4e9da17a7006b92092b4716c27b66e28e8ed97751adc193e81e841"}
Jul 17 17:32:40.008373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:40.008315    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" podStartSLOduration=19.008300177 podStartE2EDuration="19.008300177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:40.006321505 +0000 UTC m=+3993.991409043" watchObservedRunningTime="2026-07-17 17:32:40.008300177 +0000 UTC m=+3993.993387715"
Jul 17 17:32:40.508778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:40.508741    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:32:40.508971 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:40.508900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:32:41.005010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:41.004973    2561 generic.go:356] "Generic (PLEG): container finished" podID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerID="9fb9d03b4b4e9da17a7006b92092b4716c27b66e28e8ed97751adc193e81e841" exitCode=7
Jul 17 17:32:41.005446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:41.005064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerDied","Data":"9fb9d03b4b4e9da17a7006b92092b4716c27b66e28e8ed97751adc193e81e841"}
Jul 17 17:32:41.005446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:41.005151    2561 scope.go:117] "RemoveContainer" containerID="93085c93a6200d674f570aa4216eb231a24dd0c820ae7eedeab474f41d7967a6"
Jul 17 17:32:41.005446 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:41.005404    2561 scope.go:117] "RemoveContainer" containerID="9fb9d03b4b4e9da17a7006b92092b4716c27b66e28e8ed97751adc193e81e841"
Jul 17 17:32:41.005625 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:41.005607    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx_odh-ai-gateway-infra(53570f15-6a93-4e63-b8b0-04f5c706bae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6"
Jul 17 17:32:41.017626 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:41.017605    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:42.145843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:42.145817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:42.263812 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:42.263717    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca\") pod \"53570f15-6a93-4e63-b8b0-04f5c706bae6\" (UID: \"53570f15-6a93-4e63-b8b0-04f5c706bae6\") "
Jul 17 17:32:42.264124 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:42.264100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53570f15-6a93-4e63-b8b0-04f5c706bae6" (UID: "53570f15-6a93-4e63-b8b0-04f5c706bae6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:32:42.364596 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:42.364560    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53570f15-6a93-4e63-b8b0-04f5c706bae6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:32:43.017692 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.017660    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"
Jul 17 17:32:43.017901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.017695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx" event={"ID":"53570f15-6a93-4e63-b8b0-04f5c706bae6","Type":"ContainerDied","Data":"d71a73f1f7f0b363176fe2d37858aa2ba86c3fa5b3cd2147cfacb4b216d8621c"}
Jul 17 17:32:43.017901 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.017762    2561 scope.go:117] "RemoveContainer" containerID="9fb9d03b4b4e9da17a7006b92092b4716c27b66e28e8ed97751adc193e81e841"
Jul 17 17:32:43.033696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.033669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:43.044008 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.043984    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-dfhdx"]
Jul 17 17:32:43.597032 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.596998    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:32:43.597455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597441    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597457    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597470    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597477    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597483    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597493    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597504 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597510    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597523    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597528    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597538    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597556    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597607    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597616    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597623    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597631    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597637    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:32:43.597733 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.597643    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:32:43.600674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.600657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.603376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.603349    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"openshift-service-ca.crt\""
Jul 17 17:32:43.603688 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.603667    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"default-dockercfg-q4tsr\""
Jul 17 17:32:43.604222 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.604205    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"kube-root-ca.crt\""
Jul 17 17:32:43.604496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.604482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"auth-test-model-4f5e7b-kserve-self-signed-certs\""
Jul 17 17:32:43.612374 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.612353    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:32:43.628646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.628604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:32:43.675568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.675746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.675746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n295\" (UniqueName: \"kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.675746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.675746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.675746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.675706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n295\" (UniqueName: \"kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777401 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777619 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.777734 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.777708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.779484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.779463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.779759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.779743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.785479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.785457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n295\" (UniqueName: \"kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295\") pod \"auth-test-model-4f5e7b-kserve-847db79f9-bzj2v\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") " pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:43.911890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:43.911795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:44.045866 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:44.045843    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:32:44.047655 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:32:44.047626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a6c2e9_3d52_413d_9617_f038377227d5.slice/crio-d660b12095eb5920d6931b934ccb79ef8c5bb6bece1fa508dc0a259ca7e69142 WatchSource:0}: Error finding container d660b12095eb5920d6931b934ccb79ef8c5bb6bece1fa508dc0a259ca7e69142: Status 404 returned error can't find the container with id d660b12095eb5920d6931b934ccb79ef8c5bb6bece1fa508dc0a259ca7e69142
Jul 17 17:32:44.514781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:44.514748    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" path="/var/lib/kubelet/pods/53570f15-6a93-4e63-b8b0-04f5c706bae6/volumes"
Jul 17 17:32:45.031473 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:45.031442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" event={"ID":"00a6c2e9-3d52-413d-9617-f038377227d5","Type":"ContainerStarted","Data":"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"}
Jul 17 17:32:45.031881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:45.031488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" event={"ID":"00a6c2e9-3d52-413d-9617-f038377227d5","Type":"ContainerStarted","Data":"d660b12095eb5920d6931b934ccb79ef8c5bb6bece1fa508dc0a259ca7e69142"}
Jul 17 17:32:45.053722 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:45.053673    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" podStartSLOduration=2.053656858 podStartE2EDuration="2.053656858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:32:45.053115444 +0000 UTC m=+3999.038202976" watchObservedRunningTime="2026-07-17 17:32:45.053656858 +0000 UTC m=+3999.038744396"
Jul 17 17:32:47.508581 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:47.508548    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:32:47.508993 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:47.508740    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:32:47.771636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:47.771549    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:32:53.913006 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:53.912959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:53.913530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:53.913020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:53.925756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:53.925728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:54.086824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:54.086794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:32:55.509220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:32:55.509187    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:32:55.509626 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:32:55.509379    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:02.509346 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:02.509308    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:33:02.509827 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:02.509535    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:33:06.066588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:06.066551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:33:06.511174 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:06.511072    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:33:06.511591 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:06.511552    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:13.509167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:13.509133    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:33:13.509574 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:13.509330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:33:14.003246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003207    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:14.003622 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:33:14.003672 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003623    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:33:14.003712 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003689    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53570f15-6a93-4e63-b8b0-04f5c706bae6" containerName="revoke-keys"
Jul 17 17:33:14.003712 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003699    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d164db32-acc2-4bcf-8dd7-8f78eb8cc8fb" containerName="revoke-keys"
Jul 17 17:33:14.003712 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.003706    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27de04b9-16a4-42dc-b56f-923d209a9c05" containerName="revoke-keys"
Jul 17 17:33:14.007216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.007199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:14.009381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.009357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:33:14.012887 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.012863    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:14.061761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.061729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd\" (UID: \"91a4ede5-0ea8-4b84-95b0-084f7532b587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:14.162840 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.162802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd\" (UID: \"91a4ede5-0ea8-4b84-95b0-084f7532b587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:14.163530 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.163509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd\" (UID: \"91a4ede5-0ea8-4b84-95b0-084f7532b587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:14.321064 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.321010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:14.452902 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:14.452872    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:14.455471 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:33:14.455439    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a4ede5_0ea8_4b84_95b0_084f7532b587.slice/crio-6bb2fd1b820eba284f26c3c6440db032b1edc1b48c7ed53ed07988b82c70c619 WatchSource:0}: Error finding container 6bb2fd1b820eba284f26c3c6440db032b1edc1b48c7ed53ed07988b82c70c619: Status 404 returned error can't find the container with id 6bb2fd1b820eba284f26c3c6440db032b1edc1b48c7ed53ed07988b82c70c619
Jul 17 17:33:15.187243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:15.187206    2561 generic.go:356] "Generic (PLEG): container finished" podID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerID="4eedb27a0b9a2b0edabf7d573cef4413d0e979d28209f34295e5ac17dd60092a" exitCode=7
Jul 17 17:33:15.187675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:15.187282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" event={"ID":"91a4ede5-0ea8-4b84-95b0-084f7532b587","Type":"ContainerDied","Data":"4eedb27a0b9a2b0edabf7d573cef4413d0e979d28209f34295e5ac17dd60092a"}
Jul 17 17:33:15.187675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:15.187337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" event={"ID":"91a4ede5-0ea8-4b84-95b0-084f7532b587","Type":"ContainerStarted","Data":"6bb2fd1b820eba284f26c3c6440db032b1edc1b48c7ed53ed07988b82c70c619"}
Jul 17 17:33:15.187675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:15.187507    2561 scope.go:117] "RemoveContainer" containerID="4eedb27a0b9a2b0edabf7d573cef4413d0e979d28209f34295e5ac17dd60092a"
Jul 17 17:33:16.194167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:16.194077    2561 generic.go:356] "Generic (PLEG): container finished" podID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerID="63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928" exitCode=7
Jul 17 17:33:16.194593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:16.194167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" event={"ID":"91a4ede5-0ea8-4b84-95b0-084f7532b587","Type":"ContainerDied","Data":"63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928"}
Jul 17 17:33:16.194593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:16.194220    2561 scope.go:117] "RemoveContainer" containerID="4eedb27a0b9a2b0edabf7d573cef4413d0e979d28209f34295e5ac17dd60092a"
Jul 17 17:33:16.194593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:16.194332    2561 scope.go:117] "RemoveContainer" containerID="63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928"
Jul 17 17:33:16.194593 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:16.194559    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd_odh-ai-gateway-infra(91a4ede5-0ea8-4b84-95b0-084f7532b587)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587"
Jul 17 17:33:17.201911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:17.201884    2561 scope.go:117] "RemoveContainer" containerID="63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928"
Jul 17 17:33:17.202328 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:17.202103    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd_odh-ai-gateway-infra(91a4ede5-0ea8-4b84-95b0-084f7532b587)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587"
Jul 17 17:33:19.508567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:19.508532    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:33:19.509096 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:19.508733    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:22.768393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:22.768342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:33:27.508316 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:27.508279    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:33:27.508807 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:27.508471    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:33:30.514510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:30.514433    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:33:30.514890 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:30.514630    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:32.508908 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:32.508869    2561 scope.go:117] "RemoveContainer" containerID="63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928"
Jul 17 17:33:33.293007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:33.292970    2561 generic.go:356] "Generic (PLEG): container finished" podID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerID="fe29d4632affc7955eb0c6c938e616f85974e80fbb0fd952985d09e27c402e63" exitCode=7
Jul 17 17:33:33.293272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:33.293057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" event={"ID":"91a4ede5-0ea8-4b84-95b0-084f7532b587","Type":"ContainerDied","Data":"fe29d4632affc7955eb0c6c938e616f85974e80fbb0fd952985d09e27c402e63"}
Jul 17 17:33:33.293272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:33.293122    2561 scope.go:117] "RemoveContainer" containerID="63a1340f3e4ced73f6ed55e3b181d99bd83b39619cd5305ffc32b7da8c1ad928"
Jul 17 17:33:33.293463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:33.293438    2561 scope.go:117] "RemoveContainer" containerID="fe29d4632affc7955eb0c6c938e616f85974e80fbb0fd952985d09e27c402e63"
Jul 17 17:33:33.293683 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:33.293661    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd_odh-ai-gateway-infra(91a4ede5-0ea8-4b84-95b0-084f7532b587)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587"
Jul 17 17:33:34.319914 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:34.319884    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:34.453895 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:34.453871    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:34.573679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:34.573583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca\") pod \"91a4ede5-0ea8-4b84-95b0-084f7532b587\" (UID: \"91a4ede5-0ea8-4b84-95b0-084f7532b587\") "
Jul 17 17:33:34.573987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:34.573965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91a4ede5-0ea8-4b84-95b0-084f7532b587" (UID: "91a4ede5-0ea8-4b84-95b0-084f7532b587"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:33:34.674763 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:34.674718    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a4ede5-0ea8-4b84-95b0-084f7532b587-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:33:35.306084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:35.306031    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"
Jul 17 17:33:35.306084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:35.306070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd" event={"ID":"91a4ede5-0ea8-4b84-95b0-084f7532b587","Type":"ContainerDied","Data":"6bb2fd1b820eba284f26c3c6440db032b1edc1b48c7ed53ed07988b82c70c619"}
Jul 17 17:33:35.306336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:35.306133    2561 scope.go:117] "RemoveContainer" containerID="fe29d4632affc7955eb0c6c938e616f85974e80fbb0fd952985d09e27c402e63"
Jul 17 17:33:35.327930 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:35.327894    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:35.334693 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:35.334664    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-78pjd"]
Jul 17 17:33:36.519194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:36.519155    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" path="/var/lib/kubelet/pods/91a4ede5-0ea8-4b84-95b0-084f7532b587/volumes"
Jul 17 17:33:38.508689 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:38.508656    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:33:38.509121 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:38.508855    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:33:38.761600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:38.761512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:33:42.509234 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:42.509185    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:33:42.509632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:42.509528    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:33:43.353389 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:43.353358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerStarted","Data":"7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"}
Jul 17 17:33:43.353611 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:43.353592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:33:44.360568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:44.360531    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e" exitCode=1
Jul 17 17:33:44.361059 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:44.360603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"}
Jul 17 17:33:44.361059 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:44.360660    2561 scope.go:117] "RemoveContainer" containerID="836133477a1d786ffe3feec6d1d075dc66fe7d343d65838f00ae1f0a92865158"
Jul 17 17:33:44.361059 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:44.360877    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:33:44.361202 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:44.361163    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:45.368862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:45.368835    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:33:45.369274 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:45.369035    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:47.508402 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:47.508360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:33:47.508825 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:47.508747    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:33:47.508955 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:47.508933    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:33:53.508900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:53.508864    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:33:53.509376 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:53.509100    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:33:54.967137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:54.967101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:33:59.509194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:33:59.509160    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:33:59.509688 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:33:59.509330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:34:04.508490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:04.508457    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:34:04.508908 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:04.508648    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:34:06.164785 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.164754    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165138    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165150    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165174    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165189    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165207 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165195    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165254    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165263    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.165407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.165270    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a4ede5-0ea8-4b84-95b0-084f7532b587" containerName="revoke-keys"
Jul 17 17:34:06.168467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.168448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:06.170761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.170740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:34:06.177770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.177747    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:06.280741 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.280700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-cl72t\" (UID: \"0af3f3ad-a07a-4d53-b205-c6a96b6c6804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:06.381830 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.381789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-cl72t\" (UID: \"0af3f3ad-a07a-4d53-b205-c6a96b6c6804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:06.382430 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.382409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-cl72t\" (UID: \"0af3f3ad-a07a-4d53-b205-c6a96b6c6804\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:06.482784 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.482697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:34:06.490674 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.490637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:06.627271 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:06.627245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:06.629144 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:34:06.629113    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af3f3ad_a07a_4d53_b205_c6a96b6c6804.slice/crio-8683b63e334681f66de61e6194ab0b152c3f9a6885e2e9057b7918cccb8dc207 WatchSource:0}: Error finding container 8683b63e334681f66de61e6194ab0b152c3f9a6885e2e9057b7918cccb8dc207: Status 404 returned error can't find the container with id 8683b63e334681f66de61e6194ab0b152c3f9a6885e2e9057b7918cccb8dc207
Jul 17 17:34:07.479755 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:07.479658    2561 generic.go:356] "Generic (PLEG): container finished" podID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerID="2bb7001a4d462a3fd602eca808c42d13b4d97509fb34698580b61dc1b628c5d9" exitCode=7
Jul 17 17:34:07.479755 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:07.479733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" event={"ID":"0af3f3ad-a07a-4d53-b205-c6a96b6c6804","Type":"ContainerDied","Data":"2bb7001a4d462a3fd602eca808c42d13b4d97509fb34698580b61dc1b628c5d9"}
Jul 17 17:34:07.480336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:07.479784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" event={"ID":"0af3f3ad-a07a-4d53-b205-c6a96b6c6804","Type":"ContainerStarted","Data":"8683b63e334681f66de61e6194ab0b152c3f9a6885e2e9057b7918cccb8dc207"}
Jul 17 17:34:07.480336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:07.479943    2561 scope.go:117] "RemoveContainer" containerID="2bb7001a4d462a3fd602eca808c42d13b4d97509fb34698580b61dc1b628c5d9"
Jul 17 17:34:08.487113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:08.487013    2561 generic.go:356] "Generic (PLEG): container finished" podID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerID="f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30" exitCode=7
Jul 17 17:34:08.487113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:08.487088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" event={"ID":"0af3f3ad-a07a-4d53-b205-c6a96b6c6804","Type":"ContainerDied","Data":"f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30"}
Jul 17 17:34:08.487608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:08.487164    2561 scope.go:117] "RemoveContainer" containerID="2bb7001a4d462a3fd602eca808c42d13b4d97509fb34698580b61dc1b628c5d9"
Jul 17 17:34:08.487608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:08.487349    2561 scope.go:117] "RemoveContainer" containerID="f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30"
Jul 17 17:34:08.487608 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:08.487591    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-cl72t_odh-ai-gateway-infra(0af3f3ad-a07a-4d53-b205-c6a96b6c6804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804"
Jul 17 17:34:09.494168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:09.494131    2561 scope.go:117] "RemoveContainer" containerID="f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30"
Jul 17 17:34:09.494644 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:09.494415    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-cl72t_odh-ai-gateway-infra(0af3f3ad-a07a-4d53-b205-c6a96b6c6804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804"
Jul 17 17:34:13.508591 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:13.508550    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:34:13.509069 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:13.508822    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:34:14.158162 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:14.158120    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:34:15.508363 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:15.508328    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:34:15.508776 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:15.508505    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:34:24.514778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:24.514738    2561 scope.go:117] "RemoveContainer" containerID="f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30"
Jul 17 17:34:25.509231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:25.509131    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:34:25.509405 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:25.509370    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:34:25.577022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:25.576987    2561 generic.go:356] "Generic (PLEG): container finished" podID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerID="a1dcc65937f3977ede43e979a4d8752c50fff545068c8bdcb5264f76aa484253" exitCode=7
Jul 17 17:34:25.577485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:25.577084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" event={"ID":"0af3f3ad-a07a-4d53-b205-c6a96b6c6804","Type":"ContainerDied","Data":"a1dcc65937f3977ede43e979a4d8752c50fff545068c8bdcb5264f76aa484253"}
Jul 17 17:34:25.577485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:25.577150    2561 scope.go:117] "RemoveContainer" containerID="f6852e58e01005dc3a52291ed850c1627e221781cae3cbffcc8967a82f835d30"
Jul 17 17:34:25.577485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:25.577460    2561 scope.go:117] "RemoveContainer" containerID="a1dcc65937f3977ede43e979a4d8752c50fff545068c8bdcb5264f76aa484253"
Jul 17 17:34:25.577732 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:25.577711    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-cl72t_odh-ai-gateway-infra(0af3f3ad-a07a-4d53-b205-c6a96b6c6804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804"
Jul 17 17:34:26.511490 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.511455    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:34:26.511707 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:26.511641    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:34:26.602954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.602919    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:26.738200 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.738175    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:26.771131 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.771037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca\") pod \"0af3f3ad-a07a-4d53-b205-c6a96b6c6804\" (UID: \"0af3f3ad-a07a-4d53-b205-c6a96b6c6804\") "
Jul 17 17:34:26.771386 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.771363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0af3f3ad-a07a-4d53-b205-c6a96b6c6804" (UID: "0af3f3ad-a07a-4d53-b205-c6a96b6c6804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:34:26.872223 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:26.872177    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af3f3ad-a07a-4d53-b205-c6a96b6c6804-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:34:27.590286 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:27.590258    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"
Jul 17 17:34:27.590444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:27.590298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t" event={"ID":"0af3f3ad-a07a-4d53-b205-c6a96b6c6804","Type":"ContainerDied","Data":"8683b63e334681f66de61e6194ab0b152c3f9a6885e2e9057b7918cccb8dc207"}
Jul 17 17:34:27.590444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:27.590361    2561 scope.go:117] "RemoveContainer" containerID="a1dcc65937f3977ede43e979a4d8752c50fff545068c8bdcb5264f76aa484253"
Jul 17 17:34:27.613551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:27.613520    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:27.617776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:27.617753    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-cl72t"]
Jul 17 17:34:28.514514 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:28.514478    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" path="/var/lib/kubelet/pods/0af3f3ad-a07a-4d53-b205-c6a96b6c6804/volumes"
Jul 17 17:34:31.478243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:31.478197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:34.469962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:34.465866    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:34:36.583278 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:36.583241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:36.683909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:36.683868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:37.508507 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:37.508471    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:34:37.508719 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:37.508680    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:34:38.508985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:38.508948    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:34:38.509456 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:38.509189    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:34:44.720716 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:44.720679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:49.268589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:49.268546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:34:51.508200 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:51.508161    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:34:51.508672 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:51.508468    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:34:53.509360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:53.509326    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:34:53.509784 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:34:53.509590    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:34:55.599774 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:55.599733    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:34:55.634448 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:55.634412    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:56.474114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:56.474076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:34:56.491974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:56.491940    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:59.005366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:34:59.005744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005642    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005671    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005676    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005756    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.005967 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.005764    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:34:59.008755 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.008733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.011276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.011245    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3d4103\"/\"sub-test-model-3d4103-kserve-self-signed-certs\""
Jul 17 17:34:59.018613 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.018585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:34:59.034438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.034402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:34:59.088881 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.088840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.089118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.088913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.089118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.088942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.089118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.088979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.089118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.089007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd77b\" (UniqueName: \"kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.089118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.089079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd77b\" (UniqueName: \"kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.189986 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.189880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.190308 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.190245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.190349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.190306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.190349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.190303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.192307 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.192289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.192478 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.192461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.198020 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.197991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd77b\" (UniqueName: \"kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b\") pod \"sub-test-model-3d4103-kserve-79f4457489-bvdjc\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") " pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.319996 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.319948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:34:59.455452 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.455428    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:34:59.457387 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:34:59.457356    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27db4d04_1a12_4bc1_90e8_101effaf0eac.slice/crio-e7c5da3536317a16c2be93908b9da1cc5c22d38360d0792ce36e2463a1322006 WatchSource:0}: Error finding container e7c5da3536317a16c2be93908b9da1cc5c22d38360d0792ce36e2463a1322006: Status 404 returned error can't find the container with id e7c5da3536317a16c2be93908b9da1cc5c22d38360d0792ce36e2463a1322006
Jul 17 17:34:59.759481 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.759371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" event={"ID":"27db4d04-1a12-4bc1-90e8-101effaf0eac","Type":"ContainerStarted","Data":"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"}
Jul 17 17:34:59.759481 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.759424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" event={"ID":"27db4d04-1a12-4bc1-90e8-101effaf0eac","Type":"ContainerStarted","Data":"e7c5da3536317a16c2be93908b9da1cc5c22d38360d0792ce36e2463a1322006"}
Jul 17 17:34:59.776363 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:34:59.776301    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" podStartSLOduration=1.776283153 podStartE2EDuration="1.776283153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:34:59.775791498 +0000 UTC m=+4133.760879037" watchObservedRunningTime="2026-07-17 17:34:59.776283153 +0000 UTC m=+4133.761370691"
Jul 17 17:35:04.509201 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:04.509161    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:35:04.509713 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:04.509454    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:35:06.511231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:06.511195    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:35:06.511705 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:06.511385    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:35:09.320890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:09.320846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:35:09.320890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:09.320900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:35:09.333834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:09.333808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:35:09.821990 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:09.821961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:35:10.621981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:10.621941    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" podUID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" containerName="main" containerID="cri-o://b4ad242d6d312b4590cba168d42762c2cd04fa2df5b4bb5244f60b4aabcb845a" gracePeriod=45
Jul 17 17:35:10.817126 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:10.817080    2561 generic.go:356] "Generic (PLEG): container finished" podID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" containerID="b4ad242d6d312b4590cba168d42762c2cd04fa2df5b4bb5244f60b4aabcb845a" exitCode=0
Jul 17 17:35:10.817315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:10.817143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" event={"ID":"5023241c-04b4-40c7-ba2a-5facccfbc1c1","Type":"ContainerDied","Data":"b4ad242d6d312b4590cba168d42762c2cd04fa2df5b4bb5244f60b4aabcb845a"}
Jul 17 17:35:10.870248 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:10.870220    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:35:11.010805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.010805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.011094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010805    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxs2r\" (UniqueName: \"kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.011094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010842    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.011094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.011094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.010964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs\") pod \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\" (UID: \"5023241c-04b4-40c7-ba2a-5facccfbc1c1\") "
Jul 17 17:35:11.011264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home" (OuterVolumeSpecName: "home") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.011264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011035    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.011264 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011163    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.011415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011340    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.011415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011365    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.011415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.011376    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.013150 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.013112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r" (OuterVolumeSpecName: "kube-api-access-kxs2r") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "kube-api-access-kxs2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:35:11.013285 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.013188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:35:11.013285 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.013187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm" (OuterVolumeSpecName: "dshm") pod "5023241c-04b4-40c7-ba2a-5facccfbc1c1" (UID: "5023241c-04b4-40c7-ba2a-5facccfbc1c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.112783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.112740    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5023241c-04b4-40c7-ba2a-5facccfbc1c1-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.112783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.112775    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxs2r\" (UniqueName: \"kubernetes.io/projected/5023241c-04b4-40c7-ba2a-5facccfbc1c1-kube-api-access-kxs2r\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.112783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.112786    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5023241c-04b4-40c7-ba2a-5facccfbc1c1-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.495126 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.495058    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" podUID="00a6c2e9-3d52-413d-9617-f038377227d5" containerName="main" containerID="cri-o://1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6" gracePeriod=45
Jul 17 17:35:11.757897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.757827    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:35:11.819944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.819911    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n295\" (UniqueName: \"kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820117    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820395 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820154    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home\") pod \"00a6c2e9-3d52-413d-9617-f038377227d5\" (UID: \"00a6c2e9-3d52-413d-9617-f038377227d5\") "
Jul 17 17:35:11.820395 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.820512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.820512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820467    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.820649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.820566    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home" (OuterVolumeSpecName: "home") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.822484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.822455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm" (OuterVolumeSpecName: "dshm") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:35:11.822636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.822615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295" (OuterVolumeSpecName: "kube-api-access-6n295") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "kube-api-access-6n295". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:35:11.822707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.822632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00a6c2e9-3d52-413d-9617-f038377227d5" (UID: "00a6c2e9-3d52-413d-9617-f038377227d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:35:11.823727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.823710    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"
Jul 17 17:35:11.823727 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.823717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279" event={"ID":"5023241c-04b4-40c7-ba2a-5facccfbc1c1","Type":"ContainerDied","Data":"204ab72eefeac00071c9f030ec4e52f42c8ba1c07a998b7ee199b7fb6b50b08f"}
Jul 17 17:35:11.823870 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.823764    2561 scope.go:117] "RemoveContainer" containerID="b4ad242d6d312b4590cba168d42762c2cd04fa2df5b4bb5244f60b4aabcb845a"
Jul 17 17:35:11.828872 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.828763    2561 generic.go:356] "Generic (PLEG): container finished" podID="00a6c2e9-3d52-413d-9617-f038377227d5" containerID="1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6" exitCode=0
Jul 17 17:35:11.828872 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.828812    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"
Jul 17 17:35:11.828872 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.828864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" event={"ID":"00a6c2e9-3d52-413d-9617-f038377227d5","Type":"ContainerDied","Data":"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"}
Jul 17 17:35:11.829820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.828899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v" event={"ID":"00a6c2e9-3d52-413d-9617-f038377227d5","Type":"ContainerDied","Data":"d660b12095eb5920d6931b934ccb79ef8c5bb6bece1fa508dc0a259ca7e69142"}
Jul 17 17:35:11.844033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.844015    2561 scope.go:117] "RemoveContainer" containerID="1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"
Jul 17 17:35:11.852779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.852758    2561 scope.go:117] "RemoveContainer" containerID="1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"
Jul 17 17:35:11.853091 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:11.853070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6\": container with ID starting with 1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6 not found: ID does not exist" containerID="1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"
Jul 17 17:35:11.853154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.853101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6"} err="failed to get container status \"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6\": rpc error: code = NotFound desc = could not find container \"1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6\": container with ID starting with 1eceea3b0ce00dc14b7c2f0d9ef771fa32c3219242ad93f143ce73ecfd6cb7f6 not found: ID does not exist"
Jul 17 17:35:11.856708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.856679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:35:11.863319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.863295    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/auth-test-model-3d4103-kserve-7b8c694467-8p279"]
Jul 17 17:35:11.873325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.873300    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:35:11.876778 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.876752    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/auth-test-model-4f5e7b-kserve-847db79f9-bzj2v"]
Jul 17 17:35:11.921437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.921399    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.921437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.921430    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a6c2e9-3d52-413d-9617-f038377227d5-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.921437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.921442    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n295\" (UniqueName: \"kubernetes.io/projected/00a6c2e9-3d52-413d-9617-f038377227d5-kube-api-access-6n295\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.921678 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.921457    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:11.921678 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:11.921470    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a6c2e9-3d52-413d-9617-f038377227d5-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:12.160012 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:12.159975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:35:12.520834 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:12.520748    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a6c2e9-3d52-413d-9617-f038377227d5" path="/var/lib/kubelet/pods/00a6c2e9-3d52-413d-9617-f038377227d5/volumes"
Jul 17 17:35:12.521195 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:12.521180    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" path="/var/lib/kubelet/pods/5023241c-04b4-40c7-ba2a-5facccfbc1c1/volumes"
Jul 17 17:35:16.511105 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:16.511074    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:35:16.511535 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:16.511278    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:35:21.508645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.508611    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:35:21.509172 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:21.508828    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:35:21.625240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:21.625620 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:35:21.625664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625623    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af3f3ad-a07a-4d53-b205-c6a96b6c6804" containerName="revoke-keys"
Jul 17 17:35:21.625664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625639    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" containerName="main"
Jul 17 17:35:21.625664 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625646    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" containerName="main"
Jul 17 17:35:21.625761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625674    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a6c2e9-3d52-413d-9617-f038377227d5" containerName="main"
Jul 17 17:35:21.625761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625680    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a6c2e9-3d52-413d-9617-f038377227d5" containerName="main"
Jul 17 17:35:21.625761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625736    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a6c2e9-3d52-413d-9617-f038377227d5" containerName="main"
Jul 17 17:35:21.625761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.625744    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5023241c-04b4-40c7-ba2a-5facccfbc1c1" containerName="main"
Jul 17 17:35:21.629804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.629783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:21.632333 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.632305    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:35:21.634804 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.634777    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:21.819917 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.819872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g\" (UID: \"ac394803-c367-47e7-8416-0753d7208e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:21.921364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.921321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g\" (UID: \"ac394803-c367-47e7-8416-0753d7208e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:21.921981 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.921957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g\" (UID: \"ac394803-c367-47e7-8416-0753d7208e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:21.941336 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:21.941299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:22.077934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:22.077900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:22.080179 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:35:22.080140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac394803_c367_47e7_8416_0753d7208e2f.slice/crio-1f9af3dfb13f623892493f815e2051315fa94a5fdfaf181539b4b208dbed0d63 WatchSource:0}: Error finding container 1f9af3dfb13f623892493f815e2051315fa94a5fdfaf181539b4b208dbed0d63: Status 404 returned error can't find the container with id 1f9af3dfb13f623892493f815e2051315fa94a5fdfaf181539b4b208dbed0d63
Jul 17 17:35:22.891247 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:22.891206    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac394803-c367-47e7-8416-0753d7208e2f" containerID="200473a886d98a3a60ea847ae6f24ae87eab33c317b4921d7bf09a6992c0a590" exitCode=7
Jul 17 17:35:22.891746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:22.891265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerDied","Data":"200473a886d98a3a60ea847ae6f24ae87eab33c317b4921d7bf09a6992c0a590"}
Jul 17 17:35:22.891746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:22.891304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerStarted","Data":"1f9af3dfb13f623892493f815e2051315fa94a5fdfaf181539b4b208dbed0d63"}
Jul 17 17:35:22.891746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:22.891525    2561 scope.go:117] "RemoveContainer" containerID="200473a886d98a3a60ea847ae6f24ae87eab33c317b4921d7bf09a6992c0a590"
Jul 17 17:35:23.897759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:23.897726    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac394803-c367-47e7-8416-0753d7208e2f" containerID="a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51" exitCode=7
Jul 17 17:35:23.898219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:23.897821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerDied","Data":"a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51"}
Jul 17 17:35:23.898219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:23.897888    2561 scope.go:117] "RemoveContainer" containerID="200473a886d98a3a60ea847ae6f24ae87eab33c317b4921d7bf09a6992c0a590"
Jul 17 17:35:23.898219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:23.898103    2561 scope.go:117] "RemoveContainer" containerID="a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51"
Jul 17 17:35:23.898376 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:23.898350    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g_odh-ai-gateway-infra(ac394803-c367-47e7-8416-0753d7208e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" podUID="ac394803-c367-47e7-8416-0753d7208e2f"
Jul 17 17:35:24.904988 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:24.904951    2561 scope.go:117] "RemoveContainer" containerID="a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51"
Jul 17 17:35:24.905387 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:24.905225    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g_odh-ai-gateway-infra(ac394803-c367-47e7-8416-0753d7208e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" podUID="ac394803-c367-47e7-8416-0753d7208e2f"
Jul 17 17:35:28.508869 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:28.508835    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:35:28.509425 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:28.509027    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:35:30.001269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.001223    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:35:30.004836 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.004816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.007321 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.007299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"openshift-service-ca.crt\""
Jul 17 17:35:30.008063 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.008027    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"kube-root-ca.crt\""
Jul 17 17:35:30.008163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.008029    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"default-dockercfg-q4tsr\""
Jul 17 17:35:30.008163 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.008126    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"sub-test-model-4f5e7b-kserve-self-signed-certs\""
Jul 17 17:35:30.015970 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.015940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:35:30.029411 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.029377    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:35:30.095670 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.095862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.095862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.095862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.095862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.096018 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.095878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.196800 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.196750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.196808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.196831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.196868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197030 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.196944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.197037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197372 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.197333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.197455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.197535 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.197484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.199260 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.199239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.199629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.199609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.205078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.205039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb\") pod \"sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") " pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.314257 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.314224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:30.453364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.453330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:35:30.455316 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:35:30.455287    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d3d4fd6_efeb_418b_a6c7_d0bcb7b8979e.slice/crio-b645731afdba63ca906051f1e7e71793b66507d38c993e4aa1e3c99d666d58a0 WatchSource:0}: Error finding container b645731afdba63ca906051f1e7e71793b66507d38c993e4aa1e3c99d666d58a0: Status 404 returned error can't find the container with id b645731afdba63ca906051f1e7e71793b66507d38c993e4aa1e3c99d666d58a0
Jul 17 17:35:30.936286 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.936243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" event={"ID":"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e","Type":"ContainerStarted","Data":"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"}
Jul 17 17:35:30.936533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:30.936294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" event={"ID":"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e","Type":"ContainerStarted","Data":"b645731afdba63ca906051f1e7e71793b66507d38c993e4aa1e3c99d666d58a0"}
Jul 17 17:35:33.508645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:33.508612    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:35:33.509131 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:33.508868    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:35:33.970704 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:33.970645    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" podStartSLOduration=4.970628446 podStartE2EDuration="4.970628446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:35:30.952688081 +0000 UTC m=+4164.937775620" watchObservedRunningTime="2026-07-17 17:35:33.970628446 +0000 UTC m=+4167.955715984"
Jul 17 17:35:33.971609 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:33.971592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:35:35.508749 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:35.508717    2561 scope.go:117] "RemoveContainer" containerID="a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51"
Jul 17 17:35:35.965267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:35.965220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerStarted","Data":"0a6fe808f53663b6b8b7f033832008cce89b97101ddda1974028f138900cdb5a"}
Jul 17 17:35:35.982037 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:35.981984    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" podStartSLOduration=14.981968913 podStartE2EDuration="14.981968913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:35:35.981253841 +0000 UTC m=+4169.966341403" watchObservedRunningTime="2026-07-17 17:35:35.981968913 +0000 UTC m=+4169.967056452"
Jul 17 17:35:36.972525 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:36.972493    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac394803-c367-47e7-8416-0753d7208e2f" containerID="0a6fe808f53663b6b8b7f033832008cce89b97101ddda1974028f138900cdb5a" exitCode=7
Jul 17 17:35:36.972944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:36.972565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerDied","Data":"0a6fe808f53663b6b8b7f033832008cce89b97101ddda1974028f138900cdb5a"}
Jul 17 17:35:36.972944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:36.972633    2561 scope.go:117] "RemoveContainer" containerID="a919db3405ec922033ffdb8bd002c24a4ddc3fd9b91e4342e6b194429f32ab51"
Jul 17 17:35:36.972944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:36.972929    2561 scope.go:117] "RemoveContainer" containerID="0a6fe808f53663b6b8b7f033832008cce89b97101ddda1974028f138900cdb5a"
Jul 17 17:35:36.973196 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:36.973171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g_odh-ai-gateway-infra(ac394803-c367-47e7-8416-0753d7208e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" podUID="ac394803-c367-47e7-8416-0753d7208e2f"
Jul 17 17:35:36.997629 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:36.997597    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:38.113266 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.113231    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:38.176738 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.176707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca\") pod \"ac394803-c367-47e7-8416-0753d7208e2f\" (UID: \"ac394803-c367-47e7-8416-0753d7208e2f\") "
Jul 17 17:35:38.177118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.177093    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac394803-c367-47e7-8416-0753d7208e2f" (UID: "ac394803-c367-47e7-8416-0753d7208e2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:35:38.278355 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.278316    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac394803-c367-47e7-8416-0753d7208e2f-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:35:38.985511 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.985478    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"
Jul 17 17:35:38.985708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.985485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g" event={"ID":"ac394803-c367-47e7-8416-0753d7208e2f","Type":"ContainerDied","Data":"1f9af3dfb13f623892493f815e2051315fa94a5fdfaf181539b4b208dbed0d63"}
Jul 17 17:35:38.985708 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:38.985656    2561 scope.go:117] "RemoveContainer" containerID="0a6fe808f53663b6b8b7f033832008cce89b97101ddda1974028f138900cdb5a"
Jul 17 17:35:39.000479 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:39.000453    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:39.006886 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:39.006860    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-rc89g"]
Jul 17 17:35:39.509191 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:39.509159    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:35:39.509601 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:39.509360    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:35:40.314992 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:40.314955    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:40.315218 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:40.315083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:40.327463 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:40.327434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:40.514835 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:40.514797    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac394803-c367-47e7-8416-0753d7208e2f" path="/var/lib/kubelet/pods/ac394803-c367-47e7-8416-0753d7208e2f/volumes"
Jul 17 17:35:40.517943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:40.517917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:35:41.007706 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:41.007673    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:35:46.511377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:46.511344    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:35:46.511785 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:46.511541    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:35:48.057164 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:48.057122    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:35:53.508753 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:35:53.508722    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:35:53.509176 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:35:53.508928    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:00.508818 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:00.508787    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:00.509279 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:00.508998    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:04.143101 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143065    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:04.143518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143425    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143436    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143461    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143466    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143527    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143539    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.143661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.143547    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:04.146745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.146726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:04.149350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.149329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:36:04.155795 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.155764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:04.330923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.330884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8\" (UID: \"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:04.431824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.431719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8\" (UID: \"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:04.432407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.432384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8\" (UID: \"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:04.458742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.458708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:04.598950 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:04.598897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:04.601938 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:36:04.601908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8fb64b_0462_4dfe_b978_8c1f8f74fdd6.slice/crio-76020f2a689335d3450cbfb09c059190976492f31835c6dfbc10f530de5b8c88 WatchSource:0}: Error finding container 76020f2a689335d3450cbfb09c059190976492f31835c6dfbc10f530de5b8c88: Status 404 returned error can't find the container with id 76020f2a689335d3450cbfb09c059190976492f31835c6dfbc10f530de5b8c88
Jul 17 17:36:05.118116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:05.118010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerStarted","Data":"e301dc63d05372dea5640c343db5ec11930d9d44327cdacd7e2d45bf71aaf410"}
Jul 17 17:36:05.118116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:05.118083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerStarted","Data":"76020f2a689335d3450cbfb09c059190976492f31835c6dfbc10f530de5b8c88"}
Jul 17 17:36:05.135350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:05.135300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" podStartSLOduration=1.135281368 podStartE2EDuration="1.135281368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:36:05.133671151 +0000 UTC m=+4199.118758716" watchObservedRunningTime="2026-07-17 17:36:05.135281368 +0000 UTC m=+4199.120368906"
Jul 17 17:36:06.124168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:06.124127    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerID="e301dc63d05372dea5640c343db5ec11930d9d44327cdacd7e2d45bf71aaf410" exitCode=7
Jul 17 17:36:06.124574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:06.124213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerDied","Data":"e301dc63d05372dea5640c343db5ec11930d9d44327cdacd7e2d45bf71aaf410"}
Jul 17 17:36:06.124574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:06.124434    2561 scope.go:117] "RemoveContainer" containerID="e301dc63d05372dea5640c343db5ec11930d9d44327cdacd7e2d45bf71aaf410"
Jul 17 17:36:06.725974 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:06.725949    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:36:06.735349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:06.735327    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mj6xf_4f6799b7-2071-4df3-9fb4-33d2ef840c41/ovn-acl-logging/0.log"
Jul 17 17:36:07.130898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:07.130857    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerID="13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b" exitCode=7
Jul 17 17:36:07.131424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:07.130938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerDied","Data":"13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b"}
Jul 17 17:36:07.131424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:07.131020    2561 scope.go:117] "RemoveContainer" containerID="e301dc63d05372dea5640c343db5ec11930d9d44327cdacd7e2d45bf71aaf410"
Jul 17 17:36:07.131424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:07.131266    2561 scope.go:117] "RemoveContainer" containerID="13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b"
Jul 17 17:36:07.131552 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:07.131523    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8_odh-ai-gateway-infra(6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6"
Jul 17 17:36:07.508652 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:07.508567    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:36:07.508816 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:07.508735    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:08.137554 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:08.137523    2561 scope.go:117] "RemoveContainer" containerID="13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b"
Jul 17 17:36:08.137949 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:08.137775    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8_odh-ai-gateway-infra(6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6"
Jul 17 17:36:11.057616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:11.057576    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:36:12.509416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:12.509381    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:12.509833 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:12.509540    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:17.975799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:17.975762    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:36:17.996017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:17.995986    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:36:18.508773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:18.508741    2561 scope.go:117] "RemoveContainer" containerID="13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b"
Jul 17 17:36:18.913879 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:18.913842    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:36:18.933181 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:18.933148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:36:19.193737 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:19.193705    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerID="b39cd5f108ae3ed1e762e12ebe35ba1374be6d45933e5d0638be0cb4ac19232f" exitCode=7
Jul 17 17:36:19.194194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:19.193746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerDied","Data":"b39cd5f108ae3ed1e762e12ebe35ba1374be6d45933e5d0638be0cb4ac19232f"}
Jul 17 17:36:19.194194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:19.193794    2561 scope.go:117] "RemoveContainer" containerID="13efb303d448196dab9dba729f2e6ea7ce7d176ac6c69f73108874d2fad8f22b"
Jul 17 17:36:19.194194 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:19.194116    2561 scope.go:117] "RemoveContainer" containerID="b39cd5f108ae3ed1e762e12ebe35ba1374be6d45933e5d0638be0cb4ac19232f"
Jul 17 17:36:19.194385 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:19.194359    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8_odh-ai-gateway-infra(6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6"
Jul 17 17:36:20.219191 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.219152    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:20.362562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.362534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:20.367816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.367792    2561 scope.go:117] "RemoveContainer" containerID="0016c5f96877ec218df6d716eb4d96db45a5242917b813b01e4d4b993493bd55"
Jul 17 17:36:20.375736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.375712    2561 scope.go:117] "RemoveContainer" containerID="7d27fd0243d81121c39b2109e8def119eb78541fa1b724216846788fe9426c0d"
Jul 17 17:36:20.482558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.482477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca\") pod \"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6\" (UID: \"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6\") "
Jul 17 17:36:20.482850 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.482826    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" (UID: "6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:36:20.583602 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:20.583564    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:21.206797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:21.206767    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"
Jul 17 17:36:21.206983 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:21.206807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8" event={"ID":"6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6","Type":"ContainerDied","Data":"76020f2a689335d3450cbfb09c059190976492f31835c6dfbc10f530de5b8c88"}
Jul 17 17:36:21.206983 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:21.206914    2561 scope.go:117] "RemoveContainer" containerID="b39cd5f108ae3ed1e762e12ebe35ba1374be6d45933e5d0638be0cb4ac19232f"
Jul 17 17:36:21.223486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:21.223457    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:21.226000 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:21.225975    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-z8kw8"]
Jul 17 17:36:22.509152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:22.509112    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:36:22.515190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:22.515163    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" path="/var/lib/kubelet/pods/6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6/volumes"
Jul 17 17:36:23.220973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:23.220877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerStarted","Data":"28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"}
Jul 17 17:36:23.221153 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:23.221089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:36:23.508335 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:23.508252    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:23.508484 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:23.508451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:23.954257 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:23.954219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:36:24.052693 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.052657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:36:24.228440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.228347    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eed73c2-e0b6-436d-9572-ecbcf429039a" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817" exitCode=1
Jul 17 17:36:24.228440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.228426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" event={"ID":"9eed73c2-e0b6-436d-9572-ecbcf429039a","Type":"ContainerDied","Data":"28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"}
Jul 17 17:36:24.228711 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.228493    2561 scope.go:117] "RemoveContainer" containerID="ead96621c5e06ad544bd2bcc246a0976fb204baafe9ce08c6712a7d712eb19c9"
Jul 17 17:36:24.228773 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.228733    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:36:24.229004 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:24.228983    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:24.955246 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.955199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:36:24.969723 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:24.969694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn"
Jul 17 17:36:25.235316 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:25.235235    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:36:25.235469 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:25.235410    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:26.240423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:26.240394    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:36:26.240845 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:26.240609    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:32.631573 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:32.631493    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:36:32.996167 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:32.996082    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" podUID="27db4d04-1a12-4bc1-90e8-101effaf0eac" containerName="main" containerID="cri-o://17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179" gracePeriod=45
Jul 17 17:36:33.255754 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.255685    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:36:33.276770 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.276738    2561 generic.go:356] "Generic (PLEG): container finished" podID="27db4d04-1a12-4bc1-90e8-101effaf0eac" containerID="17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179" exitCode=0
Jul 17 17:36:33.276964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.276787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"
Jul 17 17:36:33.276964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.276796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" event={"ID":"27db4d04-1a12-4bc1-90e8-101effaf0eac","Type":"ContainerDied","Data":"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"}
Jul 17 17:36:33.276964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.276834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc" event={"ID":"27db4d04-1a12-4bc1-90e8-101effaf0eac","Type":"ContainerDied","Data":"e7c5da3536317a16c2be93908b9da1cc5c22d38360d0792ce36e2463a1322006"}
Jul 17 17:36:33.276964 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.276857    2561 scope.go:117] "RemoveContainer" containerID="17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"
Jul 17 17:36:33.286768 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.286747    2561 scope.go:117] "RemoveContainer" containerID="17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"
Jul 17 17:36:33.287033 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:33.287014    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179\": container with ID starting with 17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179 not found: ID does not exist" containerID="17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"
Jul 17 17:36:33.287108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.287064    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179"} err="failed to get container status \"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179\": rpc error: code = NotFound desc = could not find container \"17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179\": container with ID starting with 17adb86692708b5e1df6413ecf0f65e067d3da25b651c6e74344ebcf036ae179 not found: ID does not exist"
Jul 17 17:36:33.416348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd77b\" (UniqueName: \"kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home\") pod \"27db4d04-1a12-4bc1-90e8-101effaf0eac\" (UID: \"27db4d04-1a12-4bc1-90e8-101effaf0eac\") "
Jul 17 17:36:33.416827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416676    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache" (OuterVolumeSpecName: "model-cache") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:33.416827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:33.416827 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home" (OuterVolumeSpecName: "home") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:33.416948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416874    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.416948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.416892    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.418690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.418668    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm" (OuterVolumeSpecName: "dshm") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:33.418796 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.418728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:36:33.418846 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.418793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b" (OuterVolumeSpecName: "kube-api-access-pd77b") pod "27db4d04-1a12-4bc1-90e8-101effaf0eac" (UID: "27db4d04-1a12-4bc1-90e8-101effaf0eac"). InnerVolumeSpecName "kube-api-access-pd77b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:36:33.518382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.518296    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27db4d04-1a12-4bc1-90e8-101effaf0eac-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.518382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.518325    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.518382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.518334    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27db4d04-1a12-4bc1-90e8-101effaf0eac-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.518382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.518342    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd77b\" (UniqueName: \"kubernetes.io/projected/27db4d04-1a12-4bc1-90e8-101effaf0eac-kube-api-access-pd77b\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:33.598301 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.598268    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:36:33.602468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.602438    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3d4103/sub-test-model-3d4103-kserve-79f4457489-bvdjc"]
Jul 17 17:36:33.935958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:33.935892    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" podUID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" containerName="main" containerID="cri-o://fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd" gracePeriod=45
Jul 17 17:36:34.190240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.190162    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:36:34.283192 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.283155    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" containerID="fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd" exitCode=0
Jul 17 17:36:34.283405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.283207    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"
Jul 17 17:36:34.283405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.283242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" event={"ID":"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e","Type":"ContainerDied","Data":"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"}
Jul 17 17:36:34.283405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.283279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p" event={"ID":"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e","Type":"ContainerDied","Data":"b645731afdba63ca906051f1e7e71793b66507d38c993e4aa1e3c99d666d58a0"}
Jul 17 17:36:34.283405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.283298    2561 scope.go:117] "RemoveContainer" containerID="fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"
Jul 17 17:36:34.291588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.291566    2561 scope.go:117] "RemoveContainer" containerID="fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"
Jul 17 17:36:34.291862 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:34.291840    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd\": container with ID starting with fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd not found: ID does not exist" containerID="fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"
Jul 17 17:36:34.291943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.291869    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd"} err="failed to get container status \"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd\": rpc error: code = NotFound desc = could not find container \"fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd\": container with ID starting with fd981c98dfaab0ca75416c9168cbf5733becdb6206a767169a698323f3d5c1dd not found: ID does not exist"
Jul 17 17:36:34.326449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326529    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326628    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache\") pod \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\" (UID: \"5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e\") "
Jul 17 17:36:34.326906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326808    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:34.326906 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home" (OuterVolumeSpecName: "home") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:34.327014 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.326923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:34.327161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.327036    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.327161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.327083    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.327161 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.327096    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.328756 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.328734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:36:34.328862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.328766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm" (OuterVolumeSpecName: "dshm") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:36:34.328862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.328784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb" (OuterVolumeSpecName: "kube-api-access-cfdxb") pod "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" (UID: "5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e"). InnerVolumeSpecName "kube-api-access-cfdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:36:34.428409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.428368    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.428409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.428402    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-kube-api-access-cfdxb\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.428409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.428417    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:34.515482 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.515397    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27db4d04-1a12-4bc1-90e8-101effaf0eac" path="/var/lib/kubelet/pods/27db4d04-1a12-4bc1-90e8-101effaf0eac/volumes"
Jul 17 17:36:34.599165 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.599124    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:36:34.603104 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:34.603075    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/sub-test-model-4f5e7b-kserve-788b6bb6d9-8rn9p"]
Jul 17 17:36:35.508999 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:35.508965    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:35.509412 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:35.509237    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:36.514799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.514762    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" path="/var/lib/kubelet/pods/5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e/volumes"
Jul 17 17:36:36.742616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.742582    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:36.742977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.742964    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.742980    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac394803-c367-47e7-8416-0753d7208e2f" containerName="revoke-keys"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.742989    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.742994    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743005    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" containerName="main"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743010    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" containerName="main"
Jul 17 17:36:36.743024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743024    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743029    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743036    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743055    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743068    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27db4d04-1a12-4bc1-90e8-101effaf0eac" containerName="main"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743076    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27db4d04-1a12-4bc1-90e8-101effaf0eac" containerName="main"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743149    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3d4fd6-efeb-418b-a6c7-d0bcb7b8979e" containerName="main"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743160    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743167    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:36.743245 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.743175    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27db4d04-1a12-4bc1-90e8-101effaf0eac" containerName="main"
Jul 17 17:36:36.747890 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.747866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:36.750342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.750315    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:36:36.755357 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.755333    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:36.851267 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.851231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb\" (UID: \"f83d63e4-801b-4409-816b-4dff7215141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:36.952134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.952093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb\" (UID: \"f83d63e4-801b-4409-816b-4dff7215141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:36.952741 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:36.952719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb\" (UID: \"f83d63e4-801b-4409-816b-4dff7215141e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:37.060720 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:37.060685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:37.193312 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:37.193284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:37.195766 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:36:37.195732    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83d63e4_801b_4409_816b_4dff7215141e.slice/crio-a0131abed395ece6bb2a289a803d72f09e7bc61dc7696b75b862d0efc7b92483 WatchSource:0}: Error finding container a0131abed395ece6bb2a289a803d72f09e7bc61dc7696b75b862d0efc7b92483: Status 404 returned error can't find the container with id a0131abed395ece6bb2a289a803d72f09e7bc61dc7696b75b862d0efc7b92483
Jul 17 17:36:37.307160 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:37.307118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" event={"ID":"f83d63e4-801b-4409-816b-4dff7215141e","Type":"ContainerStarted","Data":"a0131abed395ece6bb2a289a803d72f09e7bc61dc7696b75b862d0efc7b92483"}
Jul 17 17:36:38.062624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:38.062583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:36:38.313342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:38.313307    2561 generic.go:356] "Generic (PLEG): container finished" podID="f83d63e4-801b-4409-816b-4dff7215141e" containerID="b893fd7a4c76641a4a0b4e45f0ad3ad6df8b63af280c8c178e6bf12abd9f4f29" exitCode=7
Jul 17 17:36:38.313533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:38.313394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" event={"ID":"f83d63e4-801b-4409-816b-4dff7215141e","Type":"ContainerDied","Data":"b893fd7a4c76641a4a0b4e45f0ad3ad6df8b63af280c8c178e6bf12abd9f4f29"}
Jul 17 17:36:38.313588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:38.313551    2561 scope.go:117] "RemoveContainer" containerID="b893fd7a4c76641a4a0b4e45f0ad3ad6df8b63af280c8c178e6bf12abd9f4f29"
Jul 17 17:36:38.513856 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:38.513832    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:36:38.514035 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:38.514013    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:39.320316 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:39.320268    2561 generic.go:356] "Generic (PLEG): container finished" podID="f83d63e4-801b-4409-816b-4dff7215141e" containerID="6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e" exitCode=7
Jul 17 17:36:39.320742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:39.320343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" event={"ID":"f83d63e4-801b-4409-816b-4dff7215141e","Type":"ContainerDied","Data":"6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e"}
Jul 17 17:36:39.320742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:39.320400    2561 scope.go:117] "RemoveContainer" containerID="b893fd7a4c76641a4a0b4e45f0ad3ad6df8b63af280c8c178e6bf12abd9f4f29"
Jul 17 17:36:39.320742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:39.320577    2561 scope.go:117] "RemoveContainer" containerID="6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e"
Jul 17 17:36:39.320863 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:39.320838    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb_odh-ai-gateway-infra(f83d63e4-801b-4409-816b-4dff7215141e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" podUID="f83d63e4-801b-4409-816b-4dff7215141e"
Jul 17 17:36:40.327094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:40.327038    2561 scope.go:117] "RemoveContainer" containerID="6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e"
Jul 17 17:36:40.327541 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:40.327349    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb_odh-ai-gateway-infra(f83d63e4-801b-4409-816b-4dff7215141e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" podUID="f83d63e4-801b-4409-816b-4dff7215141e"
Jul 17 17:36:47.508222 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:47.508192    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:47.508701 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:47.508429    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:51.509272 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:51.509239    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:36:51.509710 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:51.509427    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:36:54.180985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.180943    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:36:54.181435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.181420    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8fb64b-0462-4dfe-b978-8c1f8f74fdd6" containerName="revoke-keys"
Jul 17 17:36:54.185815 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.185799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:36:54.192566 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.192538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:36:54.216714 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.216680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26\" (UID: \"3cb48b40-4ae4-4502-a925-e09f15e660c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:36:54.318155 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.318120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26\" (UID: \"3cb48b40-4ae4-4502-a925-e09f15e660c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:36:54.318768 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.318745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26\" (UID: \"3cb48b40-4ae4-4502-a925-e09f15e660c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:36:54.498744 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.498654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:36:54.509019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.508993    2561 scope.go:117] "RemoveContainer" containerID="6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e"
Jul 17 17:36:54.635926 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:54.635897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:36:54.638184 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:36:54.638142    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb48b40_4ae4_4502_a925_e09f15e660c4.slice/crio-9ac7afde20690f6c0cac3c58a129f9bca2ee19e5e39dac44e948b07f2be8c95d WatchSource:0}: Error finding container 9ac7afde20690f6c0cac3c58a129f9bca2ee19e5e39dac44e948b07f2be8c95d: Status 404 returned error can't find the container with id 9ac7afde20690f6c0cac3c58a129f9bca2ee19e5e39dac44e948b07f2be8c95d
Jul 17 17:36:55.399153 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.399116    2561 generic.go:356] "Generic (PLEG): container finished" podID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerID="9e614a3b046456bdf2845eeb6deb205f906ba0fc7dd3b2ed49624e4378e54779" exitCode=7
Jul 17 17:36:55.399656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.399204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" event={"ID":"3cb48b40-4ae4-4502-a925-e09f15e660c4","Type":"ContainerDied","Data":"9e614a3b046456bdf2845eeb6deb205f906ba0fc7dd3b2ed49624e4378e54779"}
Jul 17 17:36:55.399656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.399293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" event={"ID":"3cb48b40-4ae4-4502-a925-e09f15e660c4","Type":"ContainerStarted","Data":"9ac7afde20690f6c0cac3c58a129f9bca2ee19e5e39dac44e948b07f2be8c95d"}
Jul 17 17:36:55.399656 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.399400    2561 scope.go:117] "RemoveContainer" containerID="9e614a3b046456bdf2845eeb6deb205f906ba0fc7dd3b2ed49624e4378e54779"
Jul 17 17:36:55.401162 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.401138    2561 generic.go:356] "Generic (PLEG): container finished" podID="f83d63e4-801b-4409-816b-4dff7215141e" containerID="3043e94a5c3377b7f5907d749c2af0d69da1deac3f7fa79ec2ef0571d7ff97e8" exitCode=7
Jul 17 17:36:55.401281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.401218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" event={"ID":"f83d63e4-801b-4409-816b-4dff7215141e","Type":"ContainerDied","Data":"3043e94a5c3377b7f5907d749c2af0d69da1deac3f7fa79ec2ef0571d7ff97e8"}
Jul 17 17:36:55.401281 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.401270    2561 scope.go:117] "RemoveContainer" containerID="6f63954c454ef770f6bfb13ce72a070e0ffc5087f7408d6b6a648be7998f377e"
Jul 17 17:36:55.401533 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:55.401516    2561 scope.go:117] "RemoveContainer" containerID="3043e94a5c3377b7f5907d749c2af0d69da1deac3f7fa79ec2ef0571d7ff97e8"
Jul 17 17:36:55.401726 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:55.401707    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb_odh-ai-gateway-infra(f83d63e4-801b-4409-816b-4dff7215141e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" podUID="f83d63e4-801b-4409-816b-4dff7215141e"
Jul 17 17:36:56.408305 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.408273    2561 generic.go:356] "Generic (PLEG): container finished" podID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerID="dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92" exitCode=7
Jul 17 17:36:56.408805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.408368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" event={"ID":"3cb48b40-4ae4-4502-a925-e09f15e660c4","Type":"ContainerDied","Data":"dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92"}
Jul 17 17:36:56.408805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.408443    2561 scope.go:117] "RemoveContainer" containerID="9e614a3b046456bdf2845eeb6deb205f906ba0fc7dd3b2ed49624e4378e54779"
Jul 17 17:36:56.408805 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.408550    2561 scope.go:117] "RemoveContainer" containerID="dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92"
Jul 17 17:36:56.409009 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:56.408827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26_odh-ai-gateway-infra(3cb48b40-4ae4-4502-a925-e09f15e660c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4"
Jul 17 17:36:56.443549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.443512    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:56.579512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.579488    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:56.638187 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.638144    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca\") pod \"f83d63e4-801b-4409-816b-4dff7215141e\" (UID: \"f83d63e4-801b-4409-816b-4dff7215141e\") "
Jul 17 17:36:56.638493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.638472    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f83d63e4-801b-4409-816b-4dff7215141e" (UID: "f83d63e4-801b-4409-816b-4dff7215141e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:36:56.739587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:56.739472    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83d63e4-801b-4409-816b-4dff7215141e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:36:57.416583 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.416552    2561 scope.go:117] "RemoveContainer" containerID="dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92"
Jul 17 17:36:57.417026 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:57.416820    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26_odh-ai-gateway-infra(3cb48b40-4ae4-4502-a925-e09f15e660c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4"
Jul 17 17:36:57.417558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.417538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb" event={"ID":"f83d63e4-801b-4409-816b-4dff7215141e","Type":"ContainerDied","Data":"a0131abed395ece6bb2a289a803d72f09e7bc61dc7696b75b862d0efc7b92483"}
Jul 17 17:36:57.417558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.417553    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"
Jul 17 17:36:57.417662 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.417585    2561 scope.go:117] "RemoveContainer" containerID="3043e94a5c3377b7f5907d749c2af0d69da1deac3f7fa79ec2ef0571d7ff97e8"
Jul 17 17:36:57.444007 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.443978    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:57.449847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:57.449817    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-pkgqb"]
Jul 17 17:36:58.509026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:58.508996    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:36:58.509414 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:36:58.509224    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:36:58.514564 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:36:58.514539    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83d63e4-801b-4409-816b-4dff7215141e" path="/var/lib/kubelet/pods/f83d63e4-801b-4409-816b-4dff7215141e/volumes"
Jul 17 17:37:04.913931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.913895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:37:04.914370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914309    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.914370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914321    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.914370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914351    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.914370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914359    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.914543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914457    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.914543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.914468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:04.918910 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.918884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:04.921412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.921388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"kube-root-ca.crt\""
Jul 17 17:37:04.922096 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.922076    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"default-dockercfg-q4tsr\""
Jul 17 17:37:04.922236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.922215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"rate-test-model-4f5e7b-kserve-self-signed-certs\""
Jul 17 17:37:04.922289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.922268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f5e7b\"/\"openshift-service-ca.crt\""
Jul 17 17:37:04.928393 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.928365    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:37:04.954100 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:04.954034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:05.022263 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.022263 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.022512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.022512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.022512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.022512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.022478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9kw\" (UniqueName: \"kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9kw\" (UniqueName: \"kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123802 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123802 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.123954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.124016 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.123958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.125736 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.125711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.125973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.125955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.134853 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.134823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9kw\" (UniqueName: \"kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw\") pod \"rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") " pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.230122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.229999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:05.372420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.372392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:37:05.374035 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:05.374009    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod803f444d_6d1b_4f4f_a510_27744091b5b7.slice/crio-c103c6c3b9ebab13bbb285547fbd28f39b8a7f691ab7d2f37ee8b48149e1dcd6 WatchSource:0}: Error finding container c103c6c3b9ebab13bbb285547fbd28f39b8a7f691ab7d2f37ee8b48149e1dcd6: Status 404 returned error can't find the container with id c103c6c3b9ebab13bbb285547fbd28f39b8a7f691ab7d2f37ee8b48149e1dcd6
Jul 17 17:37:05.465680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.465645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" event={"ID":"803f444d-6d1b-4f4f-a510-27744091b5b7","Type":"ContainerStarted","Data":"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"}
Jul 17 17:37:05.465851 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.465694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" event={"ID":"803f444d-6d1b-4f4f-a510-27744091b5b7","Type":"ContainerStarted","Data":"c103c6c3b9ebab13bbb285547fbd28f39b8a7f691ab7d2f37ee8b48149e1dcd6"}
Jul 17 17:37:05.483900 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:05.483793    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" podStartSLOduration=1.483776708 podStartE2EDuration="1.483776708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:05.482164287 +0000 UTC m=+4259.467251827" watchObservedRunningTime="2026-07-17 17:37:05.483776708 +0000 UTC m=+4259.468864247"
Jul 17 17:37:06.304873 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.304826    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:06.305315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.305246    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:06.305315 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.305258    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:06.305405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.305322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83d63e4-801b-4409-816b-4dff7215141e" containerName="revoke-keys"
Jul 17 17:37:06.308448 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.308430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:06.315968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.315943    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:06.438180 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.438134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-w8h72\" (UID: \"4a744b2a-36dd-4d07-8a19-24e740863010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:06.514961 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.514933    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:37:06.515152 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:06.515133    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:37:06.539095 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.539040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-w8h72\" (UID: \"4a744b2a-36dd-4d07-8a19-24e740863010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:06.539668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.539648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c607776-w8h72\" (UID: \"4a744b2a-36dd-4d07-8a19-24e740863010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:06.621216 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.621127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:06.758923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:06.758866    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:06.761237 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:06.761197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a744b2a_36dd_4d07_8a19_24e740863010.slice/crio-33b4488a03d795deb12531f6d5057ab350824eaac8b28810fb57629c342d741f WatchSource:0}: Error finding container 33b4488a03d795deb12531f6d5057ab350824eaac8b28810fb57629c342d741f: Status 404 returned error can't find the container with id 33b4488a03d795deb12531f6d5057ab350824eaac8b28810fb57629c342d741f
Jul 17 17:37:07.477600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:07.477569    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a744b2a-36dd-4d07-8a19-24e740863010" containerID="f48179ae87ebd9df052ca55408aa0d4ecea17731b26613bd8cedb72dcba37fad" exitCode=7
Jul 17 17:37:07.478017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:07.477611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" event={"ID":"4a744b2a-36dd-4d07-8a19-24e740863010","Type":"ContainerDied","Data":"f48179ae87ebd9df052ca55408aa0d4ecea17731b26613bd8cedb72dcba37fad"}
Jul 17 17:37:07.478017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:07.477675    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" event={"ID":"4a744b2a-36dd-4d07-8a19-24e740863010","Type":"ContainerStarted","Data":"33b4488a03d795deb12531f6d5057ab350824eaac8b28810fb57629c342d741f"}
Jul 17 17:37:07.478017 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:07.477875    2561 scope.go:117] "RemoveContainer" containerID="f48179ae87ebd9df052ca55408aa0d4ecea17731b26613bd8cedb72dcba37fad"
Jul 17 17:37:07.508171 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:07.508147    2561 scope.go:117] "RemoveContainer" containerID="dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92"
Jul 17 17:37:08.484156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.484063    2561 generic.go:356] "Generic (PLEG): container finished" podID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerID="63b9d03bad2cc55e958ba2bd5e4c89433409e4da8637bdc472ebc5689c753767" exitCode=7
Jul 17 17:37:08.484156 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.484137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" event={"ID":"3cb48b40-4ae4-4502-a925-e09f15e660c4","Type":"ContainerDied","Data":"63b9d03bad2cc55e958ba2bd5e4c89433409e4da8637bdc472ebc5689c753767"}
Jul 17 17:37:08.484699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.484199    2561 scope.go:117] "RemoveContainer" containerID="dfb046589b8faf50b975c6363c1f92332521f8f3107e9cb3f73f21d3b15efa92"
Jul 17 17:37:08.484699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.484553    2561 scope.go:117] "RemoveContainer" containerID="63b9d03bad2cc55e958ba2bd5e4c89433409e4da8637bdc472ebc5689c753767"
Jul 17 17:37:08.484829 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:08.484805    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26_odh-ai-gateway-infra(3cb48b40-4ae4-4502-a925-e09f15e660c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4"
Jul 17 17:37:08.486109 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.486088    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a744b2a-36dd-4d07-8a19-24e740863010" containerID="fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9" exitCode=7
Jul 17 17:37:08.486241 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.486216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" event={"ID":"4a744b2a-36dd-4d07-8a19-24e740863010","Type":"ContainerDied","Data":"fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9"}
Jul 17 17:37:08.486343 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.486332    2561 scope.go:117] "RemoveContainer" containerID="fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9"
Jul 17 17:37:08.486525 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:08.486508    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-w8h72_odh-ai-gateway-infra(4a744b2a-36dd-4d07-8a19-24e740863010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" podUID="4a744b2a-36dd-4d07-8a19-24e740863010"
Jul 17 17:37:08.495437 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.495419    2561 scope.go:117] "RemoveContainer" containerID="f48179ae87ebd9df052ca55408aa0d4ecea17731b26613bd8cedb72dcba37fad"
Jul 17 17:37:08.858584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:08.858546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:37:09.495484 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.495457    2561 scope.go:117] "RemoveContainer" containerID="fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9"
Jul 17 17:37:09.496095 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:09.495750    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-w8h72_odh-ai-gateway-infra(4a744b2a-36dd-4d07-8a19-24e740863010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" podUID="4a744b2a-36dd-4d07-8a19-24e740863010"
Jul 17 17:37:09.511621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.511591    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:37:09.650746 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.650719    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:37:09.773937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.773902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca\") pod \"3cb48b40-4ae4-4502-a925-e09f15e660c4\" (UID: \"3cb48b40-4ae4-4502-a925-e09f15e660c4\") "
Jul 17 17:37:09.774319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.774296    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cb48b40-4ae4-4502-a925-e09f15e660c4" (UID: "3cb48b40-4ae4-4502-a925-e09f15e660c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:37:09.875796 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:09.875731    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb48b40-4ae4-4502-a925-e09f15e660c4-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:37:10.501424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:10.501385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26" event={"ID":"3cb48b40-4ae4-4502-a925-e09f15e660c4","Type":"ContainerDied","Data":"9ac7afde20690f6c0cac3c58a129f9bca2ee19e5e39dac44e948b07f2be8c95d"}
Jul 17 17:37:10.501424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:10.501419    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"
Jul 17 17:37:10.501953 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:10.501480    2561 scope.go:117] "RemoveContainer" containerID="63b9d03bad2cc55e958ba2bd5e4c89433409e4da8637bdc472ebc5689c753767"
Jul 17 17:37:10.528196 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:10.528164    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:37:10.535551 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:10.535523    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-jwm26"]
Jul 17 17:37:11.709009 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.708971    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:11.709416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709371    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709383    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709393    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709416 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709398    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709462    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709470    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.709567 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.709479    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:11.713923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.713904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:11.719649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.719612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:11.896588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.896546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz\" (UID: \"b458466c-354b-475e-9ecd-3f27a7990e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:11.997471 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.997376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz\" (UID: \"b458466c-354b-475e-9ecd-3f27a7990e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:11.998040 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:11.998017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz\" (UID: \"b458466c-354b-475e-9ecd-3f27a7990e8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:12.027366 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:12.027321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:12.189352 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:12.189323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:12.191905 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:12.191877    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb458466c_354b_475e_9ecd_3f27a7990e8a.slice/crio-a05685f39e690f5147c72bca7625da48eb6975a53d42188463eb61f7336c65cc WatchSource:0}: Error finding container a05685f39e690f5147c72bca7625da48eb6975a53d42188463eb61f7336c65cc: Status 404 returned error can't find the container with id a05685f39e690f5147c72bca7625da48eb6975a53d42188463eb61f7336c65cc
Jul 17 17:37:12.515093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:12.515067    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" path="/var/lib/kubelet/pods/3cb48b40-4ae4-4502-a925-e09f15e660c4/volumes"
Jul 17 17:37:12.515385 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:12.515363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" event={"ID":"b458466c-354b-475e-9ecd-3f27a7990e8a","Type":"ContainerStarted","Data":"a05685f39e690f5147c72bca7625da48eb6975a53d42188463eb61f7336c65cc"}
Jul 17 17:37:12.515799 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:12.515783    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:37:12.515984 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:12.515967    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:37:13.522026 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:13.521989    2561 generic.go:356] "Generic (PLEG): container finished" podID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerID="d88f44faf5f4e570a1149c05b8201b6a26986c137041dc1003bfe4b805df5b9f" exitCode=7
Jul 17 17:37:13.522510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:13.522072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" event={"ID":"b458466c-354b-475e-9ecd-3f27a7990e8a","Type":"ContainerDied","Data":"d88f44faf5f4e570a1149c05b8201b6a26986c137041dc1003bfe4b805df5b9f"}
Jul 17 17:37:13.522510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:13.522298    2561 scope.go:117] "RemoveContainer" containerID="d88f44faf5f4e570a1149c05b8201b6a26986c137041dc1003bfe4b805df5b9f"
Jul 17 17:37:14.529492 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:14.529455    2561 generic.go:356] "Generic (PLEG): container finished" podID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerID="f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966" exitCode=7
Jul 17 17:37:14.529908 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:14.529531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" event={"ID":"b458466c-354b-475e-9ecd-3f27a7990e8a","Type":"ContainerDied","Data":"f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966"}
Jul 17 17:37:14.529908 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:14.529613    2561 scope.go:117] "RemoveContainer" containerID="d88f44faf5f4e570a1149c05b8201b6a26986c137041dc1003bfe4b805df5b9f"
Jul 17 17:37:14.529908 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:14.529783    2561 scope.go:117] "RemoveContainer" containerID="f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966"
Jul 17 17:37:14.530135 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:14.530009    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz_odh-ai-gateway-infra(b458466c-354b-475e-9ecd-3f27a7990e8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a"
Jul 17 17:37:15.230128 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:15.230089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:15.230128 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:15.230136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:15.242775 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:15.242742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:15.546090 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:15.546059    2561 scope.go:117] "RemoveContainer" containerID="f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966"
Jul 17 17:37:15.546574 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:15.546283    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz_odh-ai-gateway-infra(b458466c-354b-475e-9ecd-3f27a7990e8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a"
Jul 17 17:37:15.557707 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:15.557675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:37:18.509314 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:18.509277    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:37:18.509830 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:18.509637    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:37:18.685598 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:18.685560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:18.788120 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:18.788090    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:20.405817 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:20.405781    2561 scope.go:117] "RemoveContainer" containerID="1695bc3caf6ef5ed5468a962888b5f4b7e98d2ccdc2ec605978a4a35f7b7717a"
Jul 17 17:37:24.508745 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:24.508705    2561 scope.go:117] "RemoveContainer" containerID="fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9"
Jul 17 17:37:25.508960 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:25.508872    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:37:25.509367 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:25.509120    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:37:25.597963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:25.597931    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a744b2a-36dd-4d07-8a19-24e740863010" containerID="c719609b4f2545c896ede30a82821df33529ffb08a92bb89420eef5ad8c36da2" exitCode=7
Jul 17 17:37:25.598188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:25.597972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" event={"ID":"4a744b2a-36dd-4d07-8a19-24e740863010","Type":"ContainerDied","Data":"c719609b4f2545c896ede30a82821df33529ffb08a92bb89420eef5ad8c36da2"}
Jul 17 17:37:25.598188 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:25.598033    2561 scope.go:117] "RemoveContainer" containerID="fd3f49b5c0a5053d321802610140bc362d7d506c8ca2525551bc8acb1a4ff8e9"
Jul 17 17:37:25.598415 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:25.598395    2561 scope.go:117] "RemoveContainer" containerID="c719609b4f2545c896ede30a82821df33529ffb08a92bb89420eef5ad8c36da2"
Jul 17 17:37:25.598646 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:25.598625    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-8c607776-w8h72_odh-ai-gateway-infra(4a744b2a-36dd-4d07-8a19-24e740863010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" podUID="4a744b2a-36dd-4d07-8a19-24e740863010"
Jul 17 17:37:26.623980 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.623941    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:26.759994 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.759964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:26.766075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.766031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:37:26.851412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.851375    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca\") pod \"4a744b2a-36dd-4d07-8a19-24e740863010\" (UID: \"4a744b2a-36dd-4d07-8a19-24e740863010\") "
Jul 17 17:37:26.851761 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.851738    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a744b2a-36dd-4d07-8a19-24e740863010" (UID: "4a744b2a-36dd-4d07-8a19-24e740863010"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:37:26.953106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:26.952993    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a744b2a-36dd-4d07-8a19-24e740863010-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:37:27.518061 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.518000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:27.609931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.609899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72" event={"ID":"4a744b2a-36dd-4d07-8a19-24e740863010","Type":"ContainerDied","Data":"33b4488a03d795deb12531f6d5057ab350824eaac8b28810fb57629c342d741f"}
Jul 17 17:37:27.609931 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.609929    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"
Jul 17 17:37:27.610198 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.609959    2561 scope.go:117] "RemoveContainer" containerID="c719609b4f2545c896ede30a82821df33529ffb08a92bb89420eef5ad8c36da2"
Jul 17 17:37:27.622219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.622147    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:27.638671 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.638640    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:27.642562 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:27.642533    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c607776-w8h72"]
Jul 17 17:37:28.514412 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:28.514379    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" path="/var/lib/kubelet/pods/4a744b2a-36dd-4d07-8a19-24e740863010/volumes"
Jul 17 17:37:29.508441 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:29.508408    2561 scope.go:117] "RemoveContainer" containerID="f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966"
Jul 17 17:37:30.626939 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:30.626905    2561 generic.go:356] "Generic (PLEG): container finished" podID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerID="95adb4985a9aca487e38a98d6c83470b0784cc3094a63587ce867552c27f2f96" exitCode=7
Jul 17 17:37:30.627400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:30.626972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" event={"ID":"b458466c-354b-475e-9ecd-3f27a7990e8a","Type":"ContainerDied","Data":"95adb4985a9aca487e38a98d6c83470b0784cc3094a63587ce867552c27f2f96"}
Jul 17 17:37:30.627400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:30.627031    2561 scope.go:117] "RemoveContainer" containerID="f35808f4e6d98b7edaf4aba1695dfc8e1784a3c54d0a8d5555392bb25eea1966"
Jul 17 17:37:30.627400 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:30.627333    2561 scope.go:117] "RemoveContainer" containerID="95adb4985a9aca487e38a98d6c83470b0784cc3094a63587ce867552c27f2f96"
Jul 17 17:37:30.627634 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:30.627609    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz_odh-ai-gateway-infra(b458466c-354b-475e-9ecd-3f27a7990e8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a"
Jul 17 17:37:31.654108 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:31.654074    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:31.792896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:31.792870    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:31.897978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:31.897915    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca\") pod \"b458466c-354b-475e-9ecd-3f27a7990e8a\" (UID: \"b458466c-354b-475e-9ecd-3f27a7990e8a\") "
Jul 17 17:37:31.898340 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:31.898317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b458466c-354b-475e-9ecd-3f27a7990e8a" (UID: "b458466c-354b-475e-9ecd-3f27a7990e8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:37:31.999488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:31.999374    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b458466c-354b-475e-9ecd-3f27a7990e8a-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:37:32.508422 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.508381    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:37:32.508643 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:32.508600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:37:32.640110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.640076    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"
Jul 17 17:37:32.640110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.640093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz" event={"ID":"b458466c-354b-475e-9ecd-3f27a7990e8a","Type":"ContainerDied","Data":"a05685f39e690f5147c72bca7625da48eb6975a53d42188463eb61f7336c65cc"}
Jul 17 17:37:32.640404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.640171    2561 scope.go:117] "RemoveContainer" containerID="95adb4985a9aca487e38a98d6c83470b0784cc3094a63587ce867552c27f2f96"
Jul 17 17:37:32.657009 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.656975    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:32.661269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:32.661236    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-knxlz"]
Jul 17 17:37:34.065002 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:34.064964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:34.514350 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:34.514256    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" path="/var/lib/kubelet/pods/b458466c-354b-475e-9ecd-3f27a7990e8a/volumes"
Jul 17 17:37:36.791304 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791270    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:36.791823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791794    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.791823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791814    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791855    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791865    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791911    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791920    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791931    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791940    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.791954 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791961    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.791978    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb48b40-4ae4-4502-a925-e09f15e660c4" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.792102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.792115    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.792126    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:36.792338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.792142    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:36.796574 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.796552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:36.799762 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.799741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:37:36.802004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.801979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:36.948837 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:36.948796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k\" (UID: \"1e559812-b138-4ab5-a526-397e9489c04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:37.049969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.049876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k\" (UID: \"1e559812-b138-4ab5-a526-397e9489c04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:37.050558 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.050533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k\" (UID: \"1e559812-b138-4ab5-a526-397e9489c04e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:37.108888 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.108833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:37.242523 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.242498    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:37.245568 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:37.245531    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e559812_b138_4ab5_a526_397e9489c04e.slice/crio-30fc91497375affc385b92b0c34f1def9874875251621c31de6216f914e68ec5 WatchSource:0}: Error finding container 30fc91497375affc385b92b0c34f1def9874875251621c31de6216f914e68ec5: Status 404 returned error can't find the container with id 30fc91497375affc385b92b0c34f1def9874875251621c31de6216f914e68ec5
Jul 17 17:37:37.667646 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.667607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerStarted","Data":"ec6404f6304aa1bf1de5a649d45c0a68177490235b6e9cfbea5f0f5bb184d305"}
Jul 17 17:37:37.667863 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.667665    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerStarted","Data":"30fc91497375affc385b92b0c34f1def9874875251621c31de6216f914e68ec5"}
Jul 17 17:37:37.684424 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:37.684321    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" podStartSLOduration=1.6843024469999999 podStartE2EDuration="1.684302447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:37:37.681996383 +0000 UTC m=+4291.667083920" watchObservedRunningTime="2026-07-17 17:37:37.684302447 +0000 UTC m=+4291.669389985"
Jul 17 17:37:38.673537 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:38.673506    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e559812-b138-4ab5-a526-397e9489c04e" containerID="ec6404f6304aa1bf1de5a649d45c0a68177490235b6e9cfbea5f0f5bb184d305" exitCode=7
Jul 17 17:37:38.673941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:38.673589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerDied","Data":"ec6404f6304aa1bf1de5a649d45c0a68177490235b6e9cfbea5f0f5bb184d305"}
Jul 17 17:37:38.673941 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:38.673748    2561 scope.go:117] "RemoveContainer" containerID="ec6404f6304aa1bf1de5a649d45c0a68177490235b6e9cfbea5f0f5bb184d305"
Jul 17 17:37:39.679874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:39.679842    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e559812-b138-4ab5-a526-397e9489c04e" containerID="f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747" exitCode=7
Jul 17 17:37:39.680413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:39.679914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerDied","Data":"f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747"}
Jul 17 17:37:39.680413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:39.679983    2561 scope.go:117] "RemoveContainer" containerID="ec6404f6304aa1bf1de5a649d45c0a68177490235b6e9cfbea5f0f5bb184d305"
Jul 17 17:37:39.680413 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:39.680213    2561 scope.go:117] "RemoveContainer" containerID="f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747"
Jul 17 17:37:39.680592 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:39.680460    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k_odh-ai-gateway-infra(1e559812-b138-4ab5-a526-397e9489c04e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" podUID="1e559812-b138-4ab5-a526-397e9489c04e"
Jul 17 17:37:40.509224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:40.509187    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:37:40.509411 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:40.509358    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:37:40.686117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:40.686091    2561 scope.go:117] "RemoveContainer" containerID="f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747"
Jul 17 17:37:40.686578 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:40.686311    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k_odh-ai-gateway-infra(1e559812-b138-4ab5-a526-397e9489c04e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" podUID="1e559812-b138-4ab5-a526-397e9489c04e"
Jul 17 17:37:42.554019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:42.553983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:37:44.226659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.226623    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:37:44.227087 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.227012    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:44.227087 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.227023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:44.227179 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.227094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a744b2a-36dd-4d07-8a19-24e740863010" containerName="revoke-keys"
Jul 17 17:37:44.227179 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.227108    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b458466c-354b-475e-9ecd-3f27a7990e8a" containerName="revoke-keys"
Jul 17 17:37:44.231478 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.231460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:37:44.238402 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.238370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:37:44.321634 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.321586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct\" (UID: \"b70fa2a5-267f-4677-971b-6d91862f8ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:37:44.422985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.422948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct\" (UID: \"b70fa2a5-267f-4677-971b-6d91862f8ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:37:44.423593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.423571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct\" (UID: \"b70fa2a5-267f-4677-971b-6d91862f8ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:37:44.543436 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.543403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:37:44.680701 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.680659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:37:44.680870 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:44.680849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70fa2a5_267f_4677_971b_6d91862f8ae5.slice/crio-2f46333647abcf8334084896233a423040fb0964c81f589a4da93768bb9ddfbf WatchSource:0}: Error finding container 2f46333647abcf8334084896233a423040fb0964c81f589a4da93768bb9ddfbf: Status 404 returned error can't find the container with id 2f46333647abcf8334084896233a423040fb0964c81f589a4da93768bb9ddfbf
Jul 17 17:37:44.706789 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:44.706763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" event={"ID":"b70fa2a5-267f-4677-971b-6d91862f8ae5","Type":"ContainerStarted","Data":"2f46333647abcf8334084896233a423040fb0964c81f589a4da93768bb9ddfbf"}
Jul 17 17:37:45.712616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:45.712522    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerID="29eaf875cbb4615426f0848e1e8ca00e7438512e61de01ec5a89aab0b6bc4e44" exitCode=7
Jul 17 17:37:45.712616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:45.712601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" event={"ID":"b70fa2a5-267f-4677-971b-6d91862f8ae5","Type":"ContainerDied","Data":"29eaf875cbb4615426f0848e1e8ca00e7438512e61de01ec5a89aab0b6bc4e44"}
Jul 17 17:37:45.713140 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:45.712804    2561 scope.go:117] "RemoveContainer" containerID="29eaf875cbb4615426f0848e1e8ca00e7438512e61de01ec5a89aab0b6bc4e44"
Jul 17 17:37:46.510903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:46.510873    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:37:46.511102 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:46.511074    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:37:46.719513 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:46.719426    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerID="8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838" exitCode=7
Jul 17 17:37:46.719513 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:46.719480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" event={"ID":"b70fa2a5-267f-4677-971b-6d91862f8ae5","Type":"ContainerDied","Data":"8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838"}
Jul 17 17:37:46.719978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:46.719527    2561 scope.go:117] "RemoveContainer" containerID="29eaf875cbb4615426f0848e1e8ca00e7438512e61de01ec5a89aab0b6bc4e44"
Jul 17 17:37:46.719978 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:46.719721    2561 scope.go:117] "RemoveContainer" containerID="8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838"
Jul 17 17:37:46.720100 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:46.719981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct_odh-ai-gateway-infra(b70fa2a5-267f-4677-971b-6d91862f8ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5"
Jul 17 17:37:47.725991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:47.725957    2561 scope.go:117] "RemoveContainer" containerID="8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838"
Jul 17 17:37:47.726489 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:47.726277    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct_odh-ai-gateway-infra(b70fa2a5-267f-4677-971b-6d91862f8ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5"
Jul 17 17:37:50.315257 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:50.315221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:51.182904 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:51.182873    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:37:51.202584 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:51.202549    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:52.631094 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:52.631035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:37:54.968975 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:54.968932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-139893856617520"]
Jul 17 17:37:54.973478 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:54.973461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:54.976134 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:54.976107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-24kmc\""
Jul 17 17:37:54.980480 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:54.980450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856617520"]
Jul 17 17:37:55.009448 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.009411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4nn\" (UniqueName: \"kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn\") pod \"test-curl-7546-139893856617520\" (UID: \"b2eb06c6-fb60-445d-96a6-ecf9bacf393d\") " pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:55.110300 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.110266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4nn\" (UniqueName: \"kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn\") pod \"test-curl-7546-139893856617520\" (UID: \"b2eb06c6-fb60-445d-96a6-ecf9bacf393d\") " pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:55.118919 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.118885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4nn\" (UniqueName: \"kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn\") pod \"test-curl-7546-139893856617520\" (UID: \"b2eb06c6-fb60-445d-96a6-ecf9bacf393d\") " pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:55.283482 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.283449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:55.423588 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.423551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856617520"]
Jul 17 17:37:55.424716 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:55.424689    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2eb06c6_fb60_445d_96a6_ecf9bacf393d.slice/crio-9ebc03760e5c48c99e673767d6aaa9577e92394806ec8929080dd0e2312e8e55 WatchSource:0}: Error finding container 9ebc03760e5c48c99e673767d6aaa9577e92394806ec8929080dd0e2312e8e55: Status 404 returned error can't find the container with id 9ebc03760e5c48c99e673767d6aaa9577e92394806ec8929080dd0e2312e8e55
Jul 17 17:37:55.508847 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.508818    2561 scope.go:117] "RemoveContainer" containerID="f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747"
Jul 17 17:37:55.509084 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.508955    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:37:55.509210 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:55.509176    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:37:55.769236 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:55.769195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856617520" event={"ID":"b2eb06c6-fb60-445d-96a6-ecf9bacf393d","Type":"ContainerStarted","Data":"9ebc03760e5c48c99e673767d6aaa9577e92394806ec8929080dd0e2312e8e55"}
Jul 17 17:37:56.775316 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.775279    2561 generic.go:356] "Generic (PLEG): container finished" podID="b2eb06c6-fb60-445d-96a6-ecf9bacf393d" containerID="2915838baad063f1024929f3a6ef65a7a52c1c5d20ef4cda0ae301c196247f54" exitCode=0
Jul 17 17:37:56.775894 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.775367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856617520" event={"ID":"b2eb06c6-fb60-445d-96a6-ecf9bacf393d","Type":"ContainerDied","Data":"2915838baad063f1024929f3a6ef65a7a52c1c5d20ef4cda0ae301c196247f54"}
Jul 17 17:37:56.777373 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.777352    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e559812-b138-4ab5-a526-397e9489c04e" containerID="09e6c6bfdf5841596ac25d01fd663b57f36fa34c402dd42f6d45a19fc567175c" exitCode=7
Jul 17 17:37:56.777486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.777421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerDied","Data":"09e6c6bfdf5841596ac25d01fd663b57f36fa34c402dd42f6d45a19fc567175c"}
Jul 17 17:37:56.777486 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.777476    2561 scope.go:117] "RemoveContainer" containerID="f0914670addccddd51baca60ba4122bc95cd6a6a853d4e2bbdd8038dc7d89747"
Jul 17 17:37:56.777763 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:56.777748    2561 scope.go:117] "RemoveContainer" containerID="09e6c6bfdf5841596ac25d01fd663b57f36fa34c402dd42f6d45a19fc567175c"
Jul 17 17:37:56.778039 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:56.778015    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k_odh-ai-gateway-infra(1e559812-b138-4ab5-a526-397e9489c04e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" podUID="1e559812-b138-4ab5-a526-397e9489c04e"
Jul 17 17:37:57.661480 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.661439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:37:57.816843 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.816801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:57.932341 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.932311    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:57.960225 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.955754    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-139893856617520_b2eb06c6-fb60-445d-96a6-ecf9bacf393d/test-curl-7546-139893856617520/0.log"
Jul 17 17:37:57.971654 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.971631    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:57.984077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.984037    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-139893856617520"]
Jul 17 17:37:57.994345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:57.994307    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-139893856617520"]
Jul 17 17:37:58.040228 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.040194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca\") pod \"1e559812-b138-4ab5-a526-397e9489c04e\" (UID: \"1e559812-b138-4ab5-a526-397e9489c04e\") "
Jul 17 17:37:58.040442 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.040235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq4nn\" (UniqueName: \"kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn\") pod \"b2eb06c6-fb60-445d-96a6-ecf9bacf393d\" (UID: \"b2eb06c6-fb60-445d-96a6-ecf9bacf393d\") "
Jul 17 17:37:58.040637 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.040611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e559812-b138-4ab5-a526-397e9489c04e" (UID: "1e559812-b138-4ab5-a526-397e9489c04e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:37:58.042440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.042416    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn" (OuterVolumeSpecName: "kube-api-access-qq4nn") pod "b2eb06c6-fb60-445d-96a6-ecf9bacf393d" (UID: "b2eb06c6-fb60-445d-96a6-ecf9bacf393d"). InnerVolumeSpecName "kube-api-access-qq4nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:37:58.141243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.141207    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e559812-b138-4ab5-a526-397e9489c04e-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:37:58.141243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.141239    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq4nn\" (UniqueName: \"kubernetes.io/projected/b2eb06c6-fb60-445d-96a6-ecf9bacf393d-kube-api-access-qq4nn\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:37:58.155356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155323    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:37:58.155728 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155715    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155731    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155750    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2eb06c6-fb60-445d-96a6-ecf9bacf393d" containerName="test-curl-7546-139893856617520"
Jul 17 17:37:58.155822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2eb06c6-fb60-445d-96a6-ecf9bacf393d" containerName="test-curl-7546-139893856617520"
Jul 17 17:37:58.155822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155775    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155822 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155781    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155843    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155853    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:37:58.155987 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.155860    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2eb06c6-fb60-445d-96a6-ecf9bacf393d" containerName="test-curl-7546-139893856617520"
Jul 17 17:37:58.158996 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.158975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:37:58.165114 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.165022    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:37:58.242221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.242183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscr8\" (UniqueName: \"kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8\") pod \"test-curl-7546-139893856445040\" (UID: \"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:37:58.342783 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.342695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscr8\" (UniqueName: \"kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8\") pod \"test-curl-7546-139893856445040\" (UID: \"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:37:58.352500 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.352469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscr8\" (UniqueName: \"kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8\") pod \"test-curl-7546-139893856445040\" (UID: \"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:37:58.468269 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.468185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:37:58.509532 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.508849    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:37:58.509532 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:37:58.509174    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:37:58.514923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.514889    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2eb06c6-fb60-445d-96a6-ecf9bacf393d" path="/var/lib/kubelet/pods/b2eb06c6-fb60-445d-96a6-ecf9bacf393d/volumes"
Jul 17 17:37:58.603668 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.603642    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:37:58.605990 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:37:58.605960    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f5ac49c_f63a_4000_bf08_c5e884dbc0d4.slice/crio-fa2087aa84fc13ad1b7e46ad65d8c741239de75869e52322c1042e741b0aa58c WatchSource:0}: Error finding container fa2087aa84fc13ad1b7e46ad65d8c741239de75869e52322c1042e741b0aa58c: Status 404 returned error can't find the container with id fa2087aa84fc13ad1b7e46ad65d8c741239de75869e52322c1042e741b0aa58c
Jul 17 17:37:58.790855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.790826    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"
Jul 17 17:37:58.791086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.790856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k" event={"ID":"1e559812-b138-4ab5-a526-397e9489c04e","Type":"ContainerDied","Data":"30fc91497375affc385b92b0c34f1def9874875251621c31de6216f914e68ec5"}
Jul 17 17:37:58.791086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.790924    2561 scope.go:117] "RemoveContainer" containerID="09e6c6bfdf5841596ac25d01fd663b57f36fa34c402dd42f6d45a19fc567175c"
Jul 17 17:37:58.792386 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.792358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856445040" event={"ID":"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4","Type":"ContainerStarted","Data":"fa2087aa84fc13ad1b7e46ad65d8c741239de75869e52322c1042e741b0aa58c"}
Jul 17 17:37:58.793963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.793942    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856617520"
Jul 17 17:37:58.800286 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.800270    2561 scope.go:117] "RemoveContainer" containerID="2915838baad063f1024929f3a6ef65a7a52c1c5d20ef4cda0ae301c196247f54"
Jul 17 17:37:58.808327 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.808277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:58.810673 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:58.810649    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-9v79k"]
Jul 17 17:37:59.800923 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:59.800885    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" containerID="e8b644203570fd73fd1688b1ba6f23fbcbe62ebf8537aa948a3226038170f0b3" exitCode=0
Jul 17 17:37:59.801392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:37:59.800974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856445040" event={"ID":"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4","Type":"ContainerDied","Data":"e8b644203570fd73fd1688b1ba6f23fbcbe62ebf8537aa948a3226038170f0b3"}
Jul 17 17:38:00.514565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.514535    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e559812-b138-4ab5-a526-397e9489c04e" path="/var/lib/kubelet/pods/1e559812-b138-4ab5-a526-397e9489c04e/volumes"
Jul 17 17:38:00.944381 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.944357    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:00.964024 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.964000    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-139893856445040_0f5ac49c-f63a-4000-bf08-c5e884dbc0d4/test-curl-7546-139893856445040/0.log"
Jul 17 17:38:00.966220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.966199    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bscr8\" (UniqueName: \"kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8\") pod \"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4\" (UID: \"0f5ac49c-f63a-4000-bf08-c5e884dbc0d4\") "
Jul 17 17:38:00.968219 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.968197    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8" (OuterVolumeSpecName: "kube-api-access-bscr8") pod "0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" (UID: "0f5ac49c-f63a-4000-bf08-c5e884dbc0d4"). InnerVolumeSpecName "kube-api-access-bscr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:00.987465 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.987435    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:00.991550 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:00.991527    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:01.067009 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.066915    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscr8\" (UniqueName: \"kubernetes.io/projected/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4-kube-api-access-bscr8\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:01.110661 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.110616    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:01.111115 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111096    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:01.111224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111119    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:01.111224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111146    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:38:01.111224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111154    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:38:01.111392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111272    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:01.111392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.111289    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e559812-b138-4ab5-a526-397e9489c04e" containerName="revoke-keys"
Jul 17 17:38:01.114331 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.114309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.120455 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.120427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:01.168169 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.168134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djsh\" (UniqueName: \"kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh\") pod \"test-curl-7546-139893856445040\" (UID: \"d02511b6-a4f9-4697-9156-971de0f49901\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.269031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.268988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djsh\" (UniqueName: \"kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh\") pod \"test-curl-7546-139893856445040\" (UID: \"d02511b6-a4f9-4697-9156-971de0f49901\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.279168 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.279136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djsh\" (UniqueName: \"kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh\") pod \"test-curl-7546-139893856445040\" (UID: \"d02511b6-a4f9-4697-9156-971de0f49901\") " pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.425224 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.425131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.558409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.558376    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:01.560867 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:01.560837    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02511b6_a4f9_4697_9156_971de0f49901.slice/crio-f2f9f0b4e7844131477ce7c9ef0940c65f9de09c242115a3a6e4afa520a7c147 WatchSource:0}: Error finding container f2f9f0b4e7844131477ce7c9ef0940c65f9de09c242115a3a6e4afa520a7c147: Status 404 returned error can't find the container with id f2f9f0b4e7844131477ce7c9ef0940c65f9de09c242115a3a6e4afa520a7c147
Jul 17 17:38:01.815589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.815556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856445040" event={"ID":"d02511b6-a4f9-4697-9156-971de0f49901","Type":"ContainerStarted","Data":"f2f9f0b4e7844131477ce7c9ef0940c65f9de09c242115a3a6e4afa520a7c147"}
Jul 17 17:38:01.816679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.816658    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2087aa84fc13ad1b7e46ad65d8c741239de75869e52322c1042e741b0aa58c"
Jul 17 17:38:01.816679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.816675    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:01.829580 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:01.829547    2561 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7546-139893856445040" oldPodUID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" podUID="d02511b6-a4f9-4697-9156-971de0f49901"
Jul 17 17:38:02.516276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:02.509583    2561 scope.go:117] "RemoveContainer" containerID="8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838"
Jul 17 17:38:02.520532 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:02.520503    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f5ac49c-f63a-4000-bf08-c5e884dbc0d4" path="/var/lib/kubelet/pods/0f5ac49c-f63a-4000-bf08-c5e884dbc0d4/volumes"
Jul 17 17:38:02.822823 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:02.822795    2561 generic.go:356] "Generic (PLEG): container finished" podID="d02511b6-a4f9-4697-9156-971de0f49901" containerID="c0418f4fcec74ddb3abab0c4e04c3d6c90c7103b6f93c2f70e699cec240585d3" exitCode=0
Jul 17 17:38:02.822969 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:02.822862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856445040" event={"ID":"d02511b6-a4f9-4697-9156-971de0f49901","Type":"ContainerDied","Data":"c0418f4fcec74ddb3abab0c4e04c3d6c90c7103b6f93c2f70e699cec240585d3"}
Jul 17 17:38:03.829821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.829786    2561 generic.go:356] "Generic (PLEG): container finished" podID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerID="8de70fe6a0262ea407496e6fe9f3a564b5b69c5b3e277515eda9cb4175486ac2" exitCode=7
Jul 17 17:38:03.830345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.829860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" event={"ID":"b70fa2a5-267f-4677-971b-6d91862f8ae5","Type":"ContainerDied","Data":"8de70fe6a0262ea407496e6fe9f3a564b5b69c5b3e277515eda9cb4175486ac2"}
Jul 17 17:38:03.830345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.829932    2561 scope.go:117] "RemoveContainer" containerID="8d43c129ae127050aab6606960028e9550f8edb64818521e1ceb51d1d9763838"
Jul 17 17:38:03.830345 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.830239    2561 scope.go:117] "RemoveContainer" containerID="8de70fe6a0262ea407496e6fe9f3a564b5b69c5b3e277515eda9cb4175486ac2"
Jul 17 17:38:03.830876 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:03.830498    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct_odh-ai-gateway-infra(b70fa2a5-267f-4677-971b-6d91862f8ae5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5"
Jul 17 17:38:03.967776 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.967751    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:03.990409 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.990343    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-139893856445040_d02511b6-a4f9-4697-9156-971de0f49901/test-curl-7546-139893856445040/0.log"
Jul 17 17:38:03.993874 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.993844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5djsh\" (UniqueName: \"kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh\") pod \"d02511b6-a4f9-4697-9156-971de0f49901\" (UID: \"d02511b6-a4f9-4697-9156-971de0f49901\") "
Jul 17 17:38:03.996023 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:03.995992    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh" (OuterVolumeSpecName: "kube-api-access-5djsh") pod "d02511b6-a4f9-4697-9156-971de0f49901" (UID: "d02511b6-a4f9-4697-9156-971de0f49901"). InnerVolumeSpecName "kube-api-access-5djsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:04.014097 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.014039    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:04.017982 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.017954    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-139893856445040"]
Jul 17 17:38:04.095575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.095490    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djsh\" (UniqueName: \"kubernetes.io/projected/d02511b6-a4f9-4697-9156-971de0f49901-kube-api-access-5djsh\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:04.140467 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.140428    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:04.140848 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.140832    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02511b6-a4f9-4697-9156-971de0f49901" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:04.140899 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.140850    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02511b6-a4f9-4697-9156-971de0f49901" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:04.140934 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.140919    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02511b6-a4f9-4697-9156-971de0f49901" containerName="test-curl-7546-139893856445040"
Jul 17 17:38:04.143973 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.143953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:04.151075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.151031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:04.196862 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.196825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmtn\" (UniqueName: \"kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn\") pod \"test-curl-7546-139893856446048\" (UID: \"4931eaf5-6692-499d-8712-94fa5d36f029\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:04.297675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.297639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmtn\" (UniqueName: \"kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn\") pod \"test-curl-7546-139893856446048\" (UID: \"4931eaf5-6692-499d-8712-94fa5d36f029\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:04.307552 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.307520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmtn\" (UniqueName: \"kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn\") pod \"test-curl-7546-139893856446048\" (UID: \"4931eaf5-6692-499d-8712-94fa5d36f029\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:04.454142 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.454035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:04.516106 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.516074    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02511b6-a4f9-4697-9156-971de0f49901" path="/var/lib/kubelet/pods/d02511b6-a4f9-4697-9156-971de0f49901/volumes"
Jul 17 17:38:04.591943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.591912    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:04.593673 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:04.593648    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4931eaf5_6692_499d_8712_94fa5d36f029.slice/crio-df18cf6737e6d182bba687b6d6b8d9eefcc703e837655d00f4e69e4b640926bd WatchSource:0}: Error finding container df18cf6737e6d182bba687b6d6b8d9eefcc703e837655d00f4e69e4b640926bd: Status 404 returned error can't find the container with id df18cf6737e6d182bba687b6d6b8d9eefcc703e837655d00f4e69e4b640926bd
Jul 17 17:38:04.836033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.835998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856446048" event={"ID":"4931eaf5-6692-499d-8712-94fa5d36f029","Type":"ContainerStarted","Data":"df18cf6737e6d182bba687b6d6b8d9eefcc703e837655d00f4e69e4b640926bd"}
Jul 17 17:38:04.838690 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.838672    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856445040"
Jul 17 17:38:04.838813 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.838720    2561 scope.go:117] "RemoveContainer" containerID="c0418f4fcec74ddb3abab0c4e04c3d6c90c7103b6f93c2f70e699cec240585d3"
Jul 17 17:38:04.857395 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:04.857370    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:38:05.007951 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.007926    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:38:05.105820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.105727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca\") pod \"b70fa2a5-267f-4677-971b-6d91862f8ae5\" (UID: \"b70fa2a5-267f-4677-971b-6d91862f8ae5\") "
Jul 17 17:38:05.106118 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.106097    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b70fa2a5-267f-4677-971b-6d91862f8ae5" (UID: "b70fa2a5-267f-4677-971b-6d91862f8ae5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:05.206538 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.206506    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b70fa2a5-267f-4677-971b-6d91862f8ae5-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:05.846001 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.845961    2561 generic.go:356] "Generic (PLEG): container finished" podID="4931eaf5-6692-499d-8712-94fa5d36f029" containerID="574d7929f0a9d918b12a963016d12cff960504add606e224710f143df7b2683c" exitCode=0
Jul 17 17:38:05.846488 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.846052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856446048" event={"ID":"4931eaf5-6692-499d-8712-94fa5d36f029","Type":"ContainerDied","Data":"574d7929f0a9d918b12a963016d12cff960504add606e224710f143df7b2683c"}
Jul 17 17:38:05.847371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.847354    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"
Jul 17 17:38:05.847371 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.847359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct" event={"ID":"b70fa2a5-267f-4677-971b-6d91862f8ae5","Type":"ContainerDied","Data":"2f46333647abcf8334084896233a423040fb0964c81f589a4da93768bb9ddfbf"}
Jul 17 17:38:05.847538 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.847418    2561 scope.go:117] "RemoveContainer" containerID="8de70fe6a0262ea407496e6fe9f3a564b5b69c5b3e277515eda9cb4175486ac2"
Jul 17 17:38:05.877254 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.877220    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:38:05.882549 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:05.882513    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cb0f3bd-pxjct"]
Jul 17 17:38:06.203712 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.203624    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" podUID="803f444d-6d1b-4f4f-a510-27744091b5b7" containerName="main" containerID="cri-o://4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c" gracePeriod=45
Jul 17 17:38:06.470528 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.470502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:38:06.511597 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.511569    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:38:06.511796 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:06.511775    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:06.517182 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517151    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" path="/var/lib/kubelet/pods/b70fa2a5-267f-4677-971b-6d91862f8ae5/volumes"
Jul 17 17:38:06.517348 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517405 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517447 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517485 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517542 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm9kw\" (UniqueName: \"kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs\") pod \"803f444d-6d1b-4f4f-a510-27744091b5b7\" (UID: \"803f444d-6d1b-4f4f-a510-27744091b5b7\") "
Jul 17 17:38:06.517659 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.517636    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home" (OuterVolumeSpecName: "home") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:38:06.518170 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.518140    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:38:06.518522 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.518478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:38:06.520203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.519648    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-home\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.526621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.521596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw" (OuterVolumeSpecName: "kube-api-access-lm9kw") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "kube-api-access-lm9kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:06.526621 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.521833    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:38:06.526784 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.526675    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm" (OuterVolumeSpecName: "dshm") pod "803f444d-6d1b-4f4f-a510-27744091b5b7" (UID: "803f444d-6d1b-4f4f-a510-27744091b5b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 17:38:06.621231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.621195    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9kw\" (UniqueName: \"kubernetes.io/projected/803f444d-6d1b-4f4f-a510-27744091b5b7-kube-api-access-lm9kw\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.621231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.621225    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/803f444d-6d1b-4f4f-a510-27744091b5b7-tls-certs\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.621231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.621236    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-tmp-dir\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.621231 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.621244    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-model-cache\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.621531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.621255    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/803f444d-6d1b-4f4f-a510-27744091b5b7-dshm\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:06.855220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.855178    2561 generic.go:356] "Generic (PLEG): container finished" podID="803f444d-6d1b-4f4f-a510-27744091b5b7" containerID="4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c" exitCode=0
Jul 17 17:38:06.855739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.855241    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"
Jul 17 17:38:06.855739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.855259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" event={"ID":"803f444d-6d1b-4f4f-a510-27744091b5b7","Type":"ContainerDied","Data":"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"}
Jul 17 17:38:06.855739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.855299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb" event={"ID":"803f444d-6d1b-4f4f-a510-27744091b5b7","Type":"ContainerDied","Data":"c103c6c3b9ebab13bbb285547fbd28f39b8a7f691ab7d2f37ee8b48149e1dcd6"}
Jul 17 17:38:06.855739 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.855316    2561 scope.go:117] "RemoveContainer" containerID="4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"
Jul 17 17:38:06.865226 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.865207    2561 scope.go:117] "RemoveContainer" containerID="4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"
Jul 17 17:38:06.865495 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:06.865473    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c\": container with ID starting with 4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c not found: ID does not exist" containerID="4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"
Jul 17 17:38:06.865599 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.865499    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c"} err="failed to get container status \"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c\": rpc error: code = NotFound desc = could not find container \"4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c\": container with ID starting with 4ad400b332fff2fba37ef3f2956ace2ce4e209af8940d53b1e454c42c057141c not found: ID does not exist"
Jul 17 17:38:06.880122 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.880085    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:38:06.886171 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.886091    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f5e7b/rate-test-model-4f5e7b-kserve-d7c4f8cdc-tkhxb"]
Jul 17 17:38:06.989536 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:06.989511    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.013675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.013644    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-139893856446048_4931eaf5-6692-499d-8712-94fa5d36f029/test-curl-7546-139893856446048/0.log"
Jul 17 17:38:07.024111 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.024077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpmtn\" (UniqueName: \"kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn\") pod \"4931eaf5-6692-499d-8712-94fa5d36f029\" (UID: \"4931eaf5-6692-499d-8712-94fa5d36f029\") "
Jul 17 17:38:07.026181 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.026154    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn" (OuterVolumeSpecName: "kube-api-access-tpmtn") pod "4931eaf5-6692-499d-8712-94fa5d36f029" (UID: "4931eaf5-6692-499d-8712-94fa5d36f029"). InnerVolumeSpecName "kube-api-access-tpmtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:07.040679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.040636    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:07.045943 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.045917    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:07.125891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.125807    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmtn\" (UniqueName: \"kubernetes.io/projected/4931eaf5-6692-499d-8712-94fa5d36f029-kube-api-access-tpmtn\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:07.185080 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185032    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:07.185418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185406    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185419    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185437    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="803f444d-6d1b-4f4f-a510-27744091b5b7" containerName="main"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185443    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="803f444d-6d1b-4f4f-a510-27744091b5b7" containerName="main"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185451    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185456    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185466 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185463    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4931eaf5-6692-499d-8712-94fa5d36f029" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185470    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4931eaf5-6692-499d-8712-94fa5d36f029" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185483    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185545    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4931eaf5-6692-499d-8712-94fa5d36f029" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185552    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="803f444d-6d1b-4f4f-a510-27744091b5b7" containerName="main"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185559    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185565    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.185653 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.185572    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70fa2a5-267f-4677-971b-6d91862f8ae5" containerName="revoke-keys"
Jul 17 17:38:07.188575 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.188559    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.198505 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.198479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:07.226855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.226822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6h5\" (UniqueName: \"kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5\") pod \"test-curl-7546-139893856446048\" (UID: \"bfed529d-f58c-44ee-ae5d-bd309b5de80e\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.327510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.327469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6h5\" (UniqueName: \"kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5\") pod \"test-curl-7546-139893856446048\" (UID: \"bfed529d-f58c-44ee-ae5d-bd309b5de80e\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.335624 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.335586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6h5\" (UniqueName: \"kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5\") pod \"test-curl-7546-139893856446048\" (UID: \"bfed529d-f58c-44ee-ae5d-bd309b5de80e\") " pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.497636 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.497598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.635531 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.635499    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:07.638751 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:07.638724    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfed529d_f58c_44ee_ae5d_bd309b5de80e.slice/crio-14cbb55a3e00c62e27e503f3926c1fdf94e18b007204b7de198071e37640555d WatchSource:0}: Error finding container 14cbb55a3e00c62e27e503f3926c1fdf94e18b007204b7de198071e37640555d: Status 404 returned error can't find the container with id 14cbb55a3e00c62e27e503f3926c1fdf94e18b007204b7de198071e37640555d
Jul 17 17:38:07.863253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.863222    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:07.863253 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.863253    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df18cf6737e6d182bba687b6d6b8d9eefcc703e837655d00f4e69e4b640926bd"
Jul 17 17:38:07.864440 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.864415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856446048" event={"ID":"bfed529d-f58c-44ee-ae5d-bd309b5de80e","Type":"ContainerStarted","Data":"14cbb55a3e00c62e27e503f3926c1fdf94e18b007204b7de198071e37640555d"}
Jul 17 17:38:07.874779 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:07.874756    2561 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7546-139893856446048" oldPodUID="4931eaf5-6692-499d-8712-94fa5d36f029" podUID="bfed529d-f58c-44ee-ae5d-bd309b5de80e"
Jul 17 17:38:08.514963 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:08.514928    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4931eaf5-6692-499d-8712-94fa5d36f029" path="/var/lib/kubelet/pods/4931eaf5-6692-499d-8712-94fa5d36f029/volumes"
Jul 17 17:38:08.515363 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:08.515348    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="803f444d-6d1b-4f4f-a510-27744091b5b7" path="/var/lib/kubelet/pods/803f444d-6d1b-4f4f-a510-27744091b5b7/volumes"
Jul 17 17:38:08.871099 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:08.871039    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfed529d-f58c-44ee-ae5d-bd309b5de80e" containerID="8a4da122a18827e8723d4a00076cd47b803b11878584706ce7ac96e8fa90d559" exitCode=0
Jul 17 17:38:08.871529 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:08.871144    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-139893856446048" event={"ID":"bfed529d-f58c-44ee-ae5d-bd309b5de80e","Type":"ContainerDied","Data":"8a4da122a18827e8723d4a00076cd47b803b11878584706ce7ac96e8fa90d559"}
Jul 17 17:38:10.010930 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.010904    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:10.032582 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.032553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-139893856446048_bfed529d-f58c-44ee-ae5d-bd309b5de80e/test-curl-7546-139893856446048/0.log"
Jul 17 17:38:10.049797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.049769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr6h5\" (UniqueName: \"kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5\") pod \"bfed529d-f58c-44ee-ae5d-bd309b5de80e\" (UID: \"bfed529d-f58c-44ee-ae5d-bd309b5de80e\") "
Jul 17 17:38:10.051921 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.051895    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5" (OuterVolumeSpecName: "kube-api-access-tr6h5") pod "bfed529d-f58c-44ee-ae5d-bd309b5de80e" (UID: "bfed529d-f58c-44ee-ae5d-bd309b5de80e"). InnerVolumeSpecName "kube-api-access-tr6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:10.055898 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.055868    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:10.058565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.058533    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-139893856446048"]
Jul 17 17:38:10.151353 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.151259    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr6h5\" (UniqueName: \"kubernetes.io/projected/bfed529d-f58c-44ee-ae5d-bd309b5de80e-kube-api-access-tr6h5\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:10.238937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.238897    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"]
Jul 17 17:38:10.239310 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.239296    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfed529d-f58c-44ee-ae5d-bd309b5de80e" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:10.239352 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.239312    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfed529d-f58c-44ee-ae5d-bd309b5de80e" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:10.239404 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.239377    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfed529d-f58c-44ee-ae5d-bd309b5de80e" containerName="test-curl-7546-139893856446048"
Jul 17 17:38:10.242326 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.242300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:10.244423 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.244401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zx8f\""
Jul 17 17:38:10.248444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.248419    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"]
Jul 17 17:38:10.353645 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.353594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7\" (UID: \"558e9672-4379-4ee2-a343-2d7e0d9673ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:10.454821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.454719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7\" (UID: \"558e9672-4379-4ee2-a343-2d7e0d9673ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:10.455418 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.455396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7\" (UID: \"558e9672-4379-4ee2-a343-2d7e0d9673ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:10.514679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.514643    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfed529d-f58c-44ee-ae5d-bd309b5de80e" path="/var/lib/kubelet/pods/bfed529d-f58c-44ee-ae5d-bd309b5de80e/volumes"
Jul 17 17:38:10.559138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.559101    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:10.695781 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.695753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"]
Jul 17 17:38:10.698022 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:10.697996    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558e9672_4379_4ee2_a343_2d7e0d9673ec.slice/crio-3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079 WatchSource:0}: Error finding container 3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079: Status 404 returned error can't find the container with id 3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079
Jul 17 17:38:10.791600 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.791550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:10.883117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.883079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7" event={"ID":"558e9672-4379-4ee2-a343-2d7e0d9673ec","Type":"ContainerStarted","Data":"3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079"}
Jul 17 17:38:10.884283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.884265    2561 scope.go:117] "RemoveContainer" containerID="8a4da122a18827e8723d4a00076cd47b803b11878584706ce7ac96e8fa90d559"
Jul 17 17:38:10.884410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.884294    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-139893856446048"
Jul 17 17:38:10.892086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:10.892021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:11.751093 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.750983    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:11.754435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.754416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:11.761649 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.761619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:11.869259 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.869216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf\" (UID: \"8feff796-4f02-42be-a005-b3702f0779ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:11.891925 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.891882    2561 generic.go:356] "Generic (PLEG): container finished" podID="558e9672-4379-4ee2-a343-2d7e0d9673ec" containerID="7369313fd142209429c9e9b5e7981b73a3945bd96b62e491765c0d3e1be548db" exitCode=0
Jul 17 17:38:11.892129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.891971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7" event={"ID":"558e9672-4379-4ee2-a343-2d7e0d9673ec","Type":"ContainerDied","Data":"7369313fd142209429c9e9b5e7981b73a3945bd96b62e491765c0d3e1be548db"}
Jul 17 17:38:11.969933 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.969901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf\" (UID: \"8feff796-4f02-42be-a005-b3702f0779ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:11.970543 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:11.970524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf\" (UID: \"8feff796-4f02-42be-a005-b3702f0779ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:12.066099 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:12.066060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:12.198618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:12.198591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:12.201060 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:12.201026    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8feff796_4f02_42be_a005_b3702f0779ee.slice/crio-4ae001244df68f7b24287c557e33c56ce492753113548f4d8b6a2432bfdf517b WatchSource:0}: Error finding container 4ae001244df68f7b24287c557e33c56ce492753113548f4d8b6a2432bfdf517b: Status 404 returned error can't find the container with id 4ae001244df68f7b24287c557e33c56ce492753113548f4d8b6a2432bfdf517b
Jul 17 17:38:12.898220 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:12.898177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerStarted","Data":"a5449c2c83905748e615a9efa97ec6062e88c3da41d211e87c396eb1709815e2"}
Jul 17 17:38:12.898680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:12.898242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerStarted","Data":"4ae001244df68f7b24287c557e33c56ce492753113548f4d8b6a2432bfdf517b"}
Jul 17 17:38:12.920557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:12.920509    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" podStartSLOduration=1.920491918 podStartE2EDuration="1.920491918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:38:12.91740993 +0000 UTC m=+4326.902497496" watchObservedRunningTime="2026-07-17 17:38:12.920491918 +0000 UTC m=+4326.905579457"
Jul 17 17:38:13.027734 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.027702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:13.082154 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.082119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca\") pod \"558e9672-4379-4ee2-a343-2d7e0d9673ec\" (UID: \"558e9672-4379-4ee2-a343-2d7e0d9673ec\") "
Jul 17 17:38:13.082503 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.082478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "558e9672-4379-4ee2-a343-2d7e0d9673ec" (UID: "558e9672-4379-4ee2-a343-2d7e0d9673ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:13.184121 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.183993    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e9672-4379-4ee2-a343-2d7e0d9673ec-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:13.508742 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.508666    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:38:13.508919 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:13.508860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:38:13.905435 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.905400    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"
Jul 17 17:38:13.905896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.905396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7" event={"ID":"558e9672-4379-4ee2-a343-2d7e0d9673ec","Type":"ContainerDied","Data":"3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079"}
Jul 17 17:38:13.905896 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.905531    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3689eb2cbfcaec456b7d40074266a65636093b0b0ca9708540532bc94dc5a079"
Jul 17 17:38:13.907031 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.907001    2561 generic.go:356] "Generic (PLEG): container finished" podID="8feff796-4f02-42be-a005-b3702f0779ee" containerID="a5449c2c83905748e615a9efa97ec6062e88c3da41d211e87c396eb1709815e2" exitCode=7
Jul 17 17:38:13.907183 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.907032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerDied","Data":"a5449c2c83905748e615a9efa97ec6062e88c3da41d211e87c396eb1709815e2"}
Jul 17 17:38:13.907324 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:13.907306    2561 scope.go:117] "RemoveContainer" containerID="a5449c2c83905748e615a9efa97ec6062e88c3da41d211e87c396eb1709815e2"
Jul 17 17:38:14.914845 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:14.914758    2561 generic.go:356] "Generic (PLEG): container finished" podID="8feff796-4f02-42be-a005-b3702f0779ee" containerID="b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70" exitCode=7
Jul 17 17:38:14.914845 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:14.914813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerDied","Data":"b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70"}
Jul 17 17:38:14.915352 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:14.914908    2561 scope.go:117] "RemoveContainer" containerID="a5449c2c83905748e615a9efa97ec6062e88c3da41d211e87c396eb1709815e2"
Jul 17 17:38:14.915352 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:14.915120    2561 scope.go:117] "RemoveContainer" containerID="b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70"
Jul 17 17:38:14.915459 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:14.915354    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf_odh-ai-gateway-infra(8feff796-4f02-42be-a005-b3702f0779ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" podUID="8feff796-4f02-42be-a005-b3702f0779ee"
Jul 17 17:38:15.921137 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:15.921105    2561 scope.go:117] "RemoveContainer" containerID="b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70"
Jul 17 17:38:15.921535 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:15.921321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf_odh-ai-gateway-infra(8feff796-4f02-42be-a005-b3702f0779ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" podUID="8feff796-4f02-42be-a005-b3702f0779ee"
Jul 17 17:38:18.509152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:18.509117    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:38:18.509573 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:18.509332    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:20.443632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:20.443523    2561 scope.go:117] "RemoveContainer" containerID="62eff17b12be524f96cea57f3ab2733138616737333dc4d538b1ddb3ada393cf"
Jul 17 17:38:20.900828 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:20.900794    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:38:20.901035 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:20.901015    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" podUID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" containerName="maas-api" containerID="cri-o://660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720" gracePeriod=30
Jul 17 17:38:21.203676 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:21.203586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:22.152358 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.152330    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:38:22.275788 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.275758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls\") pod \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") "
Jul 17 17:38:22.275949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.275850    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca\") pod \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") "
Jul 17 17:38:22.275949 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.275919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8skz\" (UniqueName: \"kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz\") pod \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\" (UID: \"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e\") "
Jul 17 17:38:22.276255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.276220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" (UID: "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:22.277841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.277816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" (UID: "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:38:22.278067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.278031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz" (OuterVolumeSpecName: "kube-api-access-g8skz") pod "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" (UID: "9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e"). InnerVolumeSpecName "kube-api-access-g8skz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:22.377461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.377419    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:22.377461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.377453    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8skz\" (UniqueName: \"kubernetes.io/projected/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-kube-api-access-g8skz\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:22.377461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.377463    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:22.958948 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.958908    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" containerID="660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720" exitCode=0
Jul 17 17:38:22.959146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.958954    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"
Jul 17 17:38:22.959146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.958983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" event={"ID":"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e","Type":"ContainerDied","Data":"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"}
Jul 17 17:38:22.959146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.959037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m" event={"ID":"9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e","Type":"ContainerDied","Data":"c4c445811c0bccc71eed42a0356f3b94c909220471f5ddd23d8c0a02702b678e"}
Jul 17 17:38:22.959146 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.959072    2561 scope.go:117] "RemoveContainer" containerID="660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"
Jul 17 17:38:22.968078 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.968023    2561 scope.go:117] "RemoveContainer" containerID="660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"
Jul 17 17:38:22.968382 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:22.968350    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720\": container with ID starting with 660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720 not found: ID does not exist" containerID="660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"
Jul 17 17:38:22.968480 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.968389    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720"} err="failed to get container status \"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720\": rpc error: code = NotFound desc = could not find container \"660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720\": container with ID starting with 660f652f5db122298be558e1da785075c2d36f46df9e1739e3ddd2e2629e4720 not found: ID does not exist"
Jul 17 17:38:22.982438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.982408    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:38:22.985865 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:22.985839    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3d4103-5777757d8c-p2z9m"]
Jul 17 17:38:24.399255 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:24.399219    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:38:24.513909 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:24.513867    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" path="/var/lib/kubelet/pods/9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e/volumes"
Jul 17 17:38:25.372067 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:25.372018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:25.404283 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:25.404248    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"]
Jul 17 17:38:25.414697 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:25.414668    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3d4103-hclh7"]
Jul 17 17:38:25.508356 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:25.508302    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:38:25.508579 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:25.508559    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:38:25.841382 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:25.841345    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:26.514360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:26.514326    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558e9672-4379-4ee2-a343-2d7e0d9673ec" path="/var/lib/kubelet/pods/558e9672-4379-4ee2-a343-2d7e0d9673ec/volumes"
Jul 17 17:38:30.509322 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:30.509282    2561 scope.go:117] "RemoveContainer" containerID="b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70"
Jul 17 17:38:31.003544 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:31.003509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerStarted","Data":"2b6ccef3c2a192c76ea1831955547f19e3ec5f89320ad45ea7f9889658948fff"}
Jul 17 17:38:31.508758 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:31.508672    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:38:31.508925 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:31.508836    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:32.010019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.009983    2561 generic.go:356] "Generic (PLEG): container finished" podID="8feff796-4f02-42be-a005-b3702f0779ee" containerID="2b6ccef3c2a192c76ea1831955547f19e3ec5f89320ad45ea7f9889658948fff" exitCode=7
Jul 17 17:38:32.010019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.010022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerDied","Data":"2b6ccef3c2a192c76ea1831955547f19e3ec5f89320ad45ea7f9889658948fff"}
Jul 17 17:38:32.010557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.010122    2561 scope.go:117] "RemoveContainer" containerID="b32bf77f2ab46db67ecf46cfcd9adcec216839c9eabdc7af0c64d65508758e70"
Jul 17 17:38:32.010557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.010418    2561 scope.go:117] "RemoveContainer" containerID="2b6ccef3c2a192c76ea1831955547f19e3ec5f89320ad45ea7f9889658948fff"
Jul 17 17:38:32.010661 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:32.010626    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf_odh-ai-gateway-infra(8feff796-4f02-42be-a005-b3702f0779ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" podUID="8feff796-4f02-42be-a005-b3702f0779ee"
Jul 17 17:38:32.050259 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.050222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:32.915306 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:32.915271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:33.044203 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.044151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:33.139946 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.139923    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:33.247854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.247760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:33.276790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.276755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca\") pod \"8feff796-4f02-42be-a005-b3702f0779ee\" (UID: \"8feff796-4f02-42be-a005-b3702f0779ee\") "
Jul 17 17:38:33.277170 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.277147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8feff796-4f02-42be-a005-b3702f0779ee" (UID: "8feff796-4f02-42be-a005-b3702f0779ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:33.377519 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.377483    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8feff796-4f02-42be-a005-b3702f0779ee-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:33.476748 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.476708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:33.606458 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606420    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"]
Jul 17 17:38:33.606807 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558e9672-4379-4ee2-a343-2d7e0d9673ec" containerName="revoke-keys"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606809    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="558e9672-4379-4ee2-a343-2d7e0d9673ec" containerName="revoke-keys"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606817    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606824    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606835    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" containerName="maas-api"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606840    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" containerName="maas-api"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606849    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.606854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606854    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606865    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606870    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606925    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0fe5e6-70e6-4490-87ac-1ddc49b1eb2e" containerName="maas-api"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606933    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606940    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="558e9672-4379-4ee2-a343-2d7e0d9673ec" containerName="revoke-keys"
Jul 17 17:38:33.607110 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.606949    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:33.610252 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.610236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:33.620456 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.620427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"]
Jul 17 17:38:33.780489 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.780448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc\" (UID: \"6f13ac79-4d5c-4b05-ac30-689c60760890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:33.881937 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.881807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc\" (UID: \"6f13ac79-4d5c-4b05-ac30-689c60760890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:33.882518 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.882498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc\" (UID: \"6f13ac79-4d5c-4b05-ac30-689c60760890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:33.920984 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:33.920954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:34.025811 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.025771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"
Jul 17 17:38:34.025968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.025847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf" event={"ID":"8feff796-4f02-42be-a005-b3702f0779ee","Type":"ContainerDied","Data":"4ae001244df68f7b24287c557e33c56ce492753113548f4d8b6a2432bfdf517b"}
Jul 17 17:38:34.025968 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.025897    2561 scope.go:117] "RemoveContainer" containerID="2b6ccef3c2a192c76ea1831955547f19e3ec5f89320ad45ea7f9889658948fff"
Jul 17 17:38:34.068991 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.068969    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:34.070787 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.070759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"]
Jul 17 17:38:34.070918 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:34.070851    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f13ac79_4d5c_4b05_ac30_689c60760890.slice/crio-8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac WatchSource:0}: Error finding container 8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac: Status 404 returned error can't find the container with id 8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac
Jul 17 17:38:34.076911 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.076885    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d4fd87f-49qlf"]
Jul 17 17:38:34.307349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.307313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:34.514303 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.514206    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8feff796-4f02-42be-a005-b3702f0779ee" path="/var/lib/kubelet/pods/8feff796-4f02-42be-a005-b3702f0779ee/volumes"
Jul 17 17:38:34.519510 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:34.519482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:35.032202 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:35.032168    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f13ac79-4d5c-4b05-ac30-689c60760890" containerID="3826ebe901df7d7d346866dee85ba3184a74c653f1351a41c1aebedd34d56559" exitCode=0
Jul 17 17:38:35.032428 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:35.032260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc" event={"ID":"6f13ac79-4d5c-4b05-ac30-689c60760890","Type":"ContainerDied","Data":"3826ebe901df7d7d346866dee85ba3184a74c653f1351a41c1aebedd34d56559"}
Jul 17 17:38:35.032428 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:35.032325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc" event={"ID":"6f13ac79-4d5c-4b05-ac30-689c60760890","Type":"ContainerStarted","Data":"8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac"}
Jul 17 17:38:36.169589 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.169564    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:36.207040 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.206705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca\") pod \"6f13ac79-4d5c-4b05-ac30-689c60760890\" (UID: \"6f13ac79-4d5c-4b05-ac30-689c60760890\") "
Jul 17 17:38:36.207242 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.207143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f13ac79-4d5c-4b05-ac30-689c60760890" (UID: "6f13ac79-4d5c-4b05-ac30-689c60760890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:36.308438 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.308338    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f13ac79-4d5c-4b05-ac30-689c60760890-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:36.510639 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.510612    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:38:36.510798 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:36.510780    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:38:36.848977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.848941    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:36.849444 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.849426    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f13ac79-4d5c-4b05-ac30-689c60760890" containerName="revoke-keys"
Jul 17 17:38:36.849546 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.849446    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f13ac79-4d5c-4b05-ac30-689c60760890" containerName="revoke-keys"
Jul 17 17:38:36.849616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.849547    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8feff796-4f02-42be-a005-b3702f0779ee" containerName="revoke-keys"
Jul 17 17:38:36.849616 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.849560    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f13ac79-4d5c-4b05-ac30-689c60760890" containerName="revoke-keys"
Jul 17 17:38:36.852713 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.852691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:36.862133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.862102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:36.914449 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:36.914408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4\" (UID: \"2479c58b-aa8d-4fcc-96c4-886a257b59d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:37.015456 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.015418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4\" (UID: \"2479c58b-aa8d-4fcc-96c4-886a257b59d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:37.016076 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.016034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4\" (UID: \"2479c58b-aa8d-4fcc-96c4-886a257b59d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:37.044210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.044180    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"
Jul 17 17:38:37.044210 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.044199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc" event={"ID":"6f13ac79-4d5c-4b05-ac30-689c60760890","Type":"ContainerDied","Data":"8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac"}
Jul 17 17:38:37.044420 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.044253    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7fad388739c0eb0698d9a627b0c68856f9e349f6d9e95c21f668cd9290bfac"
Jul 17 17:38:37.163407 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.163312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:37.298631 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:37.298605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:37.301149 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:38:37.301119    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2479c58b_aa8d_4fcc_96c4_886a257b59d4.slice/crio-126338d6b500247d9b007948b1d34f0dab022f67fb432d8dc2c0336d411c58e2 WatchSource:0}: Error finding container 126338d6b500247d9b007948b1d34f0dab022f67fb432d8dc2c0336d411c58e2: Status 404 returned error can't find the container with id 126338d6b500247d9b007948b1d34f0dab022f67fb432d8dc2c0336d411c58e2
Jul 17 17:38:38.050461 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:38.050423    2561 generic.go:356] "Generic (PLEG): container finished" podID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerID="d39422c24e5cb56865170ec75b3b4552e51d60bdb8b33fb567005cb9af9a6590" exitCode=7
Jul 17 17:38:38.050703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:38.050480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" event={"ID":"2479c58b-aa8d-4fcc-96c4-886a257b59d4","Type":"ContainerDied","Data":"d39422c24e5cb56865170ec75b3b4552e51d60bdb8b33fb567005cb9af9a6590"}
Jul 17 17:38:38.050703 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:38.050532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" event={"ID":"2479c58b-aa8d-4fcc-96c4-886a257b59d4","Type":"ContainerStarted","Data":"126338d6b500247d9b007948b1d34f0dab022f67fb432d8dc2c0336d411c58e2"}
Jul 17 17:38:38.050797 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:38.050737    2561 scope.go:117] "RemoveContainer" containerID="d39422c24e5cb56865170ec75b3b4552e51d60bdb8b33fb567005cb9af9a6590"
Jul 17 17:38:39.057190 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:39.057155    2561 generic.go:356] "Generic (PLEG): container finished" podID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerID="93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63" exitCode=7
Jul 17 17:38:39.057644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:39.057223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" event={"ID":"2479c58b-aa8d-4fcc-96c4-886a257b59d4","Type":"ContainerDied","Data":"93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63"}
Jul 17 17:38:39.057644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:39.057273    2561 scope.go:117] "RemoveContainer" containerID="d39422c24e5cb56865170ec75b3b4552e51d60bdb8b33fb567005cb9af9a6590"
Jul 17 17:38:39.057644 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:39.057519    2561 scope.go:117] "RemoveContainer" containerID="93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63"
Jul 17 17:38:39.057792 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:39.057766    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4_odh-ai-gateway-infra(2479c58b-aa8d-4fcc-96c4-886a257b59d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4"
Jul 17 17:38:40.063493 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:40.063465    2561 scope.go:117] "RemoveContainer" containerID="93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63"
Jul 17 17:38:40.063875 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:40.063648    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4_odh-ai-gateway-infra(2479c58b-aa8d-4fcc-96c4-886a257b59d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4"
Jul 17 17:38:43.509149 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:43.509114    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:38:43.509764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:43.509559    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 17:38:44.085370 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.085331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerStarted","Data":"ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"}
Jul 17 17:38:44.085593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.085559    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:38:44.187143 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.187104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:38:44.187376 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.187351    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" podUID="09655ec8-cbb7-4c6c-87ab-72529b240e02" containerName="maas-api" containerID="cri-o://ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d" gracePeriod=30
Jul 17 17:38:44.227680 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.227594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:44.454633 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.454605    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:38:44.492496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.492400    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") pod \"09655ec8-cbb7-4c6c-87ab-72529b240e02\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") "
Jul 17 17:38:44.492496 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.492489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scm8q\" (UniqueName: \"kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q\") pod \"09655ec8-cbb7-4c6c-87ab-72529b240e02\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") "
Jul 17 17:38:44.492724 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.492520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca\") pod \"09655ec8-cbb7-4c6c-87ab-72529b240e02\" (UID: \"09655ec8-cbb7-4c6c-87ab-72529b240e02\") "
Jul 17 17:38:44.492962 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.492919    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09655ec8-cbb7-4c6c-87ab-72529b240e02" (UID: "09655ec8-cbb7-4c6c-87ab-72529b240e02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:44.494666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.494645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09655ec8-cbb7-4c6c-87ab-72529b240e02" (UID: "09655ec8-cbb7-4c6c-87ab-72529b240e02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 17:38:44.494666 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.494647    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q" (OuterVolumeSpecName: "kube-api-access-scm8q") pod "09655ec8-cbb7-4c6c-87ab-72529b240e02" (UID: "09655ec8-cbb7-4c6c-87ab-72529b240e02"). InnerVolumeSpecName "kube-api-access-scm8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 17:38:44.593816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.593771    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09655ec8-cbb7-4c6c-87ab-72529b240e02-maas-api-tls\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:44.593816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.593809    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scm8q\" (UniqueName: \"kubernetes.io/projected/09655ec8-cbb7-4c6c-87ab-72529b240e02-kube-api-access-scm8q\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:44.593816 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:44.593822    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09655ec8-cbb7-4c6c-87ab-72529b240e02-openshift-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:45.092323 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.092289    2561 generic.go:356] "Generic (PLEG): container finished" podID="09655ec8-cbb7-4c6c-87ab-72529b240e02" containerID="ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d" exitCode=0
Jul 17 17:38:45.092547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.092342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"
Jul 17 17:38:45.092547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.092367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" event={"ID":"09655ec8-cbb7-4c6c-87ab-72529b240e02","Type":"ContainerDied","Data":"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"}
Jul 17 17:38:45.092547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.092421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr" event={"ID":"09655ec8-cbb7-4c6c-87ab-72529b240e02","Type":"ContainerDied","Data":"9522ea39a9cbde59cb1599c9ab6e9601ecefd21b044321c90833404ebe725e4f"}
Jul 17 17:38:45.092547 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.092449    2561 scope.go:117] "RemoveContainer" containerID="ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"
Jul 17 17:38:45.094545 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.094508    2561 generic.go:356] "Generic (PLEG): container finished" podID="7249256d-4460-47ac-bb2e-cf16bf5030bb" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991" exitCode=1
Jul 17 17:38:45.094698 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.094622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" event={"ID":"7249256d-4460-47ac-bb2e-cf16bf5030bb","Type":"ContainerDied","Data":"ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"}
Jul 17 17:38:45.094944 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.094929    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:38:45.095186 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:45.095167    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:45.101344 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.101324    2561 scope.go:117] "RemoveContainer" containerID="ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"
Jul 17 17:38:45.101599 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:45.101581    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d\": container with ID starting with ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d not found: ID does not exist" containerID="ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"
Jul 17 17:38:45.101663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.101606    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d"} err="failed to get container status \"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d\": rpc error: code = NotFound desc = could not find container \"ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d\": container with ID starting with ca22f676a6f38e6080360f2e4e168c76c03aee1188c7f3ff268c0f768c42a82d not found: ID does not exist"
Jul 17 17:38:45.101663 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.101622    2561 scope.go:117] "RemoveContainer" containerID="7f1f7429f942a1fb3385ba001eeb40db6e0b8f7a2742127398863db20400582e"
Jul 17 17:38:45.112642 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.112608    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:38:45.115004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:45.114984    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f5e7b-7cd796f49b-wqwkr"]
Jul 17 17:38:46.102033 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:46.101995    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:38:46.102447 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:46.102197    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:46.517322 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:46.517228    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09655ec8-cbb7-4c6c-87ab-72529b240e02" path="/var/lib/kubelet/pods/09655ec8-cbb7-4c6c-87ab-72529b240e02/volumes"
Jul 17 17:38:47.508360 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:47.508313    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s"
Jul 17 17:38:47.508877 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:47.508726    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:38:47.508953 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:47.508897    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:38:47.566587 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:47.566555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wx57w"]
Jul 17 17:38:48.508854 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:48.508818    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:38:48.509350 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:48.509085    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:38:48.722351 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:48.722315    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"]
Jul 17 17:38:48.729022 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:48.728980    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:48.733392 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:48.733366    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f5e7b-qskdc"]
Jul 17 17:38:49.124086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:49.124031    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:50.514497 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:50.514460    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f13ac79-4d5c-4b05-ac30-689c60760890" path="/var/lib/kubelet/pods/6f13ac79-4d5c-4b05-ac30-689c60760890/volumes"
Jul 17 17:38:53.508958 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:53.508930    2561 scope.go:117] "RemoveContainer" containerID="93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63"
Jul 17 17:38:54.142924 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:54.142891    2561 generic.go:356] "Generic (PLEG): container finished" podID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerID="d910a98dfd429d1f8986fb554b2acfcf883a5596043ff38093a620f1f73c3509" exitCode=7
Jul 17 17:38:54.143086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:54.142959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" event={"ID":"2479c58b-aa8d-4fcc-96c4-886a257b59d4","Type":"ContainerDied","Data":"d910a98dfd429d1f8986fb554b2acfcf883a5596043ff38093a620f1f73c3509"}
Jul 17 17:38:54.143138 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:54.143077    2561 scope.go:117] "RemoveContainer" containerID="93667f0b498114fa4d0e06c1e999b353c8b8e8983109d9fb1f86daa1b3ef6e63"
Jul 17 17:38:54.143387 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:54.143369    2561 scope.go:117] "RemoveContainer" containerID="d910a98dfd429d1f8986fb554b2acfcf883a5596043ff38093a620f1f73c3509"
Jul 17 17:38:54.143614 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:54.143593    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4_odh-ai-gateway-infra(2479c58b-aa8d-4fcc-96c4-886a257b59d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4"
Jul 17 17:38:55.186349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:55.186306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:55.328077 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:55.328030    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:55.397566 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:55.397532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca\") pod \"2479c58b-aa8d-4fcc-96c4-886a257b59d4\" (UID: \"2479c58b-aa8d-4fcc-96c4-886a257b59d4\") "
Jul 17 17:38:55.397903 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:55.397881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2479c58b-aa8d-4fcc-96c4-886a257b59d4" (UID: "2479c58b-aa8d-4fcc-96c4-886a257b59d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 17:38:55.498891 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:55.498798    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2479c58b-aa8d-4fcc-96c4-886a257b59d4-maas-api-service-ca\") on node \"ip-10-0-137-172.ec2.internal\" DevicePath \"\""
Jul 17 17:38:56.156394 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.156364    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"
Jul 17 17:38:56.156611 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.156363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4" event={"ID":"2479c58b-aa8d-4fcc-96c4-886a257b59d4","Type":"ContainerDied","Data":"126338d6b500247d9b007948b1d34f0dab022f67fb432d8dc2c0336d411c58e2"}
Jul 17 17:38:56.156611 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.156515    2561 scope.go:117] "RemoveContainer" containerID="d910a98dfd429d1f8986fb554b2acfcf883a5596043ff38093a620f1f73c3509"
Jul 17 17:38:56.184183 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.184148    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:56.188921 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.188894    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a8e6750e-6cbp4"]
Jul 17 17:38:56.406277 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.406246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:56.514568 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.514485    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" path="/var/lib/kubelet/pods/2479c58b-aa8d-4fcc-96c4-886a257b59d4/volumes"
Jul 17 17:38:56.523243 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.523217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:56.726579 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.726540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:56.932332 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:56.932299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:38:58.508289 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:38:58.508249    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:38:58.508785 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:38:58.508442    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:39:02.508232 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:02.508196    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:39:02.508697 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:02.508459    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:39:04.188225 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:04.188195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc76599b9-bn7q6_9d57b6e2-07ea-4247-86e1-76ec712135cf/copy-manifests/0.log"
Jul 17 17:39:04.199117 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:04.199089    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc76599b9-bn7q6_9d57b6e2-07ea-4247-86e1-76ec712135cf/manager/0.log"
Jul 17 17:39:04.870129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:04.870096    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_7249256d-4460-47ac-bb2e-cf16bf5030bb/maas-api/11.log"
Jul 17 17:39:05.105116 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:05.105071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_9eed73c2-e0b6-436d-9572-ecbcf429039a/maas-api/9.log"
Jul 17 17:39:05.346010 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:05.345983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738460-gk8dq_85d9799e-1c73-449b-9f7b-ef03dac9952f/cleanup/0.log"
Jul 17 17:39:05.464985 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:05.464953    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738475-w8wz7_c9b42c42-f817-4067-98b8-0a05402b316a/cleanup/0.log"
Jul 17 17:39:05.582809 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:05.582781    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738490-schh5_824dd8ea-4cf6-48b8-9d76-793ee0661ffe/cleanup/0.log"
Jul 17 17:39:07.750179 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:07.750140    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9kljd_282af9ea-3859-4b50-be9b-2f2e0ad62957/kuadrant-console-plugin/0.log"
Jul 17 17:39:08.222319 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:08.222286    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wx57w_3fcda4e9-23b3-4e3d-87e3-668211edcb11/limitador/0.log"
Jul 17 17:39:09.043276 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:09.043239    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8db87d7-d8vcb_7def5c34-5bbc-4f0f-970d-6e503f4cfceb/kube-auth-proxy/0.log"
Jul 17 17:39:10.572221 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:10.572182    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-q2svx_a84c6f71-029c-40e5-a60d-f5b6bf978e73/main/0.log"
Jul 17 17:39:10.693280 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:10.693235    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xn2jl_ecc54fa1-e216-40b7-a715-1e14d3ebb39c/main/0.log"
Jul 17 17:39:10.990686 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:10.990590    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:39:11.052342 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:11.052299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:39:13.508512 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:13.508471    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:39:13.509013 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:13.508710    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:39:14.508955 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:14.508908    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:39:14.509430 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:14.509125    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:39:17.085152 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085117    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h2l68/must-gather-24gx8"]
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085463    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085474    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085485    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085500    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09655ec8-cbb7-4c6c-87ab-72529b240e02" containerName="maas-api"
Jul 17 17:39:17.085557 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085505    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09655ec8-cbb7-4c6c-87ab-72529b240e02" containerName="maas-api"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085571    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085581    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09655ec8-cbb7-4c6c-87ab-72529b240e02" containerName="maas-api"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085589    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085647    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.085767 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.085699    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2479c58b-aa8d-4fcc-96c4-886a257b59d4" containerName="revoke-keys"
Jul 17 17:39:17.090132 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.090112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.103867 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.103840    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h2l68\"/\"kube-root-ca.crt\""
Jul 17 17:39:17.104133 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.104115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h2l68\"/\"openshift-service-ca.crt\""
Jul 17 17:39:17.104364 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.104348    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h2l68\"/\"default-dockercfg-nhlfh\""
Jul 17 17:39:17.106262 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.106235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2l68/must-gather-24gx8"]
Jul 17 17:39:17.193647 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.193603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63d19608-004a-4b9f-b59d-3c74fd77ed6f-must-gather-output\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.193855 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.193673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmk9c\" (UniqueName: \"kubernetes.io/projected/63d19608-004a-4b9f-b59d-3c74fd77ed6f-kube-api-access-xmk9c\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.239338 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.239301    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:39:17.295112 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.295074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmk9c\" (UniqueName: \"kubernetes.io/projected/63d19608-004a-4b9f-b59d-3c74fd77ed6f-kube-api-access-xmk9c\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.295320 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.295168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63d19608-004a-4b9f-b59d-3c74fd77ed6f-must-gather-output\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.295475 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.295459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63d19608-004a-4b9f-b59d-3c74fd77ed6f-must-gather-output\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.322632 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.322586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmk9c\" (UniqueName: \"kubernetes.io/projected/63d19608-004a-4b9f-b59d-3c74fd77ed6f-kube-api-access-xmk9c\") pod \"must-gather-24gx8\" (UID: \"63d19608-004a-4b9f-b59d-3c74fd77ed6f\") " pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.398826 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.398725    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2l68/must-gather-24gx8"
Jul 17 17:39:17.451086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.451025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log" limit=1024
Jul 17 17:39:17.548295 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:39:17.548257    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d19608_004a_4b9f_b59d_3c74fd77ed6f.slice/crio-7a8e30c2e9245e7163e7158a62c22fbd60a7239ffe4c77a2d7ab04e1b9d7ffdf WatchSource:0}: Error finding container 7a8e30c2e9245e7163e7158a62c22fbd60a7239ffe4c77a2d7ab04e1b9d7ffdf: Status 404 returned error can't find the container with id 7a8e30c2e9245e7163e7158a62c22fbd60a7239ffe4c77a2d7ab04e1b9d7ffdf
Jul 17 17:39:17.555766 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:17.555738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2l68/must-gather-24gx8"]
Jul 17 17:39:18.267820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:18.267772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2l68/must-gather-24gx8" event={"ID":"63d19608-004a-4b9f-b59d-3c74fd77ed6f","Type":"ContainerStarted","Data":"7a8e30c2e9245e7163e7158a62c22fbd60a7239ffe4c77a2d7ab04e1b9d7ffdf"}
Jul 17 17:39:19.274897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:19.274853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2l68/must-gather-24gx8" event={"ID":"63d19608-004a-4b9f-b59d-3c74fd77ed6f","Type":"ContainerStarted","Data":"da5b86b36a847ff6117c6b5a7aedd77419c90b58a4746473871f4eab4082f482"}
Jul 17 17:39:19.274897 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:19.274905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2l68/must-gather-24gx8" event={"ID":"63d19608-004a-4b9f-b59d-3c74fd77ed6f","Type":"ContainerStarted","Data":"00c7e06d079b53e842bb401764f36ed5462f19a800e6969987795528b1c5a81a"}
Jul 17 17:39:20.690670 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:20.690638    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mtrtl_83fb69b8-a939-4d03-a0f8-995040d1230b/global-pull-secret-syncer/0.log"
Jul 17 17:39:20.765765 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:20.765695    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d48vc_8fbaaadc-1b66-4386-859a-b4320f0352fc/konnectivity-agent/0.log"
Jul 17 17:39:20.973349 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:20.973316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-172.ec2.internal_bbb9b55189c2ca1c136d2bbd25cb800a/haproxy/0.log"
Jul 17 17:39:26.453764 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:26.453714    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9kljd_282af9ea-3859-4b50-be9b-2f2e0ad62957/kuadrant-console-plugin/0.log"
Jul 17 17:39:26.649593 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:26.649519    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wx57w_3fcda4e9-23b3-4e3d-87e3-668211edcb11/limitador/0.log"
Jul 17 17:39:28.509408 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.509376    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:39:28.509880 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:28.509811    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"
Jul 17 17:39:28.510699 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.509946    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:39:28.510699 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:28.510165    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:39:28.794148 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.794110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-s967w_033ef5f1-3ba2-4148-adfc-4ff2175f51ad/kube-state-metrics/0.log"
Jul 17 17:39:28.845429 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.845392    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-s967w_033ef5f1-3ba2-4148-adfc-4ff2175f51ad/kube-rbac-proxy-main/0.log"
Jul 17 17:39:28.896238 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.896205    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-s967w_033ef5f1-3ba2-4148-adfc-4ff2175f51ad/kube-rbac-proxy-self/0.log"
Jul 17 17:39:28.973675 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:28.973643    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54fb5d875-wssh8_d385da69-b81b-49db-b821-137775c5122e/metrics-server/0.log"
Jul 17 17:39:29.223684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.223652    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7xm2_f06ab5f3-e61e-415e-8ec9-329d90f88197/node-exporter/0.log"
Jul 17 17:39:29.280820 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.280793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7xm2_f06ab5f3-e61e-415e-8ec9-329d90f88197/kube-rbac-proxy/0.log"
Jul 17 17:39:29.348965 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.348930    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7xm2_f06ab5f3-e61e-415e-8ec9-329d90f88197/init-textfile/0.log"
Jul 17 17:39:29.488824 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.488720    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8tc7p_77f6cef1-a1ce-4dec-8e35-96bedd751bde/kube-rbac-proxy-main/0.log"
Jul 17 17:39:29.541086 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.541015    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8tc7p_77f6cef1-a1ce-4dec-8e35-96bedd751bde/kube-rbac-proxy-self/0.log"
Jul 17 17:39:29.570684 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.570656    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-8tc7p_77f6cef1-a1ce-4dec-8e35-96bedd751bde/openshift-state-metrics/0.log"
Jul 17 17:39:29.621406 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.621375    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/prometheus/0.log"
Jul 17 17:39:29.656129 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.656033    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/config-reloader/0.log"
Jul 17 17:39:29.686019 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.685983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/thanos-sidecar/0.log"
Jul 17 17:39:29.714821 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.714795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/kube-rbac-proxy-web/0.log"
Jul 17 17:39:29.742075 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.741969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/kube-rbac-proxy/0.log"
Jul 17 17:39:29.775679 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.775635    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/kube-rbac-proxy-thanos/0.log"
Jul 17 17:39:29.821542 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.821512    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74245a6a-bb1a-42f7-b94a-aaa2ba733b77/init-config-reloader/0.log"
Jul 17 17:39:29.882456 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.882419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-mgt8h_28f361ff-5641-43dd-9353-028c51f1de7a/prometheus-operator/0.log"
Jul 17 17:39:29.921325 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.921290    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-mgt8h_28f361ff-5641-43dd-9353-028c51f1de7a/kube-rbac-proxy/0.log"
Jul 17 17:39:29.965772 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:29.965742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-thmth_1ac7fdd0-767a-45b9-a6b0-4a165c337dd0/prometheus-operator-admission-webhook/0.log"
Jul 17 17:39:30.193199 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.193135    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h2l68/must-gather-24gx8" podStartSLOduration=12.358002094 podStartE2EDuration="13.193112934s" totalImagesPullingTime="835.11084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:39:19.312836363 +0000 UTC m=+4393.297923900" watchObservedRunningTime="2026-07-17 17:39:30.193112934 +0000 UTC m=+4404.178200473"
Jul 17 17:39:30.194004 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.193976    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"]
Jul 17 17:39:30.200240 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.200204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.224838 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.224803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"]
Jul 17 17:39:30.335275 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.335227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-podres\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.335459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.335331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-lib-modules\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.335459 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.335389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zppn\" (UniqueName: \"kubernetes.io/projected/242f11e7-7590-4196-9250-1a182293b331-kube-api-access-5zppn\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.335555 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.335459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-sys\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.335555 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.335514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-proc\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.436706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zppn\" (UniqueName: \"kubernetes.io/projected/242f11e7-7590-4196-9250-1a182293b331-kube-api-access-5zppn\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.436795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-sys\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.436848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-proc\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.436895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-podres\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.436961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-lib-modules\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.437158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-lib-modules\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.437529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-sys\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.437581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-proc\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.437696 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.437667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/242f11e7-7590-4196-9250-1a182293b331-podres\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.447039 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.446955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zppn\" (UniqueName: \"kubernetes.io/projected/242f11e7-7590-4196-9250-1a182293b331-kube-api-access-5zppn\") pod \"perf-node-gather-daemonset-4bbnz\" (UID: \"242f11e7-7590-4196-9250-1a182293b331\") " pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.513608 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.513574    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:30.724844 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:30.724784    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"]
Jul 17 17:39:30.727870 ip-10-0-137-172 kubenswrapper[2561]: W0717 17:39:30.727835    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod242f11e7_7590_4196_9250_1a182293b331.slice/crio-b65c070e275656de692d1f961efc1ec49b9e5ed07994836e86c00e1f919160b6 WatchSource:0}: Error finding container b65c070e275656de692d1f961efc1ec49b9e5ed07994836e86c00e1f919160b6: Status 404 returned error can't find the container with id b65c070e275656de692d1f961efc1ec49b9e5ed07994836e86c00e1f919160b6
Jul 17 17:39:31.359403 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:31.359369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz" event={"ID":"242f11e7-7590-4196-9250-1a182293b331","Type":"ContainerStarted","Data":"a0dd63fd01253b09bd652261a4557e8aa7e8c0eed3d35ddd12749941c2ea3adf"}
Jul 17 17:39:31.359618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:31.359420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz" event={"ID":"242f11e7-7590-4196-9250-1a182293b331","Type":"ContainerStarted","Data":"b65c070e275656de692d1f961efc1ec49b9e5ed07994836e86c00e1f919160b6"}
Jul 17 17:39:31.359618 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:31.359513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:31.388028 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:31.387958    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz" podStartSLOduration=1.387940136 podStartE2EDuration="1.387940136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:39:31.385614959 +0000 UTC m=+4405.370702496" watchObservedRunningTime="2026-07-17 17:39:31.387940136 +0000 UTC m=+4405.373027676"
Jul 17 17:39:32.686328 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:32.686288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74dff787bd-8lqsz_7d572b0b-9c68-482e-ab4b-fba912fc463d/console/0.log"
Jul 17 17:39:32.741311 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:32.741275    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-nrbnv_2f948260-9c95-4077-8ed2-3a9e53529335/download-server/0.log"
Jul 17 17:39:34.476121 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:34.476093    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flgqs_9bfff028-810a-45b5-bf27-cf1f65f7e108/dns/0.log"
Jul 17 17:39:34.546143 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:34.546110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flgqs_9bfff028-810a-45b5-bf27-cf1f65f7e108/kube-rbac-proxy/0.log"
Jul 17 17:39:34.790565 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:34.790537    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxzbs_bdbebce2-3e59-4dd6-a5b1-f53e85c4a2d6/dns-node-resolver/0.log"
Jul 17 17:39:35.375907 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:35.375842    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d54d7c4d9-rxc4v_537e195f-3c1f-45a5-8349-79e09e38b6d6/registry/0.log"
Jul 17 17:39:35.407790 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:35.407760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c2jl7_4b5f27d7-42b3-49c3-b3b0-069f1f2ec263/node-ca/0.log"
Jul 17 17:39:36.709750 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:36.709719    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8db87d7-d8vcb_7def5c34-5bbc-4f0f-970d-6e503f4cfceb/kube-auth-proxy/0.log"
Jul 17 17:39:37.374590 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:37.374552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h2l68/perf-node-gather-daemonset-4bbnz"
Jul 17 17:39:37.648332 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:37.648246    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ldwx6_c4615327-7d8b-4645-9bc2-dea866d9673d/serve-healthcheck-canary/0.log"
Jul 17 17:39:38.217759 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:38.217711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85m6q_8afdf257-121e-4a42-9ea8-73fd40d0e70c/kube-rbac-proxy/0.log"
Jul 17 17:39:38.244113 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:38.244082    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85m6q_8afdf257-121e-4a42-9ea8-73fd40d0e70c/exporter/0.log"
Jul 17 17:39:38.269377 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:38.269346    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-85m6q_8afdf257-121e-4a42-9ea8-73fd40d0e70c/extractor/0.log"
Jul 17 17:39:41.282841 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:41.282813    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8db87d7-d8vcb_7def5c34-5bbc-4f0f-970d-6e503f4cfceb/kube-auth-proxy/0.log"
Jul 17 17:39:41.475362 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:41.475335    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc76599b9-bn7q6_9d57b6e2-07ea-4247-86e1-76ec712135cf/manager/0.log"
Jul 17 17:39:41.496468 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:41.496442    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc76599b9-bn7q6_9d57b6e2-07ea-4247-86e1-76ec712135cf/copy-manifests/0.log"
Jul 17 17:39:42.508977 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:42.508874    2561 scope.go:117] "RemoveContainer" containerID="28f86ebeaa5d730f8d31b5300d63510cd52c785859cd97d244a31beddfb16817"
Jul 17 17:39:42.509490 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:42.509163    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn_odh-ai-gateway-infra(9eed73c2-e0b6-436d-9572-ecbcf429039a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d4fd87f-7774665df7-k6qxn" podUID="9eed73c2-e0b6-436d-9572-ecbcf429039a"
Jul 17 17:39:42.755487 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:42.755458    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8db87d7-d8vcb_7def5c34-5bbc-4f0f-970d-6e503f4cfceb/kube-auth-proxy/0.log"
Jul 17 17:39:42.861879 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:42.861834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64bdb655c8-t8pzh_23a5159d-08c1-4d41-aa5f-1a49207866cb/manager/0.log"
Jul 17 17:39:43.508410 ip-10-0-137-172 kubenswrapper[2561]: I0717 17:39:43.508380    2561 scope.go:117] "RemoveContainer" containerID="ee4b27332e7bff0841aa3e20986a9c66024318584c8fc5caa62dc1a41e868991"
Jul 17 17:39:43.508888 ip-10-0-137-172 kubenswrapper[2561]: E0717 17:39:43.508853    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s_odh-ai-gateway-infra(7249256d-4460-47ac-bb2e-cf16bf5030bb)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4cb0f3bd-57d7548b6c-sts9s" podUID="7249256d-4460-47ac-bb2e-cf16bf5030bb"