Jul 17 07:09:59.036166 ip-10-0-138-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 17 07:09:59.036182 ip-10-0-138-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 17 07:09:59.036192 ip-10-0-138-131 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 17 07:09:59.036527 ip-10-0-138-131 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 17 07:10:09.182330 ip-10-0-138-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 17 07:10:09.182346 ip-10-0-138-131 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3c698020af54c6d9c2a0427304bdfff --
Jul 17 07:11:43.613634 ip-10-0-138-131 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.073285    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078941    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078953    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078958    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078962    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078965    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078968    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078971    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.086992 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078975    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078979    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078983    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078986    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078989    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078991    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.078998    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079001    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079004    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079007    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079010    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079012    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079015    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079017    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079020    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079023    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079025    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079028    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079030    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.101081 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079033    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079036    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079038    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079041    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079044    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079048    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079052    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079055    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079059    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079063    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079066    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079069    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079072    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079074    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079077    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079080    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079082    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079085    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079088    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.117309 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079091    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079094    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079097    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079099    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079102    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079105    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079107    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079109    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079112    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079115    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079117    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079120    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079122    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079126    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079128    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079131    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079156    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079160    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079163    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079166    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.120559 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079169    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079172    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079177    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079180    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079183    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079185    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079188    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079190    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079193    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079195    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079200    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079202    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079205    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079207    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079216    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079218    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079221    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079225    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079229    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079232    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.134013 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079235    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079238    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079240    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079243    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079245    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079248    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079251    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079254    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079257    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079260    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079263    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079265    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079268    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.079270    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080012    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080019    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080024    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080027    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080030    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.389981 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080033    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.156195 ip-10-0-138-131 systemd[1]: Started Kubernetes Kubelet.
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080035    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080038    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080041    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080044    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080047    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080050    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080052    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080056    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080059    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080062    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080064    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080067    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080069    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080072    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080075    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080078    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080080    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080083    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080086    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080088    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.409271 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080091    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080094    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080097    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080100    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080102    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080105    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080107    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080110    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080112    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080114    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080117    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080119    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080122    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080124    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080128    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080132    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080135    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080137    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080140    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.420723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080144    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080147    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080150    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080152    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080155    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080157    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080160    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080163    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080166    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080168    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080171    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080173    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080176    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080179    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080181    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080184    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080187    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080189    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080192    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080195    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.421783 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080198    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080201    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080203    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080206    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080208    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080211    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080213    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080216    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080220    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080236    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080241    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080244    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080247    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080250    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080253    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080256    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080259    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080261    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080264    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.422670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080267    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080270    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080273    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080276    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080278    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080281    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080283    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080286    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080289    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080291    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080294    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080296    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080299    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080301    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080304    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080690    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080695    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080698    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080703    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.423744 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080705    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080708    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080711    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080714    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080716    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080719    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080721    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080724    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080727    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080730    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080732    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080737    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080740    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080745    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080748    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080751    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080754    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080757    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080761    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.424635 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080765    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080767    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080770    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080773    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080775    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080778    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080780    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080783    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080786    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080789    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080791    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080794    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080798    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080801    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080804    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080807    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080809    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080812    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080814    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080817    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.425713 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080820    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080822    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080825    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080828    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080831    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080833    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080837    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080840    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080842    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080845    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080848    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080850    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080852    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080855    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080858    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080860    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080863    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080865    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080868    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080870    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.426765 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080873    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080875    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080878    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080880    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080884    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080888    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080891    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080893    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080896    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080898    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080901    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080903    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080906    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080908    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080911    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080913    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080916    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080919    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080922    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080925    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.427847 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080927    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080930    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080932    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080935    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080939    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080942    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080945    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080948    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080951    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080954    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080956    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080959    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080962    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080964    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080967    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.080970    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081038    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081045    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081051    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081056    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 07:11:44.428921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081060    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081064    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081067    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081072    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081075    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081078    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081081    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081084    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081091    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081094    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081099    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081102    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081104    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081107    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081111    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081114    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081117    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081120    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081124    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081127    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081131    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081134    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081137    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081140    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 07:11:44.430013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081143    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081146    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081149    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081151    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081159    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081162    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081165    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081168    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081171    2584 flags.go:64] FLAG: --enable-server="true"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081173    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081178    2584 flags.go:64] FLAG: --event-burst="100"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081181    2584 flags.go:64] FLAG: --event-qps="50"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081184    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081188    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081190    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081194    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081197    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081201    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081205    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081208    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081211    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081214    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081216    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081219    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081222    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 07:11:44.431096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081225    2584 flags.go:64] FLAG: --feature-gates=""
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081228    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081231    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081234    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081237    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081240    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081243    2584 flags.go:64] FLAG: --help="false"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081246    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081249    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081252    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081255    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081259    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081263    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081265    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081269    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081271    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081274    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081278    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081281    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081283    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081286    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081289    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081292    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081295    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 07:11:44.432420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081298    2584 flags.go:64] FLAG: --lock-file=""
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081302    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081305    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081308    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081313    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081316    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081319    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081322    2584 flags.go:64] FLAG: --logging-format="text"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081325    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081328    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081330    2584 flags.go:64] FLAG: --manifest-url=""
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081334    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081350    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081353    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081358    2584 flags.go:64] FLAG: --max-pods="110"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081360    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081364    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081366    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081369    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081374    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081377    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081380    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081386    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081389    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081392    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 07:11:44.433759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081396    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081399    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081405    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081408    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081411    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081414    2584 flags.go:64] FLAG: --port="10250"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081417    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081420    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec95bfd1f3f406d3"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081424    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081428    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081431    2584 flags.go:64] FLAG: --register-node="true"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081434    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081437    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081440    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081444    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081446    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081450    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081453    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081456    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081459    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081461    2584 flags.go:64] FLAG: --runonce="false"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081464    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081468    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081470    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081474    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 07:11:44.434955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081477    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081480    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081484    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081487    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081490    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081492    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081495    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081498    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081501    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081504    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081507    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081512    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081515    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081517    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081522    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081526    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081529    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081532    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081536    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081539    2584 flags.go:64] FLAG: --v="2"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081542    2584 flags.go:64] FLAG: --version="false"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081546    2584 flags.go:64] FLAG: --vmodule=""
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081549    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.081553    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081666    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.436030 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081670    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081673    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081676    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081679    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081682    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081685    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081688    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081691    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081694    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081698    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081701    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081704    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081707    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081710    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081712    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081715    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081717    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081720    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081722    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081725    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.437135 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081727    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081730    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081734    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081736    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081739    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081742    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081745    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081748    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081750    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081753    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081755    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081758    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081760    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081763    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081765    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081767    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081770    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081773    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081775    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081778    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.437998 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081780    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081784    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081787    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081789    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081792    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081794    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081797    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081800    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081803    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081805    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081808    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081810    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081813    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081815    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081819    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081822    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081824    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081827    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081831    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.438885 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081835    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081838    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081842    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081845    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081848    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081851    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081854    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081856    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081859    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081861    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081864    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081867    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081869    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081872    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081876    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081880    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081883    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081886    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081889    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.439749 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081892    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081895    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081897    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081900    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081902    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081905    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081908    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081910    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081914    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081916    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081919    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081922    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081924    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081927    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081930    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081932    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081935    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081938    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081940    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.440652 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.081943    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.082654    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.092535    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.092552    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.093466    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.093473    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093542    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093548    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093551    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093554    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093557    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093561    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093564    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093567    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093570    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.443879 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093573    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093576    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093578    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093581    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093584    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093587    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093589    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093592    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093595    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093598    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093600    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093603    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093608    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093613    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093617    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093621    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093624    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093627    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093629    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.444572 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093632    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093635    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093637    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093640    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093643    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093645    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093648    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093651    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093653    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093656    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093659    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093661    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093664    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093667    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093670    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093673    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093677    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093680    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093683    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093686    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.584723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093688    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093691    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093693    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093696    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093699    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093703    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093706    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093708    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093711    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093713    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093716    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093721    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093724    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093728    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093731    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093734    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093737    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093739    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093742    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.585801 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093745    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093748    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093753    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093756    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093758    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093761    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093764    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093766    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093769    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093771    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093774    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093777    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093780    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093783    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093785    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093788    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093791    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093794    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093798    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093800    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.586683 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093803    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093805    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093808    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093811    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093813    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093816    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093819    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093821    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093824    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093827    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093829    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093832    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.093836    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093955    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093960    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 07:11:44.587548 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093965    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093968    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093971    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093974    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093977    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093980    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093982    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093985    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093987    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093990    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093994    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093996    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.093999    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094001    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094004    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094008    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094010    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094013    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094015    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094018    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 07:11:44.588204 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094021    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094024    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094027    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094030    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094032    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094034    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094037    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094040    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094042    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094045    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094047    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094050    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094054    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094056    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094058    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094061    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094063    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094066    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094069    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094071    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 07:11:44.589084 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094073    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094076    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094079    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094082    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094085    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094087    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094090    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094093    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094097    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094101    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094104    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094106    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094109    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094112    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094115    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094118    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094120    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094124    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094128    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 07:11:44.589947 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094130    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094133    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094136    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094138    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094141    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094145    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094147    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094150    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094153    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094155    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094158    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094160    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094163    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094165    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094168    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094170    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094173    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094176    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094179    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 07:11:44.590822 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094181    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094185    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094188    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094190    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094193    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094196    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094198    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094201    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094204    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094206    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094209    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094211    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094214    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094216    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094219    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094221    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094225    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094229    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 07:11:44.591697 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:44.094233    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.094238    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.094362    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.096820    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.098560    2584 server.go:1019] "Starting client certificate rotation"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.098669    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.098700    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.101452    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.102603    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.116973    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.121632    2584 log.go:25] "Validated CRI v1 image API"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.122477    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 07:11:44.592570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.126112    2584 fs.go:135] Filesystem UUIDs: map[6fc616b1-2a94-4206-b054-81bd9955d3f0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fc1d225d-57f1-45a4-9385-bf7dbc959acf:/dev/nvme0n1p3]
Jul 17 07:11:44.593238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.126126    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 07:11:44.593238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.131278    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.132610    2584 manager.go:216] Machine: {Timestamp:2026-07-17 07:11:44.131075456 +0000 UTC m=+0.384498261 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099808 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2adffae68a048e5829bd51919c7477 SystemUUID:ec2adffa-e68a-048e-5829-bd51919c7477 BootID:b3c69802-0af5-4c6d-9c2a-0427304bdfff Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:59:1c:dd:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:59:1c:dd:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:56:36:a3:cc:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.132709    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.132790    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.136189    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.136212    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.137042    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.137051    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.137059    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.137073    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.139032    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143762    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143890    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143921    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143932    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143944    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143953    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.143963    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 07:11:44.593677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.144378    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.144399    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.146806    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.149440    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.149476    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150952    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150965    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150972    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150979    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150989    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.150995    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151002    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151007    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151013    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151019    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151027    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151032    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151984    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.151991    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.155085    2584 server.go:1268] "Started kubelet"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.155167    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.155158    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.155238    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.156052    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.156428    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gg4ql"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.156703    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.156703    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.157205    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.162241    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gg4ql"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.162516    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.162612    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.164970    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-131.ec2.internal.18c3023507bae043  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-131.ec2.internal,UID:ip-10-0-138-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-131.ec2.internal,},FirstTimestamp:2026-07-17 07:11:44.155058243 +0000 UTC m=+0.408481046,LastTimestamp:2026-07-17 07:11:44.155058243 +0000 UTC m=+0.408481046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-131.ec2.internal,}"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.172403    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.172418    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.172610    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173173    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173189    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173195    2584 factory.go:55] Registering systemd factory
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173210    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173249    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 07:11:44.594552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173362    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173373    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173454    2584 factory.go:153] Registering CRI-O factory
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173470    2584 factory.go:223] Registration of the crio container factory successfully
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173535    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173560    2584 factory.go:103] Registering Raw factory
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.173575    2584 manager.go:1196] Started watching for new ooms in manager
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.174032    2584 manager.go:349] Starting recovery of all containers
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.174775    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.180041    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-131.ec2.internal\" not found" node="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.184094    2584 manager.go:354] Recovery completed
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.188964    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.190996    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191021    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191032    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191510    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191520    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191538    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.191566    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.192295    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.192318    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.192327    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.193685    2584 policy_none.go:49] "None policy: Start"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.193698    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.193708    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.195675    2584 policy_none.go:47] "Start"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.254948    2584 manager.go:337] "Starting Device Plugin manager"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.254986    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.254998    2584 server.go:87] "Starting device plugin registration server"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.255277    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.255300    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.255381    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.255453    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.255462    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.255944    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.255993    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.282856    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.283976    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.284008    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.284028    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.284060    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.287045    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.355444    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.356329    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.356378    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.356394    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.356421    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.363133    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.363151    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-131.ec2.internal\": node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.596301 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.373787    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.384395    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"]
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.384443    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.385842    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.385874    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.385888    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.388176    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.388370    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.388396    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389155    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389178    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389190    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389697    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389725    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.389739    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.391911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.391933    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.392646    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.392673    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.392684    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.462541    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-131.ec2.internal\" not found" node="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.467109    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-131.ec2.internal\" not found" node="ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.473839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.473858    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.473871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.473899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca0c79730331515f06ff8b1a5acdfdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"ca0c79730331515f06ff8b1a5acdfdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.574113    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.598249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.599355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca0c79730331515f06ff8b1a5acdfdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"ca0c79730331515f06ff8b1a5acdfdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.599355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.599355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca0c79730331515f06ff8b1a5acdfdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"ca0c79730331515f06ff8b1a5acdfdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.599355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.574368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08d3b7869dae71d1c57c84be8343179a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"08d3b7869dae71d1c57c84be8343179a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.718596 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.674702    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.849807 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.765905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.849807 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:44.769633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:44.849807 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.779449    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.978377 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.879968    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:44.980763 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:44.980375    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.123248 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.080932    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.123248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.098086    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 07:11:45.123248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.098210    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 07:11:45.123248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.098244    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 07:11:45.282309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.178569    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 07:06:44 +0000 UTC" deadline="2028-04-18 04:35:42.213927288 +0000 UTC"
Jul 17 07:11:45.282309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.178597    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15381h23m57.035333377s"
Jul 17 07:11:45.282309 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.181999    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.283406 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.283112    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.432764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.313973    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:45.432764 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.383537    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.587761 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:45.474290    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d3b7869dae71d1c57c84be8343179a.slice/crio-03600ba1704804f3385584a40a5e3f8c1ad96a2408e6a60066ea1815f9973afd WatchSource:0}: Error finding container 03600ba1704804f3385584a40a5e3f8c1ad96a2408e6a60066ea1815f9973afd: Status 404 returned error can't find the container with id 03600ba1704804f3385584a40a5e3f8c1ad96a2408e6a60066ea1815f9973afd
Jul 17 07:11:45.587761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.483305    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:11:45.587761 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.483601    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.587761 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:45.496304    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0c79730331515f06ff8b1a5acdfdf5.slice/crio-ebc952cebbff36a1bba56cab213faca339da3dc9fba6fdaff1526c2e3deddcf0 WatchSource:0}: Error finding container ebc952cebbff36a1bba56cab213faca339da3dc9fba6fdaff1526c2e3deddcf0: Status 404 returned error can't find the container with id ebc952cebbff36a1bba56cab213faca339da3dc9fba6fdaff1526c2e3deddcf0
Jul 17 07:11:45.587761 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:45.584742    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found"
Jul 17 07:11:45.755948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.624907    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:45.755948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.673337    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:45.755948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.686145    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 07:11:45.755948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.687228    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"
Jul 17 07:11:45.755948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.697007    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 07:11:46.040377 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:45.974979    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:46.040377 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.027026    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 07:11:46.160334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.144575    2584 apiserver.go:52] "Watching apiserver"
Jul 17 07:11:46.160334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.153237    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 07:11:46.160334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.154598    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hktn8","kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls","openshift-multus/multus-additional-cni-plugins-vf2cz","openshift-multus/multus-pg8xv","openshift-multus/network-metrics-daemon-fv7dc","openshift-network-operator/iptables-alerter-cg4p4","openshift-ovn-kubernetes/ovnkube-node-2tp8r","kube-system/konnectivity-agent-8wl8r","openshift-cluster-node-tuning-operator/tuned-cqxz6","openshift-dns/node-resolver-hbn4j","openshift-image-registry/node-ca-pfz9q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal","openshift-network-diagnostics/network-check-target-sv54n"]
Jul 17 07:11:46.160334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.157380    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.161945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.160603    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.163574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.162782    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.167971 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.166428    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.170766 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.168920    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.174236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.171267    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.174236 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.171405    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:46.174236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.173653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.176387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.175916    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:46.176387 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.175980    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.179069    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.179286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.182016    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.184273    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.184362    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.186801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rgs\" (UniqueName: \"kubernetes.io/projected/5f12d13b-a6ab-4340-a4f2-35501cb9566e-kube-api-access-h5rgs\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjqz\" (UniqueName: \"kubernetes.io/projected/f6806af7-799d-477e-ab1a-cda776ac0337-kube-api-access-6mjqz\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cnibin\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188121    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cnibin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188145    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-conf-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-multus-certs\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188206    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-daemon-config\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-agent-certs\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-konnectivity-ca\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkxw\" (UniqueName: \"kubernetes.io/projected/352cee13-e768-429f-ac8a-7e179831a97e-kube-api-access-xgkxw\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188315    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.239024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-multus\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188396    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gws\" (UniqueName: \"kubernetes.io/projected/a31d1f41-4df8-44ef-80dd-e22d85f078f7-kube-api-access-f2gws\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352cee13-e768-429f-ac8a-7e179831a97e-tmp-dir\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188596    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-device-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-sys-fs\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-os-release\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-k8s-cni-cncf-io\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-hostroot\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-system-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjct8\" (UniqueName: \"kubernetes.io/projected/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-kube-api-access-cjct8\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-etc-kubernetes\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxw6s\" (UniqueName: \"kubernetes.io/projected/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-kube-api-access-wxw6s\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.188981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352cee13-e768-429f-ac8a-7e179831a97e-hosts-file\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.243633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cni-binary-copy\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189073    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-bin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-socket-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-socket-dir-parent\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a31d1f41-4df8-44ef-80dd-e22d85f078f7-iptables-alerter-script\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a31d1f41-4df8-44ef-80dd-e22d85f078f7-host-slash\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-registration-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189383    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-os-release\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-netns\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-kubelet\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.189486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.201918    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.202288    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.202527    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.202852    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfp9n\""
Jul 17 07:11:46.244590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.203008    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.203134    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.203262    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.203490    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.203636    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.210635    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.210832    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 07:11:46.247373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.211022    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8tdkk\""
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.273883    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.289795    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-slash\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.289853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-daemon-config\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.289899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-agent-certs\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.289955    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.289980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-systemd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-var-lib-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290075    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aca15ef-883e-4aa7-9711-050622a04421-ovn-node-metrics-cert\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-script-lib\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkxw\" (UniqueName: \"kubernetes.io/projected/352cee13-e768-429f-ac8a-7e179831a97e-kube-api-access-xgkxw\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba41118a-688c-4ba8-93aa-c61b3ddbc615-serviceca\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-run\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gws\" (UniqueName: \"kubernetes.io/projected/a31d1f41-4df8-44ef-80dd-e22d85f078f7-kube-api-access-f2gws\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290312    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352cee13-e768-429f-ac8a-7e179831a97e-tmp-dir\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-device-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.323724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-sys-fs\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290516    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-hostroot\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-etc-kubernetes\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpjh\" (UniqueName: \"kubernetes.io/projected/6aca15ef-883e-4aa7-9711-050622a04421-kube-api-access-fvpjh\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-system-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjct8\" (UniqueName: \"kubernetes.io/projected/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-kube-api-access-cjct8\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxw6s\" (UniqueName: \"kubernetes.io/projected/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-kube-api-access-wxw6s\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-kubelet\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290773    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-conf\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cni-binary-copy\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-bin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-kubelet-config\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-dbus\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysconfig\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.327614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-socket-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-node-log\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291258    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-var-lib-kubelet\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-etc-kubernetes\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks64\" (UniqueName: \"kubernetes.io/projected/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-kube-api-access-wks64\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-daemon-config\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a31d1f41-4df8-44ef-80dd-e22d85f078f7-host-slash\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.290966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352cee13-e768-429f-ac8a-7e179831a97e-tmp-dir\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-bin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-registration-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291631    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-socket-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.291644    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-os-release\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-kubelet\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291745    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-conf-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-hostroot\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.329229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-registration-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291842    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-sys\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a31d1f41-4df8-44ef-80dd-e22d85f078f7-host-slash\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-sys-fs\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.291907    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:46.791882763 +0000 UTC m=+3.045305557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cnibin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-conf-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-multus-certs\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-netns\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-multus-certs\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-systemd\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-os-release\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-host\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.291747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-system-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292131    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-kubelet\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.398016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-multus\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cnibin\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-cni-binary-copy\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292216    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-os-release\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-var-lib-cni-multus\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-k8s-cni-cncf-io\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-os-release\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-etc-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-k8s-cni-cncf-io\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292462    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292467    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-bin\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-device-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-modprobe-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292583    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-kubernetes\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"08d3b7869dae71d1c57c84be8343179a","Type":"ContainerStarted","Data":"03600ba1704804f3385584a40a5e3f8c1ad96a2408e6a60066ea1815f9973afd"}
Jul 17 07:11:46.399887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-lib-modules\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352cee13-e768-429f-ac8a-7e179831a97e-hosts-file\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292801    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/352cee13-e768-429f-ac8a-7e179831a97e-hosts-file\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-systemd-units\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-config\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.292927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-socket-dir-parent\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24gl\" (UniqueName: \"kubernetes.io/projected/ba41118a-688c-4ba8-93aa-c61b3ddbc615-kube-api-access-k24gl\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-ovn\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-log-socket\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-env-overrides\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-tmp\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a31d1f41-4df8-44ef-80dd-e22d85f078f7-iptables-alerter-script\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-cni-dir\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-multus-socket-dir-parent\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.401042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-netns\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-netd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293481    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-tuned\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rgs\" (UniqueName: \"kubernetes.io/projected/5f12d13b-a6ab-4340-a4f2-35501cb9566e-kube-api-access-h5rgs\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjqz\" (UniqueName: \"kubernetes.io/projected/f6806af7-799d-477e-ab1a-cda776ac0337-kube-api-access-6mjqz\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cnibin\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-konnectivity-ca\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba41118a-688c-4ba8-93aa-c61b3ddbc615-host\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6806af7-799d-477e-ab1a-cda776ac0337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-host-run-netns\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.293970    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cnibin\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.294021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.294034    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.297113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f12d13b-a6ab-4340-a4f2-35501cb9566e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.297673    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a31d1f41-4df8-44ef-80dd-e22d85f078f7-iptables-alerter-script\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.403113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.298298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" event={"ID":"ca0c79730331515f06ff8b1a5acdfdf5","Type":"ContainerStarted","Data":"ebc952cebbff36a1bba56cab213faca339da3dc9fba6fdaff1526c2e3deddcf0"}
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.339922    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.340133    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.340209    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-swdcr\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.340135    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.340255    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.340135    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-djd8s\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.343345    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.343808    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lj8vv\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344212    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344251    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344307    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344309    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.344368    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346219    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346220    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346448    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ppjnf\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346453    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rw5kv\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346477    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346517    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346757    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346782    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346855    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b7xqq\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.346784    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4zbc\""
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.347650    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f12d13b-a6ab-4340-a4f2-35501cb9566e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.351410    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.353005    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.353023    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.353036    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:46.404110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.353102    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:46.853083233 +0000 UTC m=+3.106506038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.355530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-konnectivity-ca\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.360092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b94e9c3-49d0-41ef-9e47-4c53d4246c96-agent-certs\") pod \"konnectivity-agent-8wl8r\" (UID: \"1b94e9c3-49d0-41ef-9e47-4c53d4246c96\") " pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.361971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxw6s\" (UniqueName: \"kubernetes.io/projected/ac1a6253-e8fe-4df5-8dc9-2c52026b78c1-kube-api-access-wxw6s\") pod \"multus-pg8xv\" (UID: \"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1\") " pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.363970    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjqz\" (UniqueName: \"kubernetes.io/projected/f6806af7-799d-477e-ab1a-cda776ac0337-kube-api-access-6mjqz\") pod \"aws-ebs-csi-driver-node-kjjls\" (UID: \"f6806af7-799d-477e-ab1a-cda776ac0337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.364483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjct8\" (UniqueName: \"kubernetes.io/projected/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-kube-api-access-cjct8\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.364988    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.367134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rgs\" (UniqueName: \"kubernetes.io/projected/5f12d13b-a6ab-4340-a4f2-35501cb9566e-kube-api-access-h5rgs\") pod \"multus-additional-cni-plugins-vf2cz\" (UID: \"5f12d13b-a6ab-4340-a4f2-35501cb9566e\") " pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.368562    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkxw\" (UniqueName: \"kubernetes.io/projected/352cee13-e768-429f-ac8a-7e179831a97e-kube-api-access-xgkxw\") pod \"node-resolver-hbn4j\" (UID: \"352cee13-e768-429f-ac8a-7e179831a97e\") " pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.369465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gws\" (UniqueName: \"kubernetes.io/projected/a31d1f41-4df8-44ef-80dd-e22d85f078f7-kube-api-access-f2gws\") pod \"iptables-alerter-cg4p4\" (UID: \"a31d1f41-4df8-44ef-80dd-e22d85f078f7\") " pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-ovn\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-log-socket\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-env-overrides\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-tmp\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394439    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-log-socket\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-netd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-ovn\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-tuned\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.406201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394614    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-netd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba41118a-688c-4ba8-93aa-c61b3ddbc615-host\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394655    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba41118a-688c-4ba8-93aa-c61b3ddbc615-host\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-slash\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394745    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-systemd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-slash\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-var-lib-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aca15ef-883e-4aa7-9711-050622a04421-ovn-node-metrics-cert\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-script-lib\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.394837    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba41118a-688c-4ba8-93aa-c61b3ddbc615-serviceca\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.394896    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:46.894875423 +0000 UTC m=+3.148298244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-run\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.394983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpjh\" (UniqueName: \"kubernetes.io/projected/6aca15ef-883e-4aa7-9711-050622a04421-kube-api-access-fvpjh\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395017    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-kubelet\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-conf\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.407620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-env-overrides\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-conf\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba41118a-688c-4ba8-93aa-c61b3ddbc615-serviceca\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-systemd\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-run\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-kubelet\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-var-lib-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395644    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-kubelet-config\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-dbus\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysconfig\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395750    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-node-log\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-var-lib-kubelet\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks64\" (UniqueName: \"kubernetes.io/projected/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-kube-api-access-wks64\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395814    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p5fg2"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-kubelet-config\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-sys\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.409004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-script-lib\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-netns\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-node-log\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-systemd\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-host\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysconfig\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-var-lib-kubelet\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.395881    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ba69a4b-39b3-4d83-83be-47497ce7d246-dbus\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-sys\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-etc-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-bin\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-modprobe-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396180    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-kubernetes\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-run-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396226    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-sysctl-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.411366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-lib-modules\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-run-netns\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396268    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-host\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-systemd-units\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396308    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-systemd\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-etc-openvswitch\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-cni-bin\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396434    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-lib-modules\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-config\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-modprobe-d\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6aca15ef-883e-4aa7-9711-050622a04421-systemd-units\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396506    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-kubernetes\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24gl\" (UniqueName: \"kubernetes.io/projected/ba41118a-688c-4ba8-93aa-c61b3ddbc615-kube-api-access-k24gl\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.396985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6aca15ef-883e-4aa7-9711-050622a04421-ovnkube-config\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.397977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6aca15ef-883e-4aa7-9711-050622a04421-ovn-node-metrics-cert\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.399232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-etc-tuned\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.399508    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-tmp\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.412284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.410121    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p5fg2"
Jul 17 07:11:46.455490 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.455361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpjh\" (UniqueName: \"kubernetes.io/projected/6aca15ef-883e-4aa7-9711-050622a04421-kube-api-access-fvpjh\") pod \"ovnkube-node-2tp8r\" (UID: \"6aca15ef-883e-4aa7-9711-050622a04421\") " pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.456999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.456975    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24gl\" (UniqueName: \"kubernetes.io/projected/ba41118a-688c-4ba8-93aa-c61b3ddbc615-kube-api-access-k24gl\") pod \"node-ca-pfz9q\" (UID: \"ba41118a-688c-4ba8-93aa-c61b3ddbc615\") " pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.458811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.458761    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks64\" (UniqueName: \"kubernetes.io/projected/5a3ac1c5-2e4f-41c9-ac55-d474949470c4-kube-api-access-wks64\") pod \"tuned-cqxz6\" (UID: \"5a3ac1c5-2e4f-41c9-ac55-d474949470c4\") " pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.468857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.468831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbn4j"
Jul 17 07:11:46.478753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.478728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls"
Jul 17 07:11:46.480951 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.480918    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352cee13_e768_429f_ac8a_7e179831a97e.slice/crio-0b2351001305dcdd5fbd28bbbcea9ead8cb87e364bf8b86c31adebbe27bf326e WatchSource:0}: Error finding container 0b2351001305dcdd5fbd28bbbcea9ead8cb87e364bf8b86c31adebbe27bf326e: Status 404 returned error can't find the container with id 0b2351001305dcdd5fbd28bbbcea9ead8cb87e364bf8b86c31adebbe27bf326e
Jul 17 07:11:46.490581 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.490549    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6806af7_799d_477e_ab1a_cda776ac0337.slice/crio-012bed0635d483e2d164b7e69002ee0eda71adbf4601e75d9ae6f1e44027e32c WatchSource:0}: Error finding container 012bed0635d483e2d164b7e69002ee0eda71adbf4601e75d9ae6f1e44027e32c: Status 404 returned error can't find the container with id 012bed0635d483e2d164b7e69002ee0eda71adbf4601e75d9ae6f1e44027e32c
Jul 17 07:11:46.492951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.492931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vf2cz"
Jul 17 07:11:46.504407 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.504026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pg8xv"
Jul 17 07:11:46.513843 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.513727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:11:46.520591 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.520555    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1a6253_e8fe_4df5_8dc9_2c52026b78c1.slice/crio-b782fd7d759a4af543baf89fdf83bcf0a8372f38209041ceab78dcffec1a91ae WatchSource:0}: Error finding container b782fd7d759a4af543baf89fdf83bcf0a8372f38209041ceab78dcffec1a91ae: Status 404 returned error can't find the container with id b782fd7d759a4af543baf89fdf83bcf0a8372f38209041ceab78dcffec1a91ae
Jul 17 07:11:46.524387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.524333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:11:46.528931 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.528909    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b94e9c3_49d0_41ef_9e47_4c53d4246c96.slice/crio-f3582618a65bb9f883fb73890373c3d7b440c18815d157c32a927b64f3e85583 WatchSource:0}: Error finding container f3582618a65bb9f883fb73890373c3d7b440c18815d157c32a927b64f3e85583: Status 404 returned error can't find the container with id f3582618a65bb9f883fb73890373c3d7b440c18815d157c32a927b64f3e85583
Jul 17 07:11:46.534488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.534317    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqxz6"
Jul 17 07:11:46.536469 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.536446    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aca15ef_883e_4aa7_9711_050622a04421.slice/crio-8e9f700ad6718ec1ef5a2a7d8f4bc347ebf5eef98623a0de8377a41b3775e218 WatchSource:0}: Error finding container 8e9f700ad6718ec1ef5a2a7d8f4bc347ebf5eef98623a0de8377a41b3775e218: Status 404 returned error can't find the container with id 8e9f700ad6718ec1ef5a2a7d8f4bc347ebf5eef98623a0de8377a41b3775e218
Jul 17 07:11:46.542373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.542224    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pfz9q"
Jul 17 07:11:46.552297 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.552264    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3ac1c5_2e4f_41c9_ac55_d474949470c4.slice/crio-09929aa27736d381df08099c7b153eca8f26ce81203eb3bf9661af8b3f69bdbc WatchSource:0}: Error finding container 09929aa27736d381df08099c7b153eca8f26ce81203eb3bf9661af8b3f69bdbc: Status 404 returned error can't find the container with id 09929aa27736d381df08099c7b153eca8f26ce81203eb3bf9661af8b3f69bdbc
Jul 17 07:11:46.556103 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.556077    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba41118a_688c_4ba8_93aa_c61b3ddbc615.slice/crio-f45ca3e6727286bb3a8de13a2580c450a065da483843f1c017e25f11dcb9b8c7 WatchSource:0}: Error finding container f45ca3e6727286bb3a8de13a2580c450a065da483843f1c017e25f11dcb9b8c7: Status 404 returned error can't find the container with id f45ca3e6727286bb3a8de13a2580c450a065da483843f1c017e25f11dcb9b8c7
Jul 17 07:11:46.639099 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.639063    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cg4p4"
Jul 17 07:11:46.651146 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:11:46.651115    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31d1f41_4df8_44ef_80dd_e22d85f078f7.slice/crio-6e742059c5f41c42a5fe2b8fc335125c59ac9b1c3ba223a771c97367e79bc4b2 WatchSource:0}: Error finding container 6e742059c5f41c42a5fe2b8fc335125c59ac9b1c3ba223a771c97367e79bc4b2: Status 404 returned error can't find the container with id 6e742059c5f41c42a5fe2b8fc335125c59ac9b1c3ba223a771c97367e79bc4b2
Jul 17 07:11:46.800799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.800699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:46.801036 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.800876    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:46.801036 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.800962    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:47.800940583 +0000 UTC m=+4.054363379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:46.902170 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.902125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:46.902349 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:46.902240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:46.902349 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902292    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:46.902509 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902390    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:47.902372959 +0000 UTC m=+4.155795773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:46.902509 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902408    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:11:46.902509 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902427    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:11:46.902509 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902452    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:46.902710 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:46.902512    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:47.902496325 +0000 UTC m=+4.155919120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:47.284757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.284651    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:47.285464 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.284806    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:47.285464 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.284873    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:47.285464 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.285026    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:47.302321 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.302286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqxz6" event={"ID":"5a3ac1c5-2e4f-41c9-ac55-d474949470c4","Type":"ContainerStarted","Data":"09929aa27736d381df08099c7b153eca8f26ce81203eb3bf9661af8b3f69bdbc"}
Jul 17 07:11:47.303739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.303708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"8e9f700ad6718ec1ef5a2a7d8f4bc347ebf5eef98623a0de8377a41b3775e218"}
Jul 17 07:11:47.305079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.305037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wl8r" event={"ID":"1b94e9c3-49d0-41ef-9e47-4c53d4246c96","Type":"ContainerStarted","Data":"f3582618a65bb9f883fb73890373c3d7b440c18815d157c32a927b64f3e85583"}
Jul 17 07:11:47.306619 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.306487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls" event={"ID":"f6806af7-799d-477e-ab1a-cda776ac0337","Type":"ContainerStarted","Data":"012bed0635d483e2d164b7e69002ee0eda71adbf4601e75d9ae6f1e44027e32c"}
Jul 17 07:11:47.307794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.307751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbn4j" event={"ID":"352cee13-e768-429f-ac8a-7e179831a97e","Type":"ContainerStarted","Data":"0b2351001305dcdd5fbd28bbbcea9ead8cb87e364bf8b86c31adebbe27bf326e"}
Jul 17 07:11:47.309150 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.309092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cg4p4" event={"ID":"a31d1f41-4df8-44ef-80dd-e22d85f078f7","Type":"ContainerStarted","Data":"6e742059c5f41c42a5fe2b8fc335125c59ac9b1c3ba223a771c97367e79bc4b2"}
Jul 17 07:11:47.310443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.310379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pfz9q" event={"ID":"ba41118a-688c-4ba8-93aa-c61b3ddbc615","Type":"ContainerStarted","Data":"f45ca3e6727286bb3a8de13a2580c450a065da483843f1c017e25f11dcb9b8c7"}
Jul 17 07:11:47.311918 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.311875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pg8xv" event={"ID":"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1","Type":"ContainerStarted","Data":"b782fd7d759a4af543baf89fdf83bcf0a8372f38209041ceab78dcffec1a91ae"}
Jul 17 07:11:47.313309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.313281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerStarted","Data":"ff201dc00eb95da36f03ad9f342d9e8d3377673799a7ebf2b3c9ef1ffa11955a"}
Jul 17 07:11:47.415442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.415323    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 07:06:47 +0000 UTC" deadline="2028-04-17 06:00:40.989247486 +0000 UTC"
Jul 17 07:11:47.415442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.415368    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15358h48m53.573885022s"
Jul 17 07:11:47.809948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.809901    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:47.821578 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.819657    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:47.821578 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.819781    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:49.819755416 +0000 UTC m=+6.073178209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:47.911197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.911152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:47.911397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:47.911222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:47.911504 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.911409    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:47.912034 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.911571    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:11:47.912034 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.911591    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:11:47.912034 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.911606    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:47.912034 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.911668    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:49.911647556 +0000 UTC m=+6.165070374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:47.912351 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:47.912174    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:49.912148426 +0000 UTC m=+6.165571237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:48.289342 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:48.289265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:48.289797 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:48.289407    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:48.416442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:48.416349    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 07:06:47 +0000 UTC" deadline="2028-01-23 22:43:27.430730809 +0000 UTC"
Jul 17 07:11:48.416442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:48.416436    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13335h31m39.014299583s"
Jul 17 07:11:49.285942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:49.285199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:49.285942 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.285350    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:49.285942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:49.285790    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:49.285942 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.285886    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:49.827173 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:49.827138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:49.827807 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.827329    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:49.827807 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.827412    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:53.8273909 +0000 UTC m=+10.080813694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:49.928443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:49.928407    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:49.928623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:49.928464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:49.928623 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.928604    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:49.928735 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.928668    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:53.928646788 +0000 UTC m=+10.182069594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:49.929104 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.929081    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:11:49.929208 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.929109    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:11:49.929208 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.929122    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:49.929208 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:49.929171    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:11:53.929155298 +0000 UTC m=+10.182578104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:50.284827 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:50.284741    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:50.284996 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:50.284886    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:51.284886 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:51.284846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:51.285399 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:51.285005    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:51.285399 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:51.285275    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:51.285399 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:51.285387    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:52.284944 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:52.284914    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:52.285378 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:52.285028    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:53.285883 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:53.285265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:53.285883 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:53.285316    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:53.285883 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.285430    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:53.285883 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.285503    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:53.864951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:53.864578    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:53.864951 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.864735    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:53.864951 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.864803    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:01.864782411 +0000 UTC m=+18.118205263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:11:53.965904 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:53.965857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:53.966069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:53.965923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:53.966127 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966065    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:11:53.966127 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966077    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:53.966127 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966092    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:11:53.966127 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966107    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:53.966281 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966149    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:01.966131134 +0000 UTC m=+18.219553933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:11:53.966281 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:53.966168    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:01.966158767 +0000 UTC m=+18.219581565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:11:54.285684 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:54.285646    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:54.285857 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:54.285776    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:55.285221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:55.284473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:55.285221 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:55.284619    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:55.285221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:55.285052    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:55.285221 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:55.285161    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:56.288490 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:56.288054    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:56.288490 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:56.288174    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:57.284642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:57.284605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:57.284642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:57.284643    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:57.284881 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:57.284782    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:11:57.284945 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:57.284882    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:58.288257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:58.288223    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:11:58.288715 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:58.288370    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:11:59.285195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:59.285154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:11:59.285409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:11:59.285208    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:11:59.285409 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:59.285308    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:11:59.285530 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:11:59.285433    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:00.284635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:00.284599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:00.285004 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:00.284712    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:01.284980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:01.284946    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:01.285483 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:01.285075    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:01.285483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:01.285127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:01.285483 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:01.285235    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:01.929350 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:01.929292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:01.929540 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:01.929449    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:12:01.929540 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:01.929519    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:17.929501806 +0000 UTC m=+34.182924596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:12:02.030457 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:02.030415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:02.030627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:02.030472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:02.030627 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.030606    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:12:02.030732 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.030641    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:12:02.033255 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.030710    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:12:02.033255 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.030899    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:12:02.033255 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.031021    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:18.030991156 +0000 UTC m=+34.284413967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:12:02.033255 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.031248    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:18.031226761 +0000 UTC m=+34.284649568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:12:02.284692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:02.284659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:02.284871 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:02.284794    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:03.284594 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:03.284562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:03.284594 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:03.284585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:03.285110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:03.284725    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:03.285110 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:03.284837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:04.285331 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:04.285291    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:04.285789 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:04.285442    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:05.284635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:05.284599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:05.284836 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:05.284599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:05.284836 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:05.284733    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:05.284948 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:05.284848    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:06.284982 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:06.284951    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:06.285463 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:06.285060    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:07.284248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.284219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:07.284248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.284246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:07.284442 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:07.284334    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:07.284484 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:07.284448    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:07.362901 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.362694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbn4j" event={"ID":"352cee13-e768-429f-ac8a-7e179831a97e","Type":"ContainerStarted","Data":"e0ea3c38ce40d78d3e84b386aa613ec1a52a95001eef409eaab8be8cdb1fa849"}
Jul 17 07:12:07.364231 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.364205    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pfz9q" event={"ID":"ba41118a-688c-4ba8-93aa-c61b3ddbc615","Type":"ContainerStarted","Data":"689e82ea5dfe46624c178424bbfef1627eb405f109f77ee6a1b2253af108ca66"}
Jul 17 07:12:07.365555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.365532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pg8xv" event={"ID":"ac1a6253-e8fe-4df5-8dc9-2c52026b78c1","Type":"ContainerStarted","Data":"5a9f6d6fe960d064bc5a6e9b991b67453b2e777c3db17836336ef150af3b0e04"}
Jul 17 07:12:07.366881 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.366860    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="c7bef58762fdafddf01fca27e9045bd99e1a8e3bec4ae04eb7fd0f844b13255c" exitCode=0
Jul 17 07:12:07.366973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.366918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"c7bef58762fdafddf01fca27e9045bd99e1a8e3bec4ae04eb7fd0f844b13255c"}
Jul 17 07:12:07.368354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.368315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" event={"ID":"ca0c79730331515f06ff8b1a5acdfdf5","Type":"ContainerStarted","Data":"c5b30c7f0b4551a7d5987c4089343493b7b8b73247c01c2febbc747179923acc"}
Jul 17 07:12:07.369732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.369709    2584 generic.go:356] "Generic (PLEG): container finished" podID="08d3b7869dae71d1c57c84be8343179a" containerID="7a7ae53388d88228f767f8f87314c121c704a1624b312dcd831a1a4527de1be6" exitCode=0
Jul 17 07:12:07.369797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.369772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"08d3b7869dae71d1c57c84be8343179a","Type":"ContainerDied","Data":"7a7ae53388d88228f767f8f87314c121c704a1624b312dcd831a1a4527de1be6"}
Jul 17 07:12:07.373942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.371622    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqxz6" event={"ID":"5a3ac1c5-2e4f-41c9-ac55-d474949470c4","Type":"ContainerStarted","Data":"16c5ae3cd0598e15e5c31501adf6f0122177b244c59fb7e0e3a0922fa1b1ed82"}
Jul 17 07:12:07.376067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.376040    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"5ccdf1e67c5c7f3e8023cebed29fe40e926664fbd7583e303ac86f68a4ea74b4"}
Jul 17 07:12:07.376163 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.376109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"98e80191321b9f9c8a5f8815d84d849f24524c7b417ac1f622abd960f258a121"}
Jul 17 07:12:07.377532 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.377510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wl8r" event={"ID":"1b94e9c3-49d0-41ef-9e47-4c53d4246c96","Type":"ContainerStarted","Data":"e5861ce7b9e861de0fd3d20ef37f6d862554fe413475db38845bcaafb7744cfe"}
Jul 17 07:12:07.378790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.378768    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls" event={"ID":"f6806af7-799d-477e-ab1a-cda776ac0337","Type":"ContainerStarted","Data":"c222b8e867024e97b9bac50599a253a0dde74dabc2f327990b86302dbe2453b5"}
Jul 17 07:12:07.380241 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.380203    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hbn4j" podStartSLOduration=2.443969218 podStartE2EDuration="22.380192518s" totalImagesPullingTime="19.9362233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.379903274 +0000 UTC m=+23.633326085" watchObservedRunningTime="2026-07-17 07:12:07.380192518 +0000 UTC m=+23.633615333"
Jul 17 07:12:07.397148 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.397104    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pg8xv" podStartSLOduration=2.465293414 podStartE2EDuration="22.397090131s" totalImagesPullingTime="19.931796717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.396925556 +0000 UTC m=+23.650348403" watchObservedRunningTime="2026-07-17 07:12:07.397090131 +0000 UTC m=+23.650512942"
Jul 17 07:12:07.413485 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.413433    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cqxz6" podStartSLOduration=2.544446774 podStartE2EDuration="22.413415741s" totalImagesPullingTime="19.868968967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.412898782 +0000 UTC m=+23.666321608" watchObservedRunningTime="2026-07-17 07:12:07.413415741 +0000 UTC m=+23.666838555"
Jul 17 07:12:07.425622 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.425581    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" podStartSLOduration=21.425565995 podStartE2EDuration="21.425565995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.425118535 +0000 UTC m=+23.678541348" watchObservedRunningTime="2026-07-17 07:12:07.425565995 +0000 UTC m=+23.678988806"
Jul 17 07:12:07.465064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.464961    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pfz9q" podStartSLOduration=2.602529181 podStartE2EDuration="22.464944861s" totalImagesPullingTime="19.86241568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.464405605 +0000 UTC m=+23.717828411" watchObservedRunningTime="2026-07-17 07:12:07.464944861 +0000 UTC m=+23.718367674"
Jul 17 07:12:07.479328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:07.479282    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8wl8r" podStartSLOduration=2.590227401 podStartE2EDuration="22.479266338s" totalImagesPullingTime="19.889038937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:07.479213955 +0000 UTC m=+23.732636767" watchObservedRunningTime="2026-07-17 07:12:07.479266338 +0000 UTC m=+23.732689149"
Jul 17 07:12:08.285506 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.285229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:08.285736 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:08.285613    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:08.383234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383206    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383575    2584 generic.go:356] "Generic (PLEG): container finished" podID="6aca15ef-883e-4aa7-9711-050622a04421" containerID="5ccdf1e67c5c7f3e8023cebed29fe40e926664fbd7583e303ac86f68a4ea74b4" exitCode=1
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerDied","Data":"5ccdf1e67c5c7f3e8023cebed29fe40e926664fbd7583e303ac86f68a4ea74b4"}
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383743    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"72ed8cedb0c813aad59f481cfd6c86fb44959b3c0c66d51d04085906d36d04ac"}
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"605f35c22f8e4e858f592dfd5fe3c682732ca3b4de6cd43eadfc8b76e1323ba3"}
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"ee58c74699278a7fe2ab6070ea9144d752b012818bd31c242f04d707ad51e839"}
Jul 17 07:12:08.384036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.383883    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"0f4f659dcc9dfae0db00cf1a501bb4dc1dd08766c75189c475ee46b3baca96e8"}
Jul 17 07:12:08.385293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.385164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cg4p4" event={"ID":"a31d1f41-4df8-44ef-80dd-e22d85f078f7","Type":"ContainerStarted","Data":"55eb4c27653efc17a247fbb3f956f7affe80eddcc96841614ab8ce62dbb6a9ca"}
Jul 17 07:12:08.401396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.401328    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cg4p4" podStartSLOduration=3.6341590249999998 podStartE2EDuration="23.401313078s" totalImagesPullingTime="19.767154053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:08.401139341 +0000 UTC m=+24.654562156" watchObservedRunningTime="2026-07-17 07:12:08.401313078 +0000 UTC m=+24.654735891"
Jul 17 07:12:08.623287 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:08.623221    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 07:12:09.269461 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.269351    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T07:12:08.623243275Z","UUID":"c7b14b04-ac61-4e3c-ba20-d8f891b15916","Handler":null,"Name":"","Endpoint":""}
Jul 17 07:12:09.271165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.271144    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 07:12:09.271310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.271186    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 07:12:09.285200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.285177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:09.285370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.285179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:09.285370 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:09.285301    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:09.285491 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:09.285402    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:09.389615 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.389575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"08d3b7869dae71d1c57c84be8343179a","Type":"ContainerStarted","Data":"86cc127fdffea9a2deac56f9adce86c04e4d78fc3308254964f9c92ad4e97112"}
Jul 17 07:12:09.392307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.392272    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls" event={"ID":"f6806af7-799d-477e-ab1a-cda776ac0337","Type":"ContainerStarted","Data":"015750d2a5d868b03740daca0a548b3439273e9a80d94af56099b28ba4b98232"}
Jul 17 07:12:09.409049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:09.409001    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" podStartSLOduration=23.408984805 podStartE2EDuration="23.408984805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:09.408373071 +0000 UTC m=+25.661795884" watchObservedRunningTime="2026-07-17 07:12:09.408984805 +0000 UTC m=+25.662407617"
Jul 17 07:12:10.285054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.285019    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:10.285203 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:10.285162    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:10.399761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.399737    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:12:10.400165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.400123    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"3a0889353ef8dd322871d7b26596c91e3bcf7710080aa527871fd9277408239f"}
Jul 17 07:12:10.402060 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.402037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls" event={"ID":"f6806af7-799d-477e-ab1a-cda776ac0337","Type":"ContainerStarted","Data":"a6fda471aa7f3d495c6a7b4caaeadf97e05a5a4e8fac401850079bcf132bfa3f"}
Jul 17 07:12:10.807477 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.807441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:12:10.808097 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.808074    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:12:10.823187 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:10.823145    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjjls" podStartSLOduration=3.25126545 podStartE2EDuration="25.823131157s" totalImagesPullingTime="22.571865707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:10.438096182 +0000 UTC m=+26.691518994" watchObservedRunningTime="2026-07-17 07:12:10.823131157 +0000 UTC m=+27.076553995"
Jul 17 07:12:11.284820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.284786    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:11.284820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.284809    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:11.285043 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:11.284932    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:11.285100 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:11.285063    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:11.405482 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.405407    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:12:11.405975 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.405764    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8wl8r"
Jul 17 07:12:11.504357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.504305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:11.598063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:11.597990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:12.284615 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:12.284586    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:12.284819 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:12.284735    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:13.284402 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.284207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:13.285203 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.284235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:13.285203 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:13.284515    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:13.285203 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:13.284544    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:13.411396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.411360    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="679c1ff60388c86a15ea76a6abf02cdcc3d9b9a37903d597bcba33806b006cc1" exitCode=0
Jul 17 07:12:13.411541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.411449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"679c1ff60388c86a15ea76a6abf02cdcc3d9b9a37903d597bcba33806b006cc1"}
Jul 17 07:12:13.415041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.415017    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:12:13.415424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.415398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"00790efdf359e7ea9b96d9226cdff5748b16ce7c8306e5901d917b8cafb7bb70"}
Jul 17 07:12:13.415822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:13.415807    2584 scope.go:117] "RemoveContainer" containerID="5ccdf1e67c5c7f3e8023cebed29fe40e926664fbd7583e303ac86f68a4ea74b4"
Jul 17 07:12:14.285810 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.285605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:14.286359 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:14.285908    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:14.421332 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.421306    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:12:14.421643 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.421620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" event={"ID":"6aca15ef-883e-4aa7-9711-050622a04421","Type":"ContainerStarted","Data":"108200aef6dda3ce54beb3cf29f0ff821d9ad475215c42e5feaaa6c6f921866d"}
Jul 17 07:12:14.421811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.421799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:14.421855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.421846    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:14.421892 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.421879    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:14.437409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.437323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:14.437545 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.437496    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:14.450727 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.450682    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r" podStartSLOduration=9.038400105 podStartE2EDuration="29.450666941s" totalImagesPullingTime="20.412266836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:14.449251388 +0000 UTC m=+30.702674228" watchObservedRunningTime="2026-07-17 07:12:14.450666941 +0000 UTC m=+30.704089752"
Jul 17 07:12:14.834661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.834557    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sv54n"]
Jul 17 07:12:14.834854 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.834730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:14.834854 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:14.834837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:14.835403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.835378    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fv7dc"]
Jul 17 07:12:14.835498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.835473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:14.835560 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:14.835544    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:14.837486 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.837464    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hktn8"]
Jul 17 07:12:14.837618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.837562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:14.837680 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:14.837653    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:14.872106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.872055    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:14.976898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:14.976861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:15.425888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:15.425853    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="910030d10b88f0708bca533720c4f03f2e7304458109b2362fd431664684b803" exitCode=0
Jul 17 07:12:15.426349 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:15.425944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"910030d10b88f0708bca533720c4f03f2e7304458109b2362fd431664684b803"}
Jul 17 07:12:16.285169 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.285143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:16.285312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.285151    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:16.285312 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:16.285245    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:16.285312 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:16.285303    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:16.285439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.285357    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:16.285439 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:16.285408    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:16.456145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.455627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:16.557002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.556970    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:16.852999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.852966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:16.925111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:16.925082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:17.433281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:17.433249    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="b9d6f8ade2bf6d24a00e86ca502bc37fe36464ff98bf9480d71fab6a575cff2d" exitCode=0
Jul 17 07:12:17.433281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:17.433285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"b9d6f8ade2bf6d24a00e86ca502bc37fe36464ff98bf9480d71fab6a575cff2d"}
Jul 17 07:12:17.954127 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:17.954093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:17.954567 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:17.954251    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:12:17.954567 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:17.954318    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs podName:c11dcfbf-059f-4cac-b79a-4206b3e6e8b1 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:49.954301714 +0000 UTC m=+66.207724507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs") pod "network-metrics-daemon-fv7dc" (UID: "c11dcfbf-059f-4cac-b79a-4206b3e6e8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 07:12:18.054991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.054910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:18.054991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.054958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055061    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055077    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055094    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055106    2584 projected.go:196] Error preparing data for projected volume kube-api-access-h7mmk for pod openshift-network-diagnostics/network-check-target-sv54n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055131    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret podName:8ba69a4b-39b3-4d83-83be-47497ce7d246 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:50.055111491 +0000 UTC m=+66.308534300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret") pod "global-pull-secret-syncer-hktn8" (UID: "8ba69a4b-39b3-4d83-83be-47497ce7d246") : object "kube-system"/"original-pull-secret" not registered
Jul 17 07:12:18.055196 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.055147    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk podName:82d92bb9-71f7-4779-a81a-99f898cd1e98 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:50.055138645 +0000 UTC m=+66.308561435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7mmk" (UniqueName: "kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk") pod "network-check-target-sv54n" (UID: "82d92bb9-71f7-4779-a81a-99f898cd1e98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 07:12:18.284557 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.284378    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:18.284731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.284384    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:18.284731 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.284656    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hktn8" podUID="8ba69a4b-39b3-4d83-83be-47497ce7d246"
Jul 17 07:12:18.284731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.284424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:18.284731 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.284704    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv7dc" podUID="c11dcfbf-059f-4cac-b79a-4206b3e6e8b1"
Jul 17 07:12:18.284935 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:18.284770    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv54n" podUID="82d92bb9-71f7-4779-a81a-99f898cd1e98"
Jul 17 07:12:18.368867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.368769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:18.483987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.483952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:18.575820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.575784    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:18.634820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:18.634743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:19.331890 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.331845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:19.429613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.429574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:19.532850 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.532778    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeReady"
Jul 17 07:12:19.533011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.532902    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 07:12:19.567286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.567257    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:12:19.571007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.570986    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.574254 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.574164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 07:12:19.574254 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.574197    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 07:12:19.574254 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.574202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 07:12:19.574700 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.574530    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lwnzz\""
Jul 17 07:12:19.577743 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.577713    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k8fp2"]
Jul 17 07:12:19.581883 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.581857    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jnzc7"]
Jul 17 07:12:19.583663 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.583064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.584621 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.584452    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 07:12:19.586326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586094    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 07:12:19.586326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586130    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 07:12:19.586326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586128    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z4klw\""
Jul 17 07:12:19.586542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586501    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 07:12:19.586719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586702    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.586875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.586829    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:12:19.589765 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.589742    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8fp2"]
Jul 17 07:12:19.589874 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.589783    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 07:12:19.589874 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.589835    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kjdqb\""
Jul 17 07:12:19.590060 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.590041    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 07:12:19.592822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.592801    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnzc7"]
Jul 17 07:12:19.661548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.661514    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"]
Jul 17 07:12:19.665194 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.665174    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.666591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m557p\" (UniqueName: \"kubernetes.io/projected/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-kube-api-access-m557p\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.666707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.666707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.666808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666706    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.666860 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-cert\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.666860 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.666936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.666871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.667020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.667003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.667079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.667040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.667079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.667068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsv4\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.668168 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.668145    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 07:12:19.668304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.668145    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 07:12:19.668304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.668145    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 07:12:19.668304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.668280    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gnhcl\""
Jul 17 07:12:19.668502 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.668324    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 07:12:19.675435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.675416    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"]
Jul 17 07:12:19.717540 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.717508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:19.767887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.767852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.767887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.767884    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.767918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea191a-a858-477f-8f34-31390a10e8a7-config-volume\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.768120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.767933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea191a-a858-477f-8f34-31390a10e8a7-tmp-dir\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.768120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768017    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsv4\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m557p\" (UniqueName: \"kubernetes.io/projected/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-kube-api-access-m557p\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768161    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmm5m\" (UniqueName: \"kubernetes.io/projected/283e90a3-f3ea-484f-88ce-effe90d837e6-kube-api-access-tmm5m\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768278    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/283e90a3-f3ea-484f-88ce-effe90d837e6-serving-cert\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-cert\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/283e90a3-f3ea-484f-88ce-effe90d837e6-config\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:19.768380    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:19.768396    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5d976689-zstdt: secret "image-registry-tls" not found
Jul 17 07:12:19.768439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kr9k\" (UniqueName: \"kubernetes.io/projected/f8ea191a-a858-477f-8f34-31390a10e8a7-kube-api-access-7kr9k\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.768988 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:19.768463    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls podName:49ec566f-672e-436f-baab-a1bc01315dae nodeName:}" failed. No retries permitted until 2026-07-17 07:12:20.268442477 +0000 UTC m=+36.521865270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls") pod "image-registry-5d5d976689-zstdt" (UID: "49ec566f-672e-436f-baab-a1bc01315dae") : secret "image-registry-tls" not found
Jul 17 07:12:19.768988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.768773    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.769090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.769071    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.772362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.772314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-cert\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.772362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.772353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.772518 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.772358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.777768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.777738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.778309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.778267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsv4\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:19.779313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.779293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m557p\" (UniqueName: \"kubernetes.io/projected/1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4-kube-api-access-m557p\") pod \"ingress-canary-k8fp2\" (UID: \"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4\") " pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.780761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.780698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:19.868892 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.868853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/283e90a3-f3ea-484f-88ce-effe90d837e6-serving-cert\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.868905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/283e90a3-f3ea-484f-88ce-effe90d837e6-config\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.868929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kr9k\" (UniqueName: \"kubernetes.io/projected/f8ea191a-a858-477f-8f34-31390a10e8a7-kube-api-access-7kr9k\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.868969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea191a-a858-477f-8f34-31390a10e8a7-config-volume\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.868985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea191a-a858-477f-8f34-31390a10e8a7-tmp-dir\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.869044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmm5m\" (UniqueName: \"kubernetes.io/projected/283e90a3-f3ea-484f-88ce-effe90d837e6-kube-api-access-tmm5m\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.869082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.869060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.869490 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:19.869134    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 07:12:19.869490 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:19.869195    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls podName:f8ea191a-a858-477f-8f34-31390a10e8a7 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:20.369177148 +0000 UTC m=+36.622599962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls") pod "dns-default-jnzc7" (UID: "f8ea191a-a858-477f-8f34-31390a10e8a7") : secret "dns-default-metrics-tls" not found
Jul 17 07:12:19.869602 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.869511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ea191a-a858-477f-8f34-31390a10e8a7-tmp-dir\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.869602 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.869567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/283e90a3-f3ea-484f-88ce-effe90d837e6-config\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.869988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.869969    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ea191a-a858-477f-8f34-31390a10e8a7-config-volume\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.871529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.871500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/283e90a3-f3ea-484f-88ce-effe90d837e6-serving-cert\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.881544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.881524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmm5m\" (UniqueName: \"kubernetes.io/projected/283e90a3-f3ea-484f-88ce-effe90d837e6-kube-api-access-tmm5m\") pod \"service-ca-operator-5bcbc94c78-75kbd\" (UID: \"283e90a3-f3ea-484f-88ce-effe90d837e6\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:19.884817 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.884795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kr9k\" (UniqueName: \"kubernetes.io/projected/f8ea191a-a858-477f-8f34-31390a10e8a7-kube-api-access-7kr9k\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:19.897839 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.897796    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8fp2"
Jul 17 07:12:19.975874 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:19.975844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"
Jul 17 07:12:20.055166 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.055104    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8fp2"]
Jul 17 07:12:20.059601 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:20.059571    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1daeaaed_fba4_4fbf_91ba_83ff2a7b28e4.slice/crio-5f0344edd97f523cd564b760f0383d13c745d0bc01521175f0605f85842a7e86 WatchSource:0}: Error finding container 5f0344edd97f523cd564b760f0383d13c745d0bc01521175f0605f85842a7e86: Status 404 returned error can't find the container with id 5f0344edd97f523cd564b760f0383d13c745d0bc01521175f0605f85842a7e86
Jul 17 07:12:20.110744 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.110717    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd"]
Jul 17 07:12:20.113654 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:20.113631    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283e90a3_f3ea_484f_88ce_effe90d837e6.slice/crio-c28847e5e197ed651a17271ea252ba9cd9e82d2de1dc6b5b4dcbac69fc69e714 WatchSource:0}: Error finding container c28847e5e197ed651a17271ea252ba9cd9e82d2de1dc6b5b4dcbac69fc69e714: Status 404 returned error can't find the container with id c28847e5e197ed651a17271ea252ba9cd9e82d2de1dc6b5b4dcbac69fc69e714
Jul 17 07:12:20.272214 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.272131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:20.272405 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:20.272253    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 07:12:20.272405 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:20.272273    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5d976689-zstdt: secret "image-registry-tls" not found
Jul 17 07:12:20.272405 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:20.272364    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls podName:49ec566f-672e-436f-baab-a1bc01315dae nodeName:}" failed. No retries permitted until 2026-07-17 07:12:21.272332208 +0000 UTC m=+37.525754997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls") pod "image-registry-5d5d976689-zstdt" (UID: "49ec566f-672e-436f-baab-a1bc01315dae") : secret "image-registry-tls" not found
Jul 17 07:12:20.284787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.284759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:20.284925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.284759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:20.284925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.284890    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:20.287504 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287480    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jgz\""
Jul 17 07:12:20.287504 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287516    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 07:12:20.287716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287480    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 07:12:20.287716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287541    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 07:12:20.287716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287541    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c28tv\""
Jul 17 07:12:20.287716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.287590    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 07:12:20.373249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.373217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:20.373900 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:20.373380    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 07:12:20.373900 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:20.373457    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls podName:f8ea191a-a858-477f-8f34-31390a10e8a7 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:21.373437495 +0000 UTC m=+37.626860286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls") pod "dns-default-jnzc7" (UID: "f8ea191a-a858-477f-8f34-31390a10e8a7") : secret "dns-default-metrics-tls" not found
Jul 17 07:12:20.440925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.440886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8fp2" event={"ID":"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4","Type":"ContainerStarted","Data":"5f0344edd97f523cd564b760f0383d13c745d0bc01521175f0605f85842a7e86"}
Jul 17 07:12:20.442051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:20.442025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd" event={"ID":"283e90a3-f3ea-484f-88ce-effe90d837e6","Type":"ContainerStarted","Data":"c28847e5e197ed651a17271ea252ba9cd9e82d2de1dc6b5b4dcbac69fc69e714"}
Jul 17 07:12:21.281593 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:21.281318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:21.281772 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:21.281475    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 07:12:21.281772 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:21.281644    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5d976689-zstdt: secret "image-registry-tls" not found
Jul 17 07:12:21.281772 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:21.281711    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls podName:49ec566f-672e-436f-baab-a1bc01315dae nodeName:}" failed. No retries permitted until 2026-07-17 07:12:23.281689124 +0000 UTC m=+39.535111928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls") pod "image-registry-5d5d976689-zstdt" (UID: "49ec566f-672e-436f-baab-a1bc01315dae") : secret "image-registry-tls" not found
Jul 17 07:12:21.383012 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:21.382969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:21.383608 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:21.383138    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 07:12:21.383608 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:21.383201    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls podName:f8ea191a-a858-477f-8f34-31390a10e8a7 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:23.383183412 +0000 UTC m=+39.636606202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls") pod "dns-default-jnzc7" (UID: "f8ea191a-a858-477f-8f34-31390a10e8a7") : secret "dns-default-metrics-tls" not found
Jul 17 07:12:22.442999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:22.442957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:22.543752 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:22.543703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:23.011642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.011610    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jmxnh"]
Jul 17 07:12:23.025192 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.025156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.026162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.026134    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmxnh"]
Jul 17 07:12:23.027846 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.027823    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gcxnf\""
Jul 17 07:12:23.029102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.028923    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 07:12:23.029102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.028954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 07:12:23.029102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.028977    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 07:12:23.029352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.029167    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 07:12:23.050855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.050815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:23.118030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.117993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:23.197263 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.197213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a186c4-375f-4991-98b8-df6ccfd0d349-crio-socket\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.197444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.197361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.197444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.197397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.197444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.197420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a186c4-375f-4991-98b8-df6ccfd0d349-data-volume\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.197561 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.197479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86nf\" (UniqueName: \"kubernetes.io/projected/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-api-access-v86nf\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86nf\" (UniqueName: \"kubernetes.io/projected/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-api-access-v86nf\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a186c4-375f-4991-98b8-df6ccfd0d349-crio-socket\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:23.298245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a186c4-375f-4991-98b8-df6ccfd0d349-data-volume\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298261    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298331    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:23.798310683 +0000 UTC m=+40.051733485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298264    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a186c4-375f-4991-98b8-df6ccfd0d349-crio-socket\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298370    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5d976689-zstdt: secret "image-registry-tls" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298269    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298404    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls podName:49ec566f-672e-436f-baab-a1bc01315dae nodeName:}" failed. No retries permitted until 2026-07-17 07:12:27.298394071 +0000 UTC m=+43.551816865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls") pod "image-registry-5d5d976689-zstdt" (UID: "49ec566f-672e-436f-baab-a1bc01315dae") : secret "image-registry-tls" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.298470    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:23.798441316 +0000 UTC m=+40.051864127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : configmap "kube-rbac-proxy" not found
Jul 17 07:12:23.298531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.298507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a186c4-375f-4991-98b8-df6ccfd0d349-data-volume\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.323566 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.323532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86nf\" (UniqueName: \"kubernetes.io/projected/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-api-access-v86nf\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.399085 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.399049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:23.399246 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.399221    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 07:12:23.399331 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.399318    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls podName:f8ea191a-a858-477f-8f34-31390a10e8a7 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:27.399292982 +0000 UTC m=+43.652715777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls") pod "dns-default-jnzc7" (UID: "f8ea191a-a858-477f-8f34-31390a10e8a7") : secret "dns-default-metrics-tls" not found
Jul 17 07:12:23.719101 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.719013    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:23.803188 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.803150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.803363 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.803194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:23.803363 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.803283    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:23.803363 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.803351    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 07:12:23.803468 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.803377    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:24.8033575 +0000 UTC m=+41.056780304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:23.803468 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:23.803396    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:24.80338569 +0000 UTC m=+41.056808489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : configmap "kube-rbac-proxy" not found
Jul 17 07:12:23.823651 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:23.823624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:24.393905 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.393882    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbn4j_352cee13-e768-429f-ac8a-7e179831a97e/dns-node-resolver/0.log"
Jul 17 07:12:24.453632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.453591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8fp2" event={"ID":"1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4","Type":"ContainerStarted","Data":"0706dcabf4e091cdc0f1a414d35208a4e02e390b159090208d98cbc3049dbeef"}
Jul 17 07:12:24.717403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.717096    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"]
Jul 17 07:12:24.727693 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.727656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"
Jul 17 07:12:24.730488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.730411    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jhsrw\""
Jul 17 07:12:24.730627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.730587    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 07:12:24.731355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.731326    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 07:12:24.733190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.733157    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"]
Jul 17 07:12:24.761867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.761834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:24.812286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.812241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:24.812286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.812287    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:24.812523 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:24.812367    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 07:12:24.812523 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:24.812405    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:24.812523 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:24.812421    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:26.812406034 +0000 UTC m=+43.065828824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : configmap "kube-rbac-proxy" not found
Jul 17 07:12:24.812523 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:24.812458    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:26.812444486 +0000 UTC m=+43.065867286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:24.842442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.842407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:24.913438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:24.913402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6j5g\" (UniqueName: \"kubernetes.io/projected/239fcabd-a0a1-4152-b8af-6097fbeaf5db-kube-api-access-x6j5g\") pod \"migrator-7cc9c7ffd7-hmgdm\" (UID: \"239fcabd-a0a1-4152-b8af-6097fbeaf5db\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"
Jul 17 07:12:25.014142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.014050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6j5g\" (UniqueName: \"kubernetes.io/projected/239fcabd-a0a1-4152-b8af-6097fbeaf5db-kube-api-access-x6j5g\") pod \"migrator-7cc9c7ffd7-hmgdm\" (UID: \"239fcabd-a0a1-4152-b8af-6097fbeaf5db\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"
Jul 17 07:12:25.024819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.024784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6j5g\" (UniqueName: \"kubernetes.io/projected/239fcabd-a0a1-4152-b8af-6097fbeaf5db-kube-api-access-x6j5g\") pod \"migrator-7cc9c7ffd7-hmgdm\" (UID: \"239fcabd-a0a1-4152-b8af-6097fbeaf5db\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"
Jul 17 07:12:25.039399 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.039367    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"
Jul 17 07:12:25.176472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.176441    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm"]
Jul 17 07:12:25.179638 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:25.179610    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239fcabd_a0a1_4152_b8af_6097fbeaf5db.slice/crio-47fd7ce3c3333a5b53823c5d2b4483be73e2714d055811d5cca388732c6d338d WatchSource:0}: Error finding container 47fd7ce3c3333a5b53823c5d2b4483be73e2714d055811d5cca388732c6d338d: Status 404 returned error can't find the container with id 47fd7ce3c3333a5b53823c5d2b4483be73e2714d055811d5cca388732c6d338d
Jul 17 07:12:25.458522 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.458483    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="d303f961b30b3ebab13e85fd123c9f467f3413c6db5593ce077c1ab60bd69498" exitCode=0
Jul 17 07:12:25.458711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.458569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"d303f961b30b3ebab13e85fd123c9f467f3413c6db5593ce077c1ab60bd69498"}
Jul 17 07:12:25.459691 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.459667    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm" event={"ID":"239fcabd-a0a1-4152-b8af-6097fbeaf5db","Type":"ContainerStarted","Data":"47fd7ce3c3333a5b53823c5d2b4483be73e2714d055811d5cca388732c6d338d"}
Jul 17 07:12:25.461480 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.461460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd" event={"ID":"283e90a3-f3ea-484f-88ce-effe90d837e6","Type":"ContainerStarted","Data":"d6c38cc6f7d5ce7c194a02e1a17edd2a0a456b501458269c155a89b11da348ed"}
Jul 17 07:12:25.501495 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.501452    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k8fp2" podStartSLOduration=2.2229112 podStartE2EDuration="6.501434232s" totalImagesPullingTime="4.278523032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:25.50128936 +0000 UTC m=+41.754712189" watchObservedRunningTime="2026-07-17 07:12:25.501434232 +0000 UTC m=+41.754857045"
Jul 17 07:12:25.703149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.703118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:25.763362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.763260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:25.793596 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:25.793572    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pfz9q_ba41118a-688c-4ba8-93aa-c61b3ddbc615/node-ca/0.log"
Jul 17 07:12:26.138558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.138523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:26.240577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.240545    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:26.320681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.320634    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd" podStartSLOduration=3.088834269 podStartE2EDuration="7.320618463s" totalImagesPullingTime="4.231784194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:25.51850015 +0000 UTC m=+41.771922988" watchObservedRunningTime="2026-07-17 07:12:26.320618463 +0000 UTC m=+42.574041275"
Jul 17 07:12:26.320839 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.320749    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"]
Jul 17 07:12:26.344671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.344643    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"]
Jul 17 07:12:26.344804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.344727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"
Jul 17 07:12:26.347244 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.347227    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zmwl8\""
Jul 17 07:12:26.396060 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.395989    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8fp2_1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4/serve-healthcheck-canary/0.log"
Jul 17 07:12:26.417038 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.416997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:26.466096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.466066    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f12d13b-a6ab-4340-a4f2-35501cb9566e" containerID="d2ed49a9eb8373e3090c210027468956d69f8d4ffc46609032ef99a25e8c1b46" exitCode=0
Jul 17 07:12:26.466222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.466140    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerDied","Data":"d2ed49a9eb8373e3090c210027468956d69f8d4ffc46609032ef99a25e8c1b46"}
Jul 17 07:12:26.491389 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.491361    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:26.528461 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.528416    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2g7\" (UniqueName: \"kubernetes.io/projected/d4500b28-d915-4a85-baac-4625551bff66-kube-api-access-rt2g7\") pod \"network-check-source-7bc5b6bf85-pdxgf\" (UID: \"d4500b28-d915-4a85-baac-4625551bff66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"
Jul 17 07:12:26.629618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.629579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2g7\" (UniqueName: \"kubernetes.io/projected/d4500b28-d915-4a85-baac-4625551bff66-kube-api-access-rt2g7\") pod \"network-check-source-7bc5b6bf85-pdxgf\" (UID: \"d4500b28-d915-4a85-baac-4625551bff66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"
Jul 17 07:12:26.638019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.637987    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2g7\" (UniqueName: \"kubernetes.io/projected/d4500b28-d915-4a85-baac-4625551bff66-kube-api-access-rt2g7\") pod \"network-check-source-7bc5b6bf85-pdxgf\" (UID: \"d4500b28-d915-4a85-baac-4625551bff66\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"
Jul 17 07:12:26.652845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.652773    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"
Jul 17 07:12:26.831903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.831876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:26.832252 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.832032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:26.832252 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:26.832146    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:26.832252 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:26.832219    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:30.832200138 +0000 UTC m=+47.085622950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:26.832549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.832497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a186c4-375f-4991-98b8-df6ccfd0d349-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:26.914181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.914102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:26.944157 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:26.944120    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf"]
Jul 17 07:12:26.946758 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:26.946737    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4500b28_d915_4a85_baac_4625551bff66.slice/crio-b23a93aac74c9c08defb94fe7c44d7e1cf82a54a0224a206c38bad0bf09d8ad5 WatchSource:0}: Error finding container b23a93aac74c9c08defb94fe7c44d7e1cf82a54a0224a206c38bad0bf09d8ad5: Status 404 returned error can't find the container with id b23a93aac74c9c08defb94fe7c44d7e1cf82a54a0224a206c38bad0bf09d8ad5
Jul 17 07:12:27.336683 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.336555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:27.336683 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:27.336604    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 07:12:27.336683 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:27.336626    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5d976689-zstdt: secret "image-registry-tls" not found
Jul 17 07:12:27.336683 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:27.336687    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls podName:49ec566f-672e-436f-baab-a1bc01315dae nodeName:}" failed. No retries permitted until 2026-07-17 07:12:35.336669202 +0000 UTC m=+51.590091998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls") pod "image-registry-5d5d976689-zstdt" (UID: "49ec566f-672e-436f-baab-a1bc01315dae") : secret "image-registry-tls" not found
Jul 17 07:12:27.437970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.437939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:27.438177 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:27.438157    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 07:12:27.438243 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:27.438232    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls podName:f8ea191a-a858-477f-8f34-31390a10e8a7 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:35.438215573 +0000 UTC m=+51.691638364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls") pod "dns-default-jnzc7" (UID: "f8ea191a-a858-477f-8f34-31390a10e8a7") : secret "dns-default-metrics-tls" not found
Jul 17 07:12:27.470443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.470333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm" event={"ID":"239fcabd-a0a1-4152-b8af-6097fbeaf5db","Type":"ContainerStarted","Data":"ec29e9b3a5bef3f070e8f7f8cbd1923ecd189c68f92033b568cacaba996e3148"}
Jul 17 07:12:27.470443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.470436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm" event={"ID":"239fcabd-a0a1-4152-b8af-6097fbeaf5db","Type":"ContainerStarted","Data":"63aa4eaafefbf51c719054ac6920e3568c2a8c186450cd209df42fb87c10a4a0"}
Jul 17 07:12:27.473554 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.473527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" event={"ID":"5f12d13b-a6ab-4340-a4f2-35501cb9566e","Type":"ContainerStarted","Data":"37ea756e6e07bb23de9f5e84886041175d0428d84ff33bb4122b795a96d4940a"}
Jul 17 07:12:27.474665 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.474641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf" event={"ID":"d4500b28-d915-4a85-baac-4625551bff66","Type":"ContainerStarted","Data":"b23a93aac74c9c08defb94fe7c44d7e1cf82a54a0224a206c38bad0bf09d8ad5"}
Jul 17 07:12:27.486371 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.486313    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-hmgdm" podStartSLOduration=1.45870107 podStartE2EDuration="3.486298429s" totalImagesPullingTime="2.027597359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:27.485763436 +0000 UTC m=+43.739186267" watchObservedRunningTime="2026-07-17 07:12:27.486298429 +0000 UTC m=+43.739721253"
Jul 17 07:12:27.506138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.506088    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vf2cz" podStartSLOduration=8.119915164 podStartE2EDuration="42.506073317s" totalImagesPullingTime="34.386158153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:27.504429232 +0000 UTC m=+43.757852044" watchObservedRunningTime="2026-07-17 07:12:27.506073317 +0000 UTC m=+43.759496128"
Jul 17 07:12:27.632171 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.632136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:27.733913 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.733830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:27.798977 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:27.798932    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:28.009673 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.009595    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-dxvvv"]
Jul 17 07:12:28.024802 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.024773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-dxvvv"]
Jul 17 07:12:28.025003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.024895    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.027827 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.027799    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bpv6p\""
Jul 17 07:12:28.028080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.028063    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 07:12:28.028248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.028227    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 07:12:28.028498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.028481    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 07:12:28.028999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.028980    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 07:12:28.042808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.042782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxdh\" (UniqueName: \"kubernetes.io/projected/dac4c095-f754-4992-a4d6-a5317131fc63-kube-api-access-kcxdh\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.042914 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.042822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dac4c095-f754-4992-a4d6-a5317131fc63-signing-cabundle\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.042914 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.042867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dac4c095-f754-4992-a4d6-a5317131fc63-signing-key\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.045184 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.045159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:28.104309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.104274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:28.143382 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.143329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxdh\" (UniqueName: \"kubernetes.io/projected/dac4c095-f754-4992-a4d6-a5317131fc63-kube-api-access-kcxdh\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.143382 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.143381    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dac4c095-f754-4992-a4d6-a5317131fc63-signing-cabundle\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.143781 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.143454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dac4c095-f754-4992-a4d6-a5317131fc63-signing-key\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.144191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.144169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dac4c095-f754-4992-a4d6-a5317131fc63-signing-cabundle\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.146154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.146128    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dac4c095-f754-4992-a4d6-a5317131fc63-signing-key\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.152813 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.152783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxdh\" (UniqueName: \"kubernetes.io/projected/dac4c095-f754-4992-a4d6-a5317131fc63-kube-api-access-kcxdh\") pod \"service-ca-58598695f7-dxvvv\" (UID: \"dac4c095-f754-4992-a4d6-a5317131fc63\") " pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.336936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.336899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-dxvvv"
Jul 17 07:12:28.488437 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:28.488404    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-dxvvv"]
Jul 17 07:12:28.492496 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:28.492474    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac4c095_f754_4992_a4d6_a5317131fc63.slice/crio-15a0e6b3a04d57743fab85fbe7758e911e773c020abf2144b687c20aa57ad5eb WatchSource:0}: Error finding container 15a0e6b3a04d57743fab85fbe7758e911e773c020abf2144b687c20aa57ad5eb: Status 404 returned error can't find the container with id 15a0e6b3a04d57743fab85fbe7758e911e773c020abf2144b687c20aa57ad5eb
Jul 17 07:12:29.112519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.112482    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:29.482274 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.482201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-dxvvv" event={"ID":"dac4c095-f754-4992-a4d6-a5317131fc63","Type":"ContainerStarted","Data":"5c348dd7c933290118b787ce001cb9f9c0070836a3d4074d539e0f92cb0d533f"}
Jul 17 07:12:29.482274 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.482259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-dxvvv" event={"ID":"dac4c095-f754-4992-a4d6-a5317131fc63","Type":"ContainerStarted","Data":"15a0e6b3a04d57743fab85fbe7758e911e773c020abf2144b687c20aa57ad5eb"}
Jul 17 07:12:29.501677 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.501610    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-dxvvv" podStartSLOduration=2.501588417 podStartE2EDuration="2.501588417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:29.500281821 +0000 UTC m=+45.753704669" watchObservedRunningTime="2026-07-17 07:12:29.501588417 +0000 UTC m=+45.755011242"
Jul 17 07:12:29.873113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.873081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:29.973109 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:29.973074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:30.509498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:30.509451    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:30.663526 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:30.663452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:30.760368 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:30.760271    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:30.868227 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:30.868184    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:30.868453 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:30.868382    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:30.868516 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:30.868470    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls podName:b1a186c4-375f-4991-98b8-df6ccfd0d349 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:38.868444776 +0000 UTC m=+55.121867570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jmxnh" (UID: "b1a186c4-375f-4991-98b8-df6ccfd0d349") : secret "insights-runtime-extractor-tls" not found
Jul 17 07:12:30.875041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:30.875009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:31.172015 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:31.171981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:31.310499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:31.310357    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:31.415770 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:31.415720    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:31.491795 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:31.491755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf" event={"ID":"d4500b28-d915-4a85-baac-4625551bff66","Type":"ContainerStarted","Data":"73b3e8a3e54a48866295b71198fbfc9d5001b22058ae0fd63a4273805b3adb83"}
Jul 17 07:12:31.507595 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:31.507548    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-pdxgf" podStartSLOduration=1.49570349 podStartE2EDuration="5.507532669s" totalImagesPullingTime="4.011829179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:31.507426123 +0000 UTC m=+47.760848950" watchObservedRunningTime="2026-07-17 07:12:31.507532669 +0000 UTC m=+47.760955481"
Jul 17 07:12:32.006747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:32.006712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:32.589089 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:32.589053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:32.688250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:32.688213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:32.859884 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:32.859781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:33.015083 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:33.015047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:33.214900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:33.214812    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:33.940389 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:33.940350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:34.201001 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:34.200923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:34.408950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:34.408911    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:34.609018 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:34.608983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:34.817076 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:34.817040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:34.887055 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:34.886977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:35.005467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.005435    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:35.211440 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.211334    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:35.405761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.405728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:35.408247 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.408225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"image-registry-5d5d976689-zstdt\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") " pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:35.487976 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.487883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:35.507136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.507114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:35.509442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.509405    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8ea191a-a858-477f-8f34-31390a10e8a7-metrics-tls\") pod \"dns-default-jnzc7\" (UID: \"f8ea191a-a858-477f-8f34-31390a10e8a7\") " pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:35.629138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.629113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:12:35.631469 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:35.631434    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ec566f_672e_436f_baab_a1bc01315dae.slice/crio-73b501b2b1b9a87eaa1e5bada478e1e3e528594156358533f25dd55e30e2cd2e WatchSource:0}: Error finding container 73b501b2b1b9a87eaa1e5bada478e1e3e528594156358533f25dd55e30e2cd2e: Status 404 returned error can't find the container with id 73b501b2b1b9a87eaa1e5bada478e1e3e528594156358533f25dd55e30e2cd2e
Jul 17 07:12:35.803636 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.803599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:35.928104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:35.928071    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnzc7"]
Jul 17 07:12:35.930567 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:35.930539    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ea191a_a858_477f_8f34_31390a10e8a7.slice/crio-017c37946be033aa18339dbf14fb7804a6498294de52d88a7214255a8f7d19f8 WatchSource:0}: Error finding container 017c37946be033aa18339dbf14fb7804a6498294de52d88a7214255a8f7d19f8: Status 404 returned error can't find the container with id 017c37946be033aa18339dbf14fb7804a6498294de52d88a7214255a8f7d19f8
Jul 17 07:12:36.219444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.219358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:36.511958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.511872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnzc7" event={"ID":"f8ea191a-a858-477f-8f34-31390a10e8a7","Type":"ContainerStarted","Data":"017c37946be033aa18339dbf14fb7804a6498294de52d88a7214255a8f7d19f8"}
Jul 17 07:12:36.513300 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.513270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" event={"ID":"49ec566f-672e-436f-baab-a1bc01315dae","Type":"ContainerStarted","Data":"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"}
Jul 17 07:12:36.513455 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.513312    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" event={"ID":"49ec566f-672e-436f-baab-a1bc01315dae","Type":"ContainerStarted","Data":"73b501b2b1b9a87eaa1e5bada478e1e3e528594156358533f25dd55e30e2cd2e"}
Jul 17 07:12:36.513455 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.513403    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:12:36.535754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.535704    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" podStartSLOduration=20.535687744 podStartE2EDuration="20.535687744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:36.535660374 +0000 UTC m=+52.789083185" watchObservedRunningTime="2026-07-17 07:12:36.535687744 +0000 UTC m=+52.789110555"
Jul 17 07:12:36.604393 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.604334    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:36.705911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.705871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:36.807177 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:36.807141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:37.009469 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:37.009365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:37.209213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:37.209139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:38.201811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.201776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:38.453525 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.453488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:38.525777 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.525739    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnzc7" event={"ID":"f8ea191a-a858-477f-8f34-31390a10e8a7","Type":"ContainerStarted","Data":"97998f37587a1c9a52ec370d1af82116ade84b1107b3fdc9ed1fb62a502fcfbc"}
Jul 17 07:12:38.525969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.525798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnzc7" event={"ID":"f8ea191a-a858-477f-8f34-31390a10e8a7","Type":"ContainerStarted","Data":"e6dfd29750c1deb64798e232b987990de4bffb2f661a60ed016f0635d7f1e369"}
Jul 17 07:12:38.525969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.525875    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:38.545311 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.545258    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jnzc7" podStartSLOduration=17.313831865 podStartE2EDuration="19.545241644s" totalImagesPullingTime="2.231409779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:38.544581935 +0000 UTC m=+54.798004747" watchObservedRunningTime="2026-07-17 07:12:38.545241644 +0000 UTC m=+54.798664453"
Jul 17 07:12:38.808599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.808568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:38.939120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.939080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:38.941481 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:38.941450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a186c4-375f-4991-98b8-df6ccfd0d349-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jmxnh\" (UID: \"b1a186c4-375f-4991-98b8-df6ccfd0d349\") " pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:39.012025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:39.011992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:39.237273 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:39.237184    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jmxnh"
Jul 17 07:12:39.365626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:39.365592    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jmxnh"]
Jul 17 07:12:39.369751 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:39.369726    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a186c4_375f_4991_98b8_df6ccfd0d349.slice/crio-6b08e275820d81612b311eae9b2899aa1783661f3e486ab3894cdd76262e0656 WatchSource:0}: Error finding container 6b08e275820d81612b311eae9b2899aa1783661f3e486ab3894cdd76262e0656: Status 404 returned error can't find the container with id 6b08e275820d81612b311eae9b2899aa1783661f3e486ab3894cdd76262e0656
Jul 17 07:12:39.531257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:39.531169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmxnh" event={"ID":"b1a186c4-375f-4991-98b8-df6ccfd0d349","Type":"ContainerStarted","Data":"6b08e275820d81612b311eae9b2899aa1783661f3e486ab3894cdd76262e0656"}
Jul 17 07:12:39.603234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:39.603182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:40.101611 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:40.101555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:40.243706 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:40.243667    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:40.411860 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:40.411753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:40.849080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:40.849037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.020710 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.020673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.208185 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.208098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.670355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.670290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.770792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.770755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.870559 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.870522    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:41.970361 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:41.970266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:42.316259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:42.316220    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:42.379037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:42.379001    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:42.576982 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:42.576886    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:42.676943 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:42.676899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:42.945486 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:42.945402    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:43.042813 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:43.042752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:43.553616 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:43.553571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:43.747219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:43.747185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:43.993868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:43.993785    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:44.553896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:44.553855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmxnh" event={"ID":"b1a186c4-375f-4991-98b8-df6ccfd0d349","Type":"ContainerStarted","Data":"659aefb65b56c0cba12c6366fba261df781da38122013c7ffa1235e3f40dcd4e"}
Jul 17 07:12:44.769428 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:44.769393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:44.870900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:44.870821    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:44.970971 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:44.970909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:46.439743 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:46.439711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2tp8r"
Jul 17 07:12:48.533244 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:48.533167    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jnzc7"
Jul 17 07:12:48.573070 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:48.573038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmxnh" event={"ID":"b1a186c4-375f-4991-98b8-df6ccfd0d349","Type":"ContainerStarted","Data":"493c5aa7a721dda9791540f469d360b3668223d26b8bfce3e7952a1e76e62ac8"}
Jul 17 07:12:50.037472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.037428    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:50.040483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.040460    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 07:12:50.050023 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.050001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c11dcfbf-059f-4cac-b79a-4206b3e6e8b1-metrics-certs\") pod \"network-metrics-daemon-fv7dc\" (UID: \"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1\") " pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:50.138648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.138609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:50.138648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.138659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:50.141544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.141504    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 07:12:50.141837 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.141812    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mmk\" (UniqueName: \"kubernetes.io/projected/82d92bb9-71f7-4779-a81a-99f898cd1e98-kube-api-access-h7mmk\") pod \"network-check-target-sv54n\" (UID: \"82d92bb9-71f7-4779-a81a-99f898cd1e98\") " pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:50.151066 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.151038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ba69a4b-39b3-4d83-83be-47497ce7d246-original-pull-secret\") pod \"global-pull-secret-syncer-hktn8\" (UID: \"8ba69a4b-39b3-4d83-83be-47497ce7d246\") " pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:50.298154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.298078    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jgz\""
Jul 17 07:12:50.303629 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.303604    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c28tv\""
Jul 17 07:12:50.305552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.305532    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv7dc"
Jul 17 07:12:50.305644 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.305595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hktn8"
Jul 17 07:12:50.311713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:50.311691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:51.512078 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.512045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:51.618098 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.618058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:51.667532 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.667497    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fv7dc"]
Jul 17 07:12:51.671640 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:51.671608    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11dcfbf_059f_4cac_b79a_4206b3e6e8b1.slice/crio-9a80e0e5322287c5d3576946e4142b1ec7cac9e2dc5182e7f5ae85ce188d66e2 WatchSource:0}: Error finding container 9a80e0e5322287c5d3576946e4142b1ec7cac9e2dc5182e7f5ae85ce188d66e2: Status 404 returned error can't find the container with id 9a80e0e5322287c5d3576946e4142b1ec7cac9e2dc5182e7f5ae85ce188d66e2
Jul 17 07:12:51.680933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.680908    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sv54n"]
Jul 17 07:12:51.683523 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:51.683498    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d92bb9_71f7_4779_a81a_99f898cd1e98.slice/crio-c5e5cb1e729c647e5c322b1e13652d1e278fb4257eea0116599b83293406aad2 WatchSource:0}: Error finding container c5e5cb1e729c647e5c322b1e13652d1e278fb4257eea0116599b83293406aad2: Status 404 returned error can't find the container with id c5e5cb1e729c647e5c322b1e13652d1e278fb4257eea0116599b83293406aad2
Jul 17 07:12:51.688066 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.688041    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"]
Jul 17 07:12:51.691853 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:51.691825    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba69a4b_39b3_4d83_83be_47497ce7d246.slice/crio-3d568b44197040bf93f8002ecdf97d5c7940179d083f6c77fee62b38fcd67cf4 WatchSource:0}: Error finding container 3d568b44197040bf93f8002ecdf97d5c7940179d083f6c77fee62b38fcd67cf4: Status 404 returned error can't find the container with id 3d568b44197040bf93f8002ecdf97d5c7940179d083f6c77fee62b38fcd67cf4
Jul 17 07:12:51.693266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.693245    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"]
Jul 17 07:12:51.693396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.693380    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.695817 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.695792    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 07:12:51.696636 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.696617    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 07:12:51.696741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.696677    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 07:12:51.697055 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.697034    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hktn8"]
Jul 17 07:12:51.697146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.697106    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jqpjz\""
Jul 17 07:12:51.697146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.697132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.697253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.697143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 07:12:51.699608 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.699589    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 07:12:51.699691 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.699610    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8v5cm\""
Jul 17 07:12:51.699691 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.699598    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 07:12:51.716316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.716286    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"]
Jul 17 07:12:51.719650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.719627    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"]
Jul 17 07:12:51.719761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.719660    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"]
Jul 17 07:12:51.719822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.719777    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.722181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.722160    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 07:12:51.745425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.745392    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"]
Jul 17 07:12:51.750859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.750821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.750974 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.750888    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddr4l\" (UniqueName: \"kubernetes.io/projected/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-kube-api-access-ddr4l\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.755375 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.755324    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:12:51.759671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.759647    2584 patch_prober.go:28] interesting pod/image-registry-5d5d976689-zstdt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 07:12:51.759772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.759688    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" podUID="49ec566f-672e-436f-baab-a1bc01315dae" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 07:12:51.795152 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.795116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:51.796256 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.796233    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-9nd6t"]
Jul 17 07:12:51.809328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.809305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:12:51.816323 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.816301    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 07:12:51.816724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.816303    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 07:12:51.816724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.816454    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bqdc4\""
Jul 17 07:12:51.817638 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.817618    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-9nd6t"]
Jul 17 07:12:51.852281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.852281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fdc295c0-e400-4da9-91fe-b23573af849e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.852549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852309    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddr4l\" (UniqueName: \"kubernetes.io/projected/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-kube-api-access-ddr4l\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.852549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/913075d4-3368-4904-b60b-d02f3c27a948-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.852549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913075d4-3368-4904-b60b-d02f3c27a948-tmp\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.852549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wn9\" (UniqueName: \"kubernetes.io/projected/913075d4-3368-4904-b60b-d02f3c27a948-kube-api-access-t4wn9\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.852549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.852477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdc295c0-e400-4da9-91fe-b23573af849e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.854736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.854713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.865367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.865315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddr4l\" (UniqueName: \"kubernetes.io/projected/1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47-kube-api-access-ddr4l\") pod \"managed-serviceaccount-addon-agent-6c6fd86468-qds29\" (UID: \"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:51.884740 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.884712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:51.891980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.891956    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666fdc4999-9prdc"]
Jul 17 07:12:51.918488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.918448    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666fdc4999-9prdc"]
Jul 17 07:12:51.918619 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.918587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:51.952864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdc295c0-e400-4da9-91fe-b23573af849e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.953013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fdc295c0-e400-4da9-91fe-b23573af849e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.953013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkzk\" (UniqueName: \"kubernetes.io/projected/9c6a280d-1dc5-491c-b5ec-77ccd0af9f48-kube-api-access-lbkzk\") pod \"downloads-5bc67dcd46-9nd6t\" (UID: \"9c6a280d-1dc5-491c-b5ec-77ccd0af9f48\") " pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:12:51.953013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952930    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/913075d4-3368-4904-b60b-d02f3c27a948-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.953013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952946    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913075d4-3368-4904-b60b-d02f3c27a948-tmp\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.953013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.952966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wn9\" (UniqueName: \"kubernetes.io/projected/913075d4-3368-4904-b60b-d02f3c27a948-kube-api-access-t4wn9\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.953423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.953402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913075d4-3368-4904-b60b-d02f3c27a948-tmp\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.953728 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.953705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fdc295c0-e400-4da9-91fe-b23573af849e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.955442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.955420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fdc295c0-e400-4da9-91fe-b23573af849e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gm8nc\" (UID: \"fdc295c0-e400-4da9-91fe-b23573af849e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:51.955548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.955531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/913075d4-3368-4904-b60b-d02f3c27a948-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.961222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.961203    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wn9\" (UniqueName: \"kubernetes.io/projected/913075d4-3368-4904-b60b-d02f3c27a948-kube-api-access-t4wn9\") pod \"klusterlet-addon-workmgr-d9b9f447c-lsnt4\" (UID: \"913075d4-3368-4904-b60b-d02f3c27a948\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:51.983419 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:51.983387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:52.013714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.013684    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"
Jul 17 07:12:52.034895 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.034863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"
Jul 17 07:12:52.052155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.052127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:12:52.053364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkzk\" (UniqueName: \"kubernetes.io/projected/9c6a280d-1dc5-491c-b5ec-77ccd0af9f48-kube-api-access-lbkzk\") pod \"downloads-5bc67dcd46-9nd6t\" (UID: \"9c6a280d-1dc5-491c-b5ec-77ccd0af9f48\") " pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:12:52.053420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-image-registry-private-configuration\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-tls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053494 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-certificates\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-kube-api-access-fffls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053584 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d91072-feb2-4a89-b1b7-96cfaa7f5416-ca-trust-extracted\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-installation-pull-secrets\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-bound-sa-token\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.053672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.053643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-trusted-ca\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.066612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.066583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkzk\" (UniqueName: \"kubernetes.io/projected/9c6a280d-1dc5-491c-b5ec-77ccd0af9f48-kube-api-access-lbkzk\") pod \"downloads-5bc67dcd46-9nd6t\" (UID: \"9c6a280d-1dc5-491c-b5ec-77ccd0af9f48\") " pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:12:52.082614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.082575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:52.119097 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.119068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-tls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-certificates\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-kube-api-access-fffls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d91072-feb2-4a89-b1b7-96cfaa7f5416-ca-trust-extracted\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-installation-pull-secrets\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.156320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-bound-sa-token\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.157970    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-certificates\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.158246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.158210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d91072-feb2-4a89-b1b7-96cfaa7f5416-ca-trust-extracted\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.159522 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.158823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-trusted-ca\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.159522 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.158905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-image-registry-private-configuration\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.166797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.159908    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d91072-feb2-4a89-b1b7-96cfaa7f5416-trusted-ca\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.172756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.171743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-installation-pull-secrets\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.172756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.171935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-bound-sa-token\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.172756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.172284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-registry-tls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.175168 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.175117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffls\" (UniqueName: \"kubernetes.io/projected/21d91072-feb2-4a89-b1b7-96cfaa7f5416-kube-api-access-fffls\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.179004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.178977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d91072-feb2-4a89-b1b7-96cfaa7f5416-image-registry-private-configuration\") pod \"image-registry-666fdc4999-9prdc\" (UID: \"21d91072-feb2-4a89-b1b7-96cfaa7f5416\") " pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.211132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.211033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:52.226444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.226376    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29"]
Jul 17 07:12:52.230367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.230174    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.269734 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.269654    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc"]
Jul 17 07:12:52.277073 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:52.277026    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc295c0_e400_4da9_91fe_b23573af849e.slice/crio-d8bc867f0fab148facc840059dc76981b88164f74c12548f6ed1647ae0ef20c0 WatchSource:0}: Error finding container d8bc867f0fab148facc840059dc76981b88164f74c12548f6ed1647ae0ef20c0: Status 404 returned error can't find the container with id d8bc867f0fab148facc840059dc76981b88164f74c12548f6ed1647ae0ef20c0
Jul 17 07:12:52.297855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.297825    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"]
Jul 17 07:12:52.356906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.356874    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-9nd6t"]
Jul 17 07:12:52.417572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.417491    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666fdc4999-9prdc"]
Jul 17 07:12:52.421422 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:52.421389    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d91072_feb2_4a89_b1b7_96cfaa7f5416.slice/crio-42e3cf9f4c96de9be1aeba1e930f32ac5ca8d684f19f3e792ded6316cea339ff WatchSource:0}: Error finding container 42e3cf9f4c96de9be1aeba1e930f32ac5ca8d684f19f3e792ded6316cea339ff: Status 404 returned error can't find the container with id 42e3cf9f4c96de9be1aeba1e930f32ac5ca8d684f19f3e792ded6316cea339ff
Jul 17 07:12:52.596385 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.596267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jmxnh" event={"ID":"b1a186c4-375f-4991-98b8-df6ccfd0d349","Type":"ContainerStarted","Data":"009697877df5546015d4fd09cb6519491b1950746f7bbf4a0869b1baa53e94b5"}
Jul 17 07:12:52.597664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.597632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc" event={"ID":"fdc295c0-e400-4da9-91fe-b23573af849e","Type":"ContainerStarted","Data":"d8bc867f0fab148facc840059dc76981b88164f74c12548f6ed1647ae0ef20c0"}
Jul 17 07:12:52.598842 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.598814    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv7dc" event={"ID":"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1","Type":"ContainerStarted","Data":"9a80e0e5322287c5d3576946e4142b1ec7cac9e2dc5182e7f5ae85ce188d66e2"}
Jul 17 07:12:52.600157 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.600123    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hktn8" event={"ID":"8ba69a4b-39b3-4d83-83be-47497ce7d246","Type":"ContainerStarted","Data":"3d568b44197040bf93f8002ecdf97d5c7940179d083f6c77fee62b38fcd67cf4"}
Jul 17 07:12:52.601774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.601752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sv54n" event={"ID":"82d92bb9-71f7-4779-a81a-99f898cd1e98","Type":"ContainerStarted","Data":"953b09407a551b69ad5c4ef1309f11a9f81457c4036623019dbbef54330cb4c2"}
Jul 17 07:12:52.601868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.601787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sv54n" event={"ID":"82d92bb9-71f7-4779-a81a-99f898cd1e98","Type":"ContainerStarted","Data":"c5e5cb1e729c647e5c322b1e13652d1e278fb4257eea0116599b83293406aad2"}
Jul 17 07:12:52.601868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.601857    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:12:52.603382 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.603362    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666fdc4999-9prdc" event={"ID":"21d91072-feb2-4a89-b1b7-96cfaa7f5416","Type":"ContainerStarted","Data":"ceff5d8abc3f62b24e318168ae396cccc7f4f4132bc277c0f9aa466daa00bf30"}
Jul 17 07:12:52.603476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.603397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666fdc4999-9prdc" event={"ID":"21d91072-feb2-4a89-b1b7-96cfaa7f5416","Type":"ContainerStarted","Data":"42e3cf9f4c96de9be1aeba1e930f32ac5ca8d684f19f3e792ded6316cea339ff"}
Jul 17 07:12:52.603476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.603462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:12:52.604550 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.604521    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-9nd6t" event={"ID":"9c6a280d-1dc5-491c-b5ec-77ccd0af9f48","Type":"ContainerStarted","Data":"dcd1abba127328485c3a9d2985fdd07aabe9454cf03ce858ef989b20cd73ad2f"}
Jul 17 07:12:52.605616 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.605597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4" event={"ID":"913075d4-3368-4904-b60b-d02f3c27a948","Type":"ContainerStarted","Data":"846af591e603c476e0bd1fdc7d180e5a9733938fc863076afa6b80ba233c9d8f"}
Jul 17 07:12:52.606641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.606623    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29" event={"ID":"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47","Type":"ContainerStarted","Data":"b39a21ad935a7b72ba0f5f4a042046ba906abdfdb04f9e2b6b1dda44154b7377"}
Jul 17 07:12:52.645022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.644971    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jmxnh" podStartSLOduration=18.570622009 podStartE2EDuration="30.64495029s" totalImagesPullingTime="12.074328281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:12:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:52.644474495 +0000 UTC m=+68.897897334" watchObservedRunningTime="2026-07-17 07:12:52.64495029 +0000 UTC m=+68.898373124"
Jul 17 07:12:52.681584 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.681524    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666fdc4999-9prdc" podStartSLOduration=1.681501367 podStartE2EDuration="1.681501367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:52.679700678 +0000 UTC m=+68.933123492" watchObservedRunningTime="2026-07-17 07:12:52.681501367 +0000 UTC m=+68.934924188"
Jul 17 07:12:52.698258 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:52.698198    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sv54n" podStartSLOduration=67.698180118 podStartE2EDuration="1m7.698180118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:52.697279977 +0000 UTC m=+68.950702790" watchObservedRunningTime="2026-07-17 07:12:52.698180118 +0000 UTC m=+68.951602931"
Jul 17 07:12:53.375124 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.375011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:53.475551 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.475503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:53.568362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.568243    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"]
Jul 17 07:12:53.571107 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.570892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:53.591131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.590094    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"]
Jul 17 07:12:53.591131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.590213    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:53.594658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.592841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 17 07:12:53.594658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.593110    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c86sv\""
Jul 17 07:12:53.618824 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.618789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv7dc" event={"ID":"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1","Type":"ContainerStarted","Data":"65e14bc02c7d429a913a73d795bccbeb58c772157e814b5b19d9b6c961f006da"}
Jul 17 07:12:53.670574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.670539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g688l\" (UID: \"a9e5dd25-9460-45cb-b6cf-e656df275567\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:53.771694 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:53.771656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g688l\" (UID: \"a9e5dd25-9460-45cb-b6cf-e656df275567\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:53.774190 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:53.774164    2584 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 17 07:12:53.774288 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:12:53.774237    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates podName:a9e5dd25-9460-45cb-b6cf-e656df275567 nodeName:}" failed. No retries permitted until 2026-07-17 07:12:54.274215974 +0000 UTC m=+70.527638770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-g688l" (UID: "a9e5dd25-9460-45cb-b6cf-e656df275567") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 17 07:12:54.279656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.279122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g688l\" (UID: \"a9e5dd25-9460-45cb-b6cf-e656df275567\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:54.291439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.291261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a9e5dd25-9460-45cb-b6cf-e656df275567-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-g688l\" (UID: \"a9e5dd25-9460-45cb-b6cf-e656df275567\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:54.293873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.293796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:54.505352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.505100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:12:54.638071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.638032    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv7dc" event={"ID":"c11dcfbf-059f-4cac-b79a-4206b3e6e8b1","Type":"ContainerStarted","Data":"9c019baeb42dac0ec607127c18c486000afea252912c41976698c721c360751e"}
Jul 17 07:12:54.780676 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.780644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:54.852235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:54.852158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:55.250559 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:55.250497    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fv7dc" podStartSLOduration=68.583438382 podStartE2EDuration="1m10.250469592s" totalImagesPullingTime="1.66703121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:12:54.65745712 +0000 UTC m=+70.910879933" watchObservedRunningTime="2026-07-17 07:12:55.250469592 +0000 UTC m=+71.503892399"
Jul 17 07:12:55.250787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:55.250763    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"]
Jul 17 07:12:57.275548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.275510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:57.390992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.390955    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:57.416149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.416114    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:12:57.419915 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.419794    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.423710 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.423683    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fgp5z\""
Jul 17 07:12:57.424013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.423982    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 07:12:57.424815 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.424797    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 07:12:57.424933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.424888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 07:12:57.425001 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.424894    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 07:12:57.425059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.424997    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 07:12:57.437926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.437865    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:12:57.507866 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.507826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.508052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.507902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.508052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.507996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.508052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.508033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.508213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.508070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrw4\" (UniqueName: \"kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.508213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.508095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.612966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.612917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.613153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.613003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.613153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.613092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrw4\" (UniqueName: \"kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.613153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.613132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.613316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.613198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.613530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.613494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.614354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.614314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.614578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.614552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.615098 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.615076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.616196 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.616171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.617305 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.617271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.625789 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.625766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrw4\" (UniqueName: \"kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4\") pod \"console-6d7848ffbf-5h256\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") " pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.715590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.715550    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:57.732548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.732511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:12:57.816266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:57.816220    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:58.248410 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:58.248365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:58.304417 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:58.304382    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e5dd25_9460_45cb_b6cf_e656df275567.slice/crio-6abca2e0a2fdb5c424e0581c3c99d01a71492f868b5301f9b328f16558e3acb4 WatchSource:0}: Error finding container 6abca2e0a2fdb5c424e0581c3c99d01a71492f868b5301f9b328f16558e3acb4: Status 404 returned error can't find the container with id 6abca2e0a2fdb5c424e0581c3c99d01a71492f868b5301f9b328f16558e3acb4
Jul 17 07:12:58.324426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:58.324397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:58.656531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:58.656489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l" event={"ID":"a9e5dd25-9460-45cb-b6cf-e656df275567","Type":"ContainerStarted","Data":"6abca2e0a2fdb5c424e0581c3c99d01a71492f868b5301f9b328f16558e3acb4"}
Jul 17 07:12:59.774667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:59.774109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:59.874129 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:59.874102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:12:59.907376 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:12:59.906018    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:12:59.907376 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:12:59.906985    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4873971c_8d3e_4203_81b5_c944af17cc32.slice/crio-9b02560ef52d037d1e3d962f366fb551294977a8e37a70002474e542193d8eac WatchSource:0}: Error finding container 9b02560ef52d037d1e3d962f366fb551294977a8e37a70002474e542193d8eac: Status 404 returned error can't find the container with id 9b02560ef52d037d1e3d962f366fb551294977a8e37a70002474e542193d8eac
Jul 17 07:13:00.668169 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.667695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hktn8" event={"ID":"8ba69a4b-39b3-4d83-83be-47497ce7d246","Type":"ContainerStarted","Data":"734d13462e941eca1a5f05bec15761530b446aeaa28a3451f0441f581b1d4b95"}
Jul 17 07:13:00.671075 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.671019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29" event={"ID":"1ee8146e-e88b-4a21-9b1b-f35fcd6b9d47","Type":"ContainerStarted","Data":"7ce5fd7ed49be426b2b34bbead5b74da9ad6977d5d07c6ba6f9d67c6977e36ea"}
Jul 17 07:13:00.673113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.672982    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc" event={"ID":"fdc295c0-e400-4da9-91fe-b23573af849e","Type":"ContainerStarted","Data":"3b9b0a9ab7dac94aa9d51c86f980e47b3b930be6daacca4990ae65192abe902f"}
Jul 17 07:13:00.674879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.674822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7848ffbf-5h256" event={"ID":"4873971c-8d3e-4203-81b5-c944af17cc32","Type":"ContainerStarted","Data":"9b02560ef52d037d1e3d962f366fb551294977a8e37a70002474e542193d8eac"}
Jul 17 07:13:00.677372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.676916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4" event={"ID":"913075d4-3368-4904-b60b-d02f3c27a948","Type":"ContainerStarted","Data":"a6622bdb09caf6be92925eaa559272bb936e2df291aeb101e80a55c31af887e9"}
Jul 17 07:13:00.677619 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.677572    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:13:00.679514 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.679486    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4"
Jul 17 07:13:00.685731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.685686    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hktn8" podStartSLOduration=67.634315325 podStartE2EDuration="1m15.685668108s" totalImagesPullingTime="8.051352783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 07:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:00.685247331 +0000 UTC m=+76.938670175" watchObservedRunningTime="2026-07-17 07:13:00.685668108 +0000 UTC m=+76.939090920"
Jul 17 07:13:00.703539 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.703431    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gm8nc" podStartSLOduration=2.247725649 podStartE2EDuration="9.703411113s" totalImagesPullingTime="7.455685464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:00.70144855 +0000 UTC m=+76.954871389" watchObservedRunningTime="2026-07-17 07:13:00.703411113 +0000 UTC m=+76.956833929"
Jul 17 07:13:00.720064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.720010    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6fd86468-qds29" podStartSLOduration=2.225517971 podStartE2EDuration="9.719990249s" totalImagesPullingTime="7.494472278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:00.719493347 +0000 UTC m=+76.972916160" watchObservedRunningTime="2026-07-17 07:13:00.719990249 +0000 UTC m=+76.973413066"
Jul 17 07:13:00.738906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.738698    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b9f447c-lsnt4" podStartSLOduration=2.285412508 podStartE2EDuration="9.738681095s" totalImagesPullingTime="7.453268587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:00.738458435 +0000 UTC m=+76.991881248" watchObservedRunningTime="2026-07-17 07:13:00.738681095 +0000 UTC m=+76.992103908"
Jul 17 07:13:00.838582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.838545    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:00.938732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:00.938578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:01.685688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:01.685224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l" event={"ID":"a9e5dd25-9460-45cb-b6cf-e656df275567","Type":"ContainerStarted","Data":"ea8c4feef2747d109db00df1716c241d3f1ee5731a3b0b1cbb1250b1e3c11a44"}
Jul 17 07:13:01.686317 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:01.686289    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:13:01.693249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:01.693220    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l"
Jul 17 07:13:01.708379 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:01.708040    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-g688l" podStartSLOduration=6.116362752 podStartE2EDuration="8.708018285s" totalImagesPullingTime="2.591655533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:01.706653641 +0000 UTC m=+77.960076477" watchObservedRunningTime="2026-07-17 07:13:01.708018285 +0000 UTC m=+77.961441097"
Jul 17 07:13:01.762047 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:01.762013    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:13:02.603227 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:02.603067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:02.710370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:02.710250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:02.813523 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:02.813471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:03.696843 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:03.696802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7848ffbf-5h256" event={"ID":"4873971c-8d3e-4203-81b5-c944af17cc32","Type":"ContainerStarted","Data":"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"}
Jul 17 07:13:03.722428 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:03.722364    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7848ffbf-5h256" podStartSLOduration=3.371939345 podStartE2EDuration="6.722325475s" totalImagesPullingTime="3.35038613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:03.721082656 +0000 UTC m=+79.974505511" watchObservedRunningTime="2026-07-17 07:13:03.722325475 +0000 UTC m=+79.975748289"
Jul 17 07:13:06.651732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:06.651600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:06.751963 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:06.751918    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:06.852839 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:06.852799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:06.914917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:06.914815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:07.733569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:07.733533    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:07.734095 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:07.733600    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:07.739480 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:07.739451    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:08.729880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:08.729850    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:08.885757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:08.885719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:08.986731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:08.986644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:09.676869 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:09.676831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:09.736374 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:09.736317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:11.441043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:11.441004    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:12.639330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:12.639298    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:12.743648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:12.743564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-9nd6t" event={"ID":"9c6a280d-1dc5-491c-b5ec-77ccd0af9f48","Type":"ContainerStarted","Data":"f7c432ee49b7352641693ed2576c9bba2e88b764a75b5c0cc2215ab6d7123a0a"}
Jul 17 07:13:12.743820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:12.743774    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:13:12.745473 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:12.745449    2584 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-9nd6t container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/\": dial tcp 10.133.0.17:8080: connect: connection refused" start-of-body=
Jul 17 07:13:12.745583 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:12.745492    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-9nd6t" podUID="9c6a280d-1dc5-491c-b5ec-77ccd0af9f48" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.17:8080/\": dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 17 07:13:13.624371 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.624320    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666fdc4999-9prdc"
Jul 17 07:13:13.647541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.647476    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-9nd6t" podStartSLOduration=2.5507198989999997 podStartE2EDuration="22.647454225s" totalImagesPullingTime="20.096734326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:12.768125981 +0000 UTC m=+89.021548806" watchObservedRunningTime="2026-07-17 07:13:13.647454225 +0000 UTC m=+89.900877037"
Jul 17 07:13:13.760245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.760211    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-9nd6t"
Jul 17 07:13:13.769279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.769244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:13.869825 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.869795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:13.971310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:13.971230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.121607 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.121564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.244796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.244715    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:13:14.415202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.415159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.514844 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.514761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.591706 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.591666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.874438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.874395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:14.974705 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:14.974656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:16.775656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:16.775617    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" podUID="49ec566f-672e-436f-baab-a1bc01315dae" containerName="registry" containerID="cri-o://dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86" gracePeriod=30
Jul 17 07:13:17.073252 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.073223    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:13:17.184310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184332    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184380    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184554    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjsv4\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184588    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184767 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184655    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184767 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184699    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca\") pod \"49ec566f-672e-436f-baab-a1bc01315dae\" (UID: \"49ec566f-672e-436f-baab-a1bc01315dae\") "
Jul 17 07:13:17.184902 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.184876    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:13:17.185270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.185229    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:13:17.187622 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.187593    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:13:17.187753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.187616    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:13:17.189587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.189550    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:13:17.189587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.189562    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:13:17.191319 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.191290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4" (OuterVolumeSpecName: "kube-api-access-gjsv4") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "kube-api-access-gjsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:13:17.196564 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.196534    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "49ec566f-672e-436f-baab-a1bc01315dae" (UID: "49ec566f-672e-436f-baab-a1bc01315dae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 07:13:17.285907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285871    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-bound-sa-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.285907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285905    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-trusted-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.285907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285914    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-registry-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.286155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285926    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-image-registry-private-configuration\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.286155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285942    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49ec566f-672e-436f-baab-a1bc01315dae-installation-pull-secrets\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.286155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285955    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49ec566f-672e-436f-baab-a1bc01315dae-registry-certificates\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.286155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285963    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjsv4\" (UniqueName: \"kubernetes.io/projected/49ec566f-672e-436f-baab-a1bc01315dae-kube-api-access-gjsv4\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.286155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.285972    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49ec566f-672e-436f-baab-a1bc01315dae-ca-trust-extracted\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:17.765800 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.765715    2584 generic.go:356] "Generic (PLEG): container finished" podID="49ec566f-672e-436f-baab-a1bc01315dae" containerID="dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86" exitCode=0
Jul 17 07:13:17.765800 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.765763    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d976689-zstdt"
Jul 17 07:13:17.766010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.765808    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" event={"ID":"49ec566f-672e-436f-baab-a1bc01315dae","Type":"ContainerDied","Data":"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"}
Jul 17 07:13:17.766010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.765872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d976689-zstdt" event={"ID":"49ec566f-672e-436f-baab-a1bc01315dae","Type":"ContainerDied","Data":"73b501b2b1b9a87eaa1e5bada478e1e3e528594156358533f25dd55e30e2cd2e"}
Jul 17 07:13:17.766010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.765900    2584 scope.go:117] "RemoveContainer" containerID="dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"
Jul 17 07:13:17.776033 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.776016    2584 scope.go:117] "RemoveContainer" containerID="dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"
Jul 17 07:13:17.776375 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:13:17.776297    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86\": container with ID starting with dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86 not found: ID does not exist" containerID="dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"
Jul 17 07:13:17.776375 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.776323    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86"} err="failed to get container status \"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86\": rpc error: code = NotFound desc = could not find container \"dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86\": container with ID starting with dd139d42b14fc8fb3de9e45183b34a0e618941ea543731a81c00fb3d14998a86 not found: ID does not exist"
Jul 17 07:13:17.788820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.788794    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:13:17.793648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:17.793624    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d5d976689-zstdt"]
Jul 17 07:13:18.129774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.129731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.239944 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.239905    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.296813 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.296657    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ec566f-672e-436f-baab-a1bc01315dae" path="/var/lib/kubelet/pods/49ec566f-672e-436f-baab-a1bc01315dae/volumes"
Jul 17 07:13:18.338405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.338370    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.360426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.360395    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qbsqc"]
Jul 17 07:13:18.360774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.360754    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ec566f-672e-436f-baab-a1bc01315dae" containerName="registry"
Jul 17 07:13:18.360850 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.360777    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ec566f-672e-436f-baab-a1bc01315dae" containerName="registry"
Jul 17 07:13:18.360902 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.360856    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ec566f-672e-436f-baab-a1bc01315dae" containerName="registry"
Jul 17 07:13:18.395687 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.394854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.398934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.398694    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 07:13:18.398934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.398717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 07:13:18.398934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.398752    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 07:13:18.398934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.398889    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 07:13:18.399211 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.399128    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 07:13:18.399263 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.399220    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 07:13:18.399748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.399726    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g952c\""
Jul 17 07:13:18.464620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.464499    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.496827    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-root\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.496872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-tls\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.496898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-textfile\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.496933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.496990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-metrics-client-ca\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.497059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-wtmp\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.497102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.497132    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-sys\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.497198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.497163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpfr\" (UniqueName: \"kubernetes.io/projected/450a10a1-b8ba-413d-bbcf-23cad9bc2461-kube-api-access-mrpfr\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.551492 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.551454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-metrics-client-ca\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-wtmp\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598166    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-sys\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpfr\" (UniqueName: \"kubernetes.io/projected/450a10a1-b8ba-413d-bbcf-23cad9bc2461-kube-api-access-mrpfr\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-root\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-tls\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-textfile\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.598650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-wtmp\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.599238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598705    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-root\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.599238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598742    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-metrics-client-ca\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.599238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.598884    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.599238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.599032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-textfile\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.599238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.599098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450a10a1-b8ba-413d-bbcf-23cad9bc2461-sys\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.601696 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.601635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.601818 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.601765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/450a10a1-b8ba-413d-bbcf-23cad9bc2461-node-exporter-tls\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.606853 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.606828    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpfr\" (UniqueName: \"kubernetes.io/projected/450a10a1-b8ba-413d-bbcf-23cad9bc2461-kube-api-access-mrpfr\") pod \"node-exporter-qbsqc\" (UID: \"450a10a1-b8ba-413d-bbcf-23cad9bc2461\") " pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.649934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.649856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:18.708074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.708034    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbsqc"
Jul 17 07:13:18.743770 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:13:18.742706    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450a10a1_b8ba_413d_bbcf_23cad9bc2461.slice/crio-5b155ce294f874fa3813d5638f0772b5f23d1763018a9c58d7185422975c4b0e WatchSource:0}: Error finding container 5b155ce294f874fa3813d5638f0772b5f23d1763018a9c58d7185422975c4b0e: Status 404 returned error can't find the container with id 5b155ce294f874fa3813d5638f0772b5f23d1763018a9c58d7185422975c4b0e
Jul 17 07:13:18.773133 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:18.773097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbsqc" event={"ID":"450a10a1-b8ba-413d-bbcf-23cad9bc2461","Type":"ContainerStarted","Data":"5b155ce294f874fa3813d5638f0772b5f23d1763018a9c58d7185422975c4b0e"}
Jul 17 07:13:19.159579 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:19.159539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:19.258861 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:19.258825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:19.356452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:19.356413    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:19.465791 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:19.465512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:19.668156 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:19.668120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:20.167150 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:20.167115    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:20.251500 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:20.251461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:20.784753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:20.784711    2584 generic.go:356] "Generic (PLEG): container finished" podID="450a10a1-b8ba-413d-bbcf-23cad9bc2461" containerID="ea004ce336c8b7148f14c9362766fa69e5b36a6742742d0ebc8d63678e07759f" exitCode=0
Jul 17 07:13:20.784927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:20.784770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbsqc" event={"ID":"450a10a1-b8ba-413d-bbcf-23cad9bc2461","Type":"ContainerDied","Data":"ea004ce336c8b7148f14c9362766fa69e5b36a6742742d0ebc8d63678e07759f"}
Jul 17 07:13:21.202212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.202129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:21.300305 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.300222    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:21.541708 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.541664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:21.676603 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.676552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:21.794604 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.794508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbsqc" event={"ID":"450a10a1-b8ba-413d-bbcf-23cad9bc2461","Type":"ContainerStarted","Data":"fa43a6b63e28d863d172bea3052b76602b3fe247b5a709b70bcc3a66218301a3"}
Jul 17 07:13:21.794744 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:21.794610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbsqc" event={"ID":"450a10a1-b8ba-413d-bbcf-23cad9bc2461","Type":"ContainerStarted","Data":"d876f5fb227c2c6df3bf2f0c866fde32fdf93a5a7f107b31af69d96cb6327bb9"}
Jul 17 07:13:23.051420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.051382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:23.135469 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.135423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:23.234126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.234091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:23.335298 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.335224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:23.393226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.393189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:23.621877 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.621800    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sv54n"
Jul 17 07:13:23.639606 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:23.639547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qbsqc" podStartSLOduration=3.843598687 podStartE2EDuration="5.63952974s" totalImagesPullingTime="1.686898944s" totalInitContainerRuntime="109.032109ms" isStatefulPod=true podCreationTimestamp="2026-07-17 07:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:21.813634171 +0000 UTC m=+98.067056986" watchObservedRunningTime="2026-07-17 07:13:23.63952974 +0000 UTC m=+99.892952551"
Jul 17 07:13:24.570969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:24.570934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:29.867717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:29.867672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:29.935085 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:29.935044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:30.571577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:30.571537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:30.671658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:30.671629    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:32.382310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.382275    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:13:32.435324 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.435291    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:13:32.435483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.435412    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.443867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.443844    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 07:13:32.516652 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516611    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql82q\" (UniqueName: \"kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516808    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.516972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.516840    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.617960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.617925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.617960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.617962    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.617988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql82q\" (UniqueName: \"kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.618916 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618885    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.619065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.619065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.619065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.618967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.620647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.620625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.620763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.620663    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.628812 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.628783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql82q\" (UniqueName: \"kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q\") pod \"console-5b889cc589-t6ktx\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") " pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.746142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.746051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:32.897552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:32.897528    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:13:32.899321 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:13:32.899293    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba96f42_bd79_4b38_bd90_19088f14e15e.slice/crio-7be0a345617fefe6a344e19187d15c6bb5f389c7ef6fe8426ad07ffaf74a3a44 WatchSource:0}: Error finding container 7be0a345617fefe6a344e19187d15c6bb5f389c7ef6fe8426ad07ffaf74a3a44: Status 404 returned error can't find the container with id 7be0a345617fefe6a344e19187d15c6bb5f389c7ef6fe8426ad07ffaf74a3a44
Jul 17 07:13:33.844094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:33.844055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b889cc589-t6ktx" event={"ID":"5ba96f42-bd79-4b38-bd90-19088f14e15e","Type":"ContainerStarted","Data":"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"}
Jul 17 07:13:33.844544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:33.844114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b889cc589-t6ktx" event={"ID":"5ba96f42-bd79-4b38-bd90-19088f14e15e","Type":"ContainerStarted","Data":"7be0a345617fefe6a344e19187d15c6bb5f389c7ef6fe8426ad07ffaf74a3a44"}
Jul 17 07:13:33.887289 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:33.887242    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b889cc589-t6ktx" podStartSLOduration=1.8872265110000002 podStartE2EDuration="1.887226511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:13:33.886664898 +0000 UTC m=+110.140087734" watchObservedRunningTime="2026-07-17 07:13:33.887226511 +0000 UTC m=+110.140649322"
Jul 17 07:13:37.156741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:37.156688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:37.257560 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:37.257524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:37.319956 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:37.319928    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:39.271735 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.271700    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d7848ffbf-5h256" podUID="4873971c-8d3e-4203-81b5-c944af17cc32" containerName="console" containerID="cri-o://1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d" gracePeriod=15
Jul 17 07:13:39.561857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.561824    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:39.674674 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674637    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.674674 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674678    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.674903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674810    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkrw4\" (UniqueName: \"kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.674903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674852    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.674903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674882    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.675054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.674916    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca\") pod \"4873971c-8d3e-4203-81b5-c944af17cc32\" (UID: \"4873971c-8d3e-4203-81b5-c944af17cc32\") "
Jul 17 07:13:39.675159 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.675121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config" (OuterVolumeSpecName: "console-config") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:13:39.675443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.675221    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:13:39.675530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.675462    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca" (OuterVolumeSpecName: "service-ca") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:13:39.677122 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.677092    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4" (OuterVolumeSpecName: "kube-api-access-pkrw4") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "kube-api-access-pkrw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:13:39.677122 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.677109    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:13:39.677246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.677152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4873971c-8d3e-4203-81b5-c944af17cc32" (UID: "4873971c-8d3e-4203-81b5-c944af17cc32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775716    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-oauth-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775747    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-oauth-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775758    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775766    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4873971c-8d3e-4203-81b5-c944af17cc32-console-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775776    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4873971c-8d3e-4203-81b5-c944af17cc32-console-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.775804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.775784    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrw4\" (UniqueName: \"kubernetes.io/projected/4873971c-8d3e-4203-81b5-c944af17cc32-kube-api-access-pkrw4\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:13:39.867042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.867007    2584 generic.go:356] "Generic (PLEG): container finished" podID="4873971c-8d3e-4203-81b5-c944af17cc32" containerID="1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d" exitCode=0
Jul 17 07:13:39.867239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.867057    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7848ffbf-5h256"
Jul 17 07:13:39.867239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.867100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7848ffbf-5h256" event={"ID":"4873971c-8d3e-4203-81b5-c944af17cc32","Type":"ContainerDied","Data":"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"}
Jul 17 07:13:39.867239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.867151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7848ffbf-5h256" event={"ID":"4873971c-8d3e-4203-81b5-c944af17cc32","Type":"ContainerDied","Data":"9b02560ef52d037d1e3d962f366fb551294977a8e37a70002474e542193d8eac"}
Jul 17 07:13:39.867239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.867173    2584 scope.go:117] "RemoveContainer" containerID="1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"
Jul 17 07:13:39.878156 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.878136    2584 scope.go:117] "RemoveContainer" containerID="1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"
Jul 17 07:13:39.878427 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:13:39.878403    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d\": container with ID starting with 1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d not found: ID does not exist" containerID="1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"
Jul 17 07:13:39.878517 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.878435    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d"} err="failed to get container status \"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d\": rpc error: code = NotFound desc = could not find container \"1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d\": container with ID starting with 1c9d717302e8240a44c9633d7b1c768178dfcf37a5f89683f781685e91ea8a4d not found: ID does not exist"
Jul 17 07:13:39.890615 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.890580    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:13:39.894277 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:39.894257    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d7848ffbf-5h256"]
Jul 17 07:13:40.289684 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:40.289654    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4873971c-8d3e-4203-81b5-c944af17cc32" path="/var/lib/kubelet/pods/4873971c-8d3e-4203-81b5-c944af17cc32/volumes"
Jul 17 07:13:42.746689 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:42.746647    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:42.747104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:42.746708    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:42.751318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:42.751296    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:42.882147 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:42.882121    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:13:43.050732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:43.050688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:43.154167 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:43.154133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:43.219740 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:43.219700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:44.879165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:44.879120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:44.946984 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:44.946948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:13:45.889769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:45.889734    2584 generic.go:356] "Generic (PLEG): container finished" podID="283e90a3-f3ea-484f-88ce-effe90d837e6" containerID="d6c38cc6f7d5ce7c194a02e1a17edd2a0a456b501458269c155a89b11da348ed" exitCode=0
Jul 17 07:13:45.890124 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:45.889801    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd" event={"ID":"283e90a3-f3ea-484f-88ce-effe90d837e6","Type":"ContainerDied","Data":"d6c38cc6f7d5ce7c194a02e1a17edd2a0a456b501458269c155a89b11da348ed"}
Jul 17 07:13:45.890208 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:45.890188    2584 scope.go:117] "RemoveContainer" containerID="d6c38cc6f7d5ce7c194a02e1a17edd2a0a456b501458269c155a89b11da348ed"
Jul 17 07:13:46.895089 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:46.895053    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-75kbd" event={"ID":"283e90a3-f3ea-484f-88ce-effe90d837e6","Type":"ContainerStarted","Data":"b987b20313ae53c67b43c7f67c7cba3235547eb5e48a1211a07eaebfa0fd9629"}
Jul 17 07:13:53.272927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:13:53.272891    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:15.812304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:15.812246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:15.879048 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:15.879017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:25.236827 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:25.236782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:25.296709 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:25.296674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:34.358046 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:34.358010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:34.421701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:34.421675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:41.682790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.682706    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:14:41.683147 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.683055    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4873971c-8d3e-4203-81b5-c944af17cc32" containerName="console"
Jul 17 07:14:41.683147 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.683070    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4873971c-8d3e-4203-81b5-c944af17cc32" containerName="console"
Jul 17 07:14:41.683147 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.683123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4873971c-8d3e-4203-81b5-c944af17cc32" containerName="console"
Jul 17 07:14:41.685969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.685944    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.696464 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.696443    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:14:41.737660 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737631    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.737804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.737804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737695    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.737924 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdxf\" (UniqueName: \"kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.737924 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.737924 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.738045 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.737931    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.765176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.765146    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:41.838948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.838911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.838948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.838954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.838975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdxf\" (UniqueName: \"kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839877 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839855    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839957 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.839994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.839934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.841624 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.841606    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.841679 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.841644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.847038 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.847015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdxf\" (UniqueName: \"kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf\") pod \"console-dc545f6f6-2cpf6\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") " pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:41.865576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.865546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:41.995450 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:41.995363    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:42.126655 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:42.126627    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:14:42.129226 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:14:42.129202    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494f8e9a_2d4e_4131_b940_cdfa0d817168.slice/crio-e065d762038d6fdf49404265bc999f8f742086d64a4479571ae1ebb4ca201d00 WatchSource:0}: Error finding container e065d762038d6fdf49404265bc999f8f742086d64a4479571ae1ebb4ca201d00: Status 404 returned error can't find the container with id e065d762038d6fdf49404265bc999f8f742086d64a4479571ae1ebb4ca201d00
Jul 17 07:14:43.092614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:43.092579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545f6f6-2cpf6" event={"ID":"494f8e9a-2d4e-4131-b940-cdfa0d817168","Type":"ContainerStarted","Data":"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"}
Jul 17 07:14:43.092996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:43.092630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545f6f6-2cpf6" event={"ID":"494f8e9a-2d4e-4131-b940-cdfa0d817168","Type":"ContainerStarted","Data":"e065d762038d6fdf49404265bc999f8f742086d64a4479571ae1ebb4ca201d00"}
Jul 17 07:14:43.112949 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:43.112897    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc545f6f6-2cpf6" podStartSLOduration=2.112881816 podStartE2EDuration="2.112881816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:14:43.111777544 +0000 UTC m=+179.365200357" watchObservedRunningTime="2026-07-17 07:14:43.112881816 +0000 UTC m=+179.366304628"
Jul 17 07:14:46.248779 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:46.248739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:46.309087 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:46.309048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:51.995546 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:51.995505    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:51.995978 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:51.995565    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:52.000191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.000168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:52.127424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.127395    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:14:52.182994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.182961    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:14:52.300833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.300798    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:52.357396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.357331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:52.419613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:52.419565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:59.878146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:59.878111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:14:59.992267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:14:59.992233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:15.307473 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:15.307432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:15.405557 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:15.405513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:17.203685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.203646    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b889cc589-t6ktx" podUID="5ba96f42-bd79-4b38-bd90-19088f14e15e" containerName="console" containerID="cri-o://1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add" gracePeriod=15
Jul 17 07:15:17.439893 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.439870    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:15:17.533796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533718    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.533796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533762    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.533988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533806    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.533988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533828    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.533988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533863    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.534134 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.533980    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.534134 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.534037    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql82q\" (UniqueName: \"kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q\") pod \"5ba96f42-bd79-4b38-bd90-19088f14e15e\" (UID: \"5ba96f42-bd79-4b38-bd90-19088f14e15e\") "
Jul 17 07:15:17.534244 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.534224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca" (OuterVolumeSpecName: "service-ca") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:15:17.534303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.534266    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:15:17.534303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.534282    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:15:17.534439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.534358    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config" (OuterVolumeSpecName: "console-config") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:15:17.536138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.536112    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q" (OuterVolumeSpecName: "kube-api-access-ql82q") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "kube-api-access-ql82q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:15:17.536466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.536438    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:15:17.536466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.536451    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5ba96f42-bd79-4b38-bd90-19088f14e15e" (UID: "5ba96f42-bd79-4b38-bd90-19088f14e15e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:15:17.634642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634611    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql82q\" (UniqueName: \"kubernetes.io/projected/5ba96f42-bd79-4b38-bd90-19088f14e15e-kube-api-access-ql82q\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634636    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634647    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-oauth-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634656    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634665    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-trusted-ca-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634674    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-oauth-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:17.634848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:17.634682    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ba96f42-bd79-4b38-bd90-19088f14e15e-console-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:15:18.216286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.216252    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ba96f42-bd79-4b38-bd90-19088f14e15e" containerID="1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add" exitCode=0
Jul 17 07:15:18.216681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.216307    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b889cc589-t6ktx"
Jul 17 07:15:18.216681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.216353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b889cc589-t6ktx" event={"ID":"5ba96f42-bd79-4b38-bd90-19088f14e15e","Type":"ContainerDied","Data":"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"}
Jul 17 07:15:18.216681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.216410    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b889cc589-t6ktx" event={"ID":"5ba96f42-bd79-4b38-bd90-19088f14e15e","Type":"ContainerDied","Data":"7be0a345617fefe6a344e19187d15c6bb5f389c7ef6fe8426ad07ffaf74a3a44"}
Jul 17 07:15:18.216681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.216438    2584 scope.go:117] "RemoveContainer" containerID="1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"
Jul 17 07:15:18.225037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.225020    2584 scope.go:117] "RemoveContainer" containerID="1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"
Jul 17 07:15:18.225314 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:15:18.225295    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add\": container with ID starting with 1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add not found: ID does not exist" containerID="1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"
Jul 17 07:15:18.225443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.225328    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add"} err="failed to get container status \"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add\": rpc error: code = NotFound desc = could not find container \"1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add\": container with ID starting with 1bc5726664d365b557727de8bc6215dceaaeec35e67b1f219e9ef0bf53502add not found: ID does not exist"
Jul 17 07:15:18.237852 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.237832    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:15:18.241276 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.241256    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b889cc589-t6ktx"]
Jul 17 07:15:18.289849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:18.289823    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba96f42-bd79-4b38-bd90-19088f14e15e" path="/var/lib/kubelet/pods/5ba96f42-bd79-4b38-bd90-19088f14e15e/volumes"
Jul 17 07:15:22.053321 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:22.053282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:42.226145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:42.226106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:42.581602 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:42.581565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:45.830686 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:45.830651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:45.935073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:45.935040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:47.599816 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:47.599786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:47.706775 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:47.706746    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:48.061418 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:48.061379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:48.177999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:48.177966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:50.399617 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:50.399553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:50.605760 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:50.605717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:52.630906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:52.630868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:52.696180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:52.696144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:58.011212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.011169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:58.311699 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.311666    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5985m"]
Jul 17 07:15:58.311949 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.311936    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba96f42-bd79-4b38-bd90-19088f14e15e" containerName="console"
Jul 17 07:15:58.312003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.311951    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba96f42-bd79-4b38-bd90-19088f14e15e" containerName="console"
Jul 17 07:15:58.312043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.312017    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba96f42-bd79-4b38-bd90-19088f14e15e" containerName="console"
Jul 17 07:15:58.315990 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.315973    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.318832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.318807    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 07:15:58.318945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.318834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 07:15:58.320676 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.320661    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hqdm2\""
Jul 17 07:15:58.331348 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.331312    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5985m"]
Jul 17 07:15:58.438328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.438293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-bound-sa-token\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.438528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.438332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dlw\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-kube-api-access-b5dlw\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.539143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.539104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-bound-sa-token\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.539143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.539146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dlw\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-kube-api-access-b5dlw\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.548440 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.548410    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dlw\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-kube-api-access-b5dlw\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.548580 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.548483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a94e571-2b87-411f-9c62-3a589fa8ccda-bound-sa-token\") pod \"cert-manager-55b4d954d5-5985m\" (UID: \"9a94e571-2b87-411f-9c62-3a589fa8ccda\") " pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.624840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.624762    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5985m"
Jul 17 07:15:58.651892 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.651860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:58.750576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.750546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5985m"]
Jul 17 07:15:58.750894 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:58.750832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:15:58.753052 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:15:58.753022    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a94e571_2b87_411f_9c62_3a589fa8ccda.slice/crio-2baea867d9ef6c67471be033faf013317f55c2849395a57ff2d8d9fae685808b WatchSource:0}: Error finding container 2baea867d9ef6c67471be033faf013317f55c2849395a57ff2d8d9fae685808b: Status 404 returned error can't find the container with id 2baea867d9ef6c67471be033faf013317f55c2849395a57ff2d8d9fae685808b
Jul 17 07:15:59.369835 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:15:59.369796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5985m" event={"ID":"9a94e571-2b87-411f-9c62-3a589fa8ccda","Type":"ContainerStarted","Data":"2baea867d9ef6c67471be033faf013317f55c2849395a57ff2d8d9fae685808b"}
Jul 17 07:16:02.382988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:02.382953    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5985m" event={"ID":"9a94e571-2b87-411f-9c62-3a589fa8ccda","Type":"ContainerStarted","Data":"969b385cc7b2dbc2d546c52241d0f9089dd652189ccfb4657a6c5bbb6ac47a34"}
Jul 17 07:16:02.406700 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:02.406648    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5985m" podStartSLOduration=1.443629874 podStartE2EDuration="4.40663425s" totalImagesPullingTime="2.963004376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:16:02.405282082 +0000 UTC m=+258.658704897" watchObservedRunningTime="2026-07-17 07:16:02.40663425 +0000 UTC m=+258.660057061"
Jul 17 07:16:04.734564 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:04.734524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:04.835637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:04.835606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:07.041960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:07.041852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:10.110376 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:10.110323    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:10.182616 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:10.182576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:21.289231 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:21.289187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:21.407978 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:21.407948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:21.678967 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:21.678881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:21.909056 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:21.909021    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.015905 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.015819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.076897 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.076869    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"]
Jul 17 07:16:22.082484 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.082466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.085981 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.085954    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 07:16:22.086088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.086056    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 07:16:22.086088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.086074    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 07:16:22.086088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.086083    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 07:16:22.088364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.088331    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cjtgt\""
Jul 17 07:16:22.115536 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.115507    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"]
Jul 17 07:16:22.237015 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.236986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-webhook-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.237015 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.237017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmfg\" (UniqueName: \"kubernetes.io/projected/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-kube-api-access-8cmfg\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.237198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.237094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-tmp\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.237198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.237156    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-manifests\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.237198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.237184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-webhook-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmfg\" (UniqueName: \"kubernetes.io/projected/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-kube-api-access-8cmfg\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338597 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338316    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-tmp\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338597 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-manifests\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338597 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338459    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.338751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.338734    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-manifests\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.340534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.340512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-tmp\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.340662 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.340649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-webhook-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.340810 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.340791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.350871 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.350850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmfg\" (UniqueName: \"kubernetes.io/projected/fb05d8cf-34b6-4b5b-8be8-d19d288b3802-kube-api-access-8cmfg\") pod \"opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2\" (UID: \"fb05d8cf-34b6-4b5b-8be8-d19d288b3802\") " pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.392229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.392204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:22.404132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.404101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.506520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.506490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.538745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.538713    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"]
Jul 17 07:16:22.542852 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:16:22.542827    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb05d8cf_34b6_4b5b_8be8_d19d288b3802.slice/crio-7d460726a05410162ec2c399554044d402c80001f754b68f8e4f89184acc0d1a WatchSource:0}: Error finding container 7d460726a05410162ec2c399554044d402c80001f754b68f8e4f89184acc0d1a: Status 404 returned error can't find the container with id 7d460726a05410162ec2c399554044d402c80001f754b68f8e4f89184acc0d1a
Jul 17 07:16:22.607459 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.607388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.695003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.694974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.796378 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.796332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:22.900774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:22.900691    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:23.462739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:23.462686    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2" event={"ID":"fb05d8cf-34b6-4b5b-8be8-d19d288b3802","Type":"ContainerStarted","Data":"7d460726a05410162ec2c399554044d402c80001f754b68f8e4f89184acc0d1a"}
Jul 17 07:16:26.477070 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:26.477030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2" event={"ID":"fb05d8cf-34b6-4b5b-8be8-d19d288b3802","Type":"ContainerStarted","Data":"5056c3c01e008c42f3ea8b9b7c0e27ce93d3ba133ec75ae90a744b471c943cd0"}
Jul 17 07:16:27.279920 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:27.279836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:27.390054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:27.390016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:27.487186 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:27.487148    2584 generic.go:356] "Generic (PLEG): container finished" podID="fb05d8cf-34b6-4b5b-8be8-d19d288b3802" containerID="5056c3c01e008c42f3ea8b9b7c0e27ce93d3ba133ec75ae90a744b471c943cd0" exitCode=0
Jul 17 07:16:27.487573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:27.487191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2" event={"ID":"fb05d8cf-34b6-4b5b-8be8-d19d288b3802","Type":"ContainerDied","Data":"5056c3c01e008c42f3ea8b9b7c0e27ce93d3ba133ec75ae90a744b471c943cd0"}
Jul 17 07:16:28.492794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:28.492750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2" event={"ID":"fb05d8cf-34b6-4b5b-8be8-d19d288b3802","Type":"ContainerStarted","Data":"ba379b91d5dbf4ebc23bcf3095e764a6e18debf89795b36b69cf9b4895f45594"}
Jul 17 07:16:28.493179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:28.492888    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:28.523025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:28.522980    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2" podStartSLOduration=1.797381987 podStartE2EDuration="6.522965444s" totalImagesPullingTime="3.699406175s" totalInitContainerRuntime="1.026177282s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:16:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:16:28.521166043 +0000 UTC m=+284.774588865" watchObservedRunningTime="2026-07-17 07:16:28.522965444 +0000 UTC m=+284.776388255"
Jul 17 07:16:39.499091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.499061    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2"
Jul 17 07:16:39.762830 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.762755    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"]
Jul 17 07:16:39.765930 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.765911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.768513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.768494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 07:16:39.768624 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.768603    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 07:16:39.768688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.768653    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 07:16:39.768902 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.768880    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ccphb\""
Jul 17 07:16:39.768937 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.768917    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 07:16:39.769476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.769461    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 07:16:39.777568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.777547    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"]
Jul 17 07:16:39.879949 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.879915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvppb\" (UniqueName: \"kubernetes.io/projected/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-kube-api-access-bvppb\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.880121 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.879976    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.880121 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.880011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-manager-config\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.880121 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.880037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-metrics-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.981180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.981154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.981279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.981188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-manager-config\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.981279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.981214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-metrics-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.981279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.981240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvppb\" (UniqueName: \"kubernetes.io/projected/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-kube-api-access-bvppb\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.981961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.981940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-manager-config\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.983498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.983478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.983664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.983644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-metrics-cert\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:39.991219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:39.991198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvppb\" (UniqueName: \"kubernetes.io/projected/4fc6871d-ac6d-4ee5-a8af-e8616b62d536-kube-api-access-bvppb\") pod \"lws-controller-manager-6fc85c6965-n8s6t\" (UID: \"4fc6871d-ac6d-4ee5-a8af-e8616b62d536\") " pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:40.075567 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:40.075534    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:40.207988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:40.207955    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"]
Jul 17 07:16:40.210851 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:16:40.210822    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc6871d_ac6d_4ee5_a8af_e8616b62d536.slice/crio-9e28e1a4f07b01de9b398881adf48307fd0b1346d367a73b3b46d37ae4d51f2d WatchSource:0}: Error finding container 9e28e1a4f07b01de9b398881adf48307fd0b1346d367a73b3b46d37ae4d51f2d: Status 404 returned error can't find the container with id 9e28e1a4f07b01de9b398881adf48307fd0b1346d367a73b3b46d37ae4d51f2d
Jul 17 07:16:40.540256 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:40.540220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t" event={"ID":"4fc6871d-ac6d-4ee5-a8af-e8616b62d536","Type":"ContainerStarted","Data":"9e28e1a4f07b01de9b398881adf48307fd0b1346d367a73b3b46d37ae4d51f2d"}
Jul 17 07:16:41.799980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:41.799927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:43.551332 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:43.551293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:43.553493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:43.553462    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t" event={"ID":"4fc6871d-ac6d-4ee5-a8af-e8616b62d536","Type":"ContainerStarted","Data":"8f2472b2339f0d2b42824b7c68e7646e8ea009df465caa570e162f3500b0cc03"}
Jul 17 07:16:43.553613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:43.553552    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:43.571122 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:43.571061    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t" podStartSLOduration=2.242338615 podStartE2EDuration="4.571040357s" totalImagesPullingTime="2.328701742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:16:43.570552886 +0000 UTC m=+299.823975690" watchObservedRunningTime="2026-07-17 07:16:43.571040357 +0000 UTC m=+299.824463173"
Jul 17 07:16:43.651691 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:43.651656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:44.181054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:44.181027    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:16:44.188023 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:44.187989    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 07:16:44.191840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:44.191811    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:16:47.960031 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:47.959987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:48.281227 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:48.281143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:49.303049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:49.303012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:49.376548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:49.376512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:50.876729 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.876691    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"]
Jul 17 07:16:50.879725 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.879708    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.883050 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.883028    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 07:16:50.883740 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.883723    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 07:16:50.883854 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.883837    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lw6s6\""
Jul 17 07:16:50.883980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.883931    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 07:16:50.884436 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.884420    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 07:16:50.884545 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.884528    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 07:16:50.884698 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.884684    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 07:16:50.896271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.896247    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"]
Jul 17 07:16:50.939438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.939397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:50.975499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4v8g\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-kube-api-access-z4v8g\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b522f30-8d75-48da-b8ec-edf7f90e359b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975583    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975612    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975786 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975786 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:50.975786 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:50.975755    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076518 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076528    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4v8g\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-kube-api-access-z4v8g\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b522f30-8d75-48da-b8ec-edf7f90e359b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.076701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.076695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.077369 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.077321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.079043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.079015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.079043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.079022    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.079043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.079031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b522f30-8d75-48da-b8ec-edf7f90e359b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.079190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.079109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.085288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.085269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.087930 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.087911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4v8g\" (UniqueName: \"kubernetes.io/projected/8b522f30-8d75-48da-b8ec-edf7f90e359b-kube-api-access-z4v8g\") pod \"istiod-openshift-gateway-94bb8fbfd-6nrdp\" (UID: \"8b522f30-8d75-48da-b8ec-edf7f90e359b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.190036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.189949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:51.348593 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.348562    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"]
Jul 17 07:16:51.348718 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:16:51.348659    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b522f30_8d75_48da_b8ec_edf7f90e359b.slice/crio-b517a6c4da51978f96ec5308e414cb4b855bbb22209d1520d84dc9fbeb678473 WatchSource:0}: Error finding container b517a6c4da51978f96ec5308e414cb4b855bbb22209d1520d84dc9fbeb678473: Status 404 returned error can't find the container with id b517a6c4da51978f96ec5308e414cb4b855bbb22209d1520d84dc9fbeb678473
Jul 17 07:16:51.350432 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.350413    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:16:51.587831 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:51.587793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp" event={"ID":"8b522f30-8d75-48da-b8ec-edf7f90e359b","Type":"ContainerStarted","Data":"b517a6c4da51978f96ec5308e414cb4b855bbb22209d1520d84dc9fbeb678473"}
Jul 17 07:16:53.563527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:53.563474    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 07:16:53.563881 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:53.563563    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 07:16:53.636350 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:53.636295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:53.753887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:53.753851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:53.859318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:53.859225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:54.003467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.003428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:54.243822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.243729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:54.347408 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.347368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:54.560867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.560836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fc85c6965-n8s6t"
Jul 17 07:16:54.605803 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.605769    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp" event={"ID":"8b522f30-8d75-48da-b8ec-edf7f90e359b","Type":"ContainerStarted","Data":"24570320cda308560f81f8a86c7299304d43613f43ffedf841a7843000aa65e7"}
Jul 17 07:16:54.606213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.605971    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:54.607714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.607694    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp"
Jul 17 07:16:54.638210 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.638158    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-6nrdp" podStartSLOduration=2.425527747 podStartE2EDuration="4.638140595s" totalImagesPullingTime="2.212612848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:16:54.636926818 +0000 UTC m=+310.890349633" watchObservedRunningTime="2026-07-17 07:16:54.638140595 +0000 UTC m=+310.891563407"
Jul 17 07:16:54.673369 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.673317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:54.880876 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:54.880771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:55.050835 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:55.050805    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:16:55.149078 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:16:55.148997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:12.068898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:12.068858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:16.367977 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:16.367933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:16.430719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:16.430689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:43.286520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:43.286425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:43.383357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:43.383302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:43.455590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:43.455561    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:44.190214 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:44.190175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:44.388232 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:44.388201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:17:50.600034 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:17:50.599999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:00.595024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:00.594988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:00.681776 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:00.681740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:12.523662 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:12.523625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:16.428283 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:16.428243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:17.738658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:17.738625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:18.176409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:18.176365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:19.196653 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:19.196619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:19.304942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:19.304908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:20.745899 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:20.745860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:20.849552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:20.849521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:22.989991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:22.989949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:24.790048 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:24.790005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:26.591676 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:26.591638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:26.690909 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:26.690878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:27.119496 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:27.119450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:27.357927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:27.357889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:28.205318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:28.205284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:35.665478 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:35.665434    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:35.736260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:35.736221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:36.805796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:36.805757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:36.908868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:36.908837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:39.227103 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:39.227064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:39.325847 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:39.325811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:43.954872 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:43.954826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:44.055965 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:44.055925    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:46.847568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:46.847532    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:18:46.953112 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:46.953078    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:47.167108 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:47.167018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:47.239276 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:47.239238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:47.769863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:47.769831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:47.869024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:47.868989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:56.434253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:56.434216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:56.499258 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:56.499225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:58.905871 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:58.905838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:18:59.001299 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:18:59.001246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:03.714124 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:03.714083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:03.842627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:03.842580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:11.867804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:11.867708    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dc545f6f6-2cpf6" podUID="494f8e9a-2d4e-4131-b940-cdfa0d817168" containerName="console" containerID="cri-o://ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e" gracePeriod=15
Jul 17 07:19:12.041555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.041520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:12.107164 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.107141    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:19:12.149392 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.149293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:12.158092 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.158064    2584 generic.go:356] "Generic (PLEG): container finished" podID="494f8e9a-2d4e-4131-b940-cdfa0d817168" containerID="ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e" exitCode=0
Jul 17 07:19:12.158271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.158104    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc545f6f6-2cpf6"
Jul 17 07:19:12.158271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.158126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545f6f6-2cpf6" event={"ID":"494f8e9a-2d4e-4131-b940-cdfa0d817168","Type":"ContainerDied","Data":"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"}
Jul 17 07:19:12.158271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.158173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc545f6f6-2cpf6" event={"ID":"494f8e9a-2d4e-4131-b940-cdfa0d817168","Type":"ContainerDied","Data":"e065d762038d6fdf49404265bc999f8f742086d64a4479571ae1ebb4ca201d00"}
Jul 17 07:19:12.158271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.158201    2584 scope.go:117] "RemoveContainer" containerID="ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"
Jul 17 07:19:12.166061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.166044    2584 scope.go:117] "RemoveContainer" containerID="ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"
Jul 17 07:19:12.166319 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:19:12.166302    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e\": container with ID starting with ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e not found: ID does not exist" containerID="ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"
Jul 17 07:19:12.166384 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.166327    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e"} err="failed to get container status \"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e\": rpc error: code = NotFound desc = could not find container \"ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e\": container with ID starting with ca6914eda239af5c978eaa04c07d2abd8f3d63b35e9396cdd9f67929de42b95e not found: ID does not exist"
Jul 17 07:19:12.220772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220740    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.220936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220784    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.220936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220805    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.220936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.220936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220877    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.220936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220919    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.221260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.220952    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsdxf\" (UniqueName: \"kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf\") pod \"494f8e9a-2d4e-4131-b940-cdfa0d817168\" (UID: \"494f8e9a-2d4e-4131-b940-cdfa0d817168\") "
Jul 17 07:19:12.221260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.221148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca" (OuterVolumeSpecName: "service-ca") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:19:12.221432 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.221406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:19:12.221491 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.221424    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:19:12.221491 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.221428    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config" (OuterVolumeSpecName: "console-config") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:19:12.223063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.223038    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:19:12.223166 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.223063    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:19:12.223166 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.223077    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf" (OuterVolumeSpecName: "kube-api-access-dsdxf") pod "494f8e9a-2d4e-4131-b940-cdfa0d817168" (UID: "494f8e9a-2d4e-4131-b940-cdfa0d817168"). InnerVolumeSpecName "kube-api-access-dsdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:19:12.307988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.307960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:12.322033 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322009    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322033 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322032    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322041    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-trusted-ca-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322052    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-oauth-serving-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322061    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322070    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/494f8e9a-2d4e-4131-b940-cdfa0d817168-console-oauth-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.322182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.322078    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdxf\" (UniqueName: \"kubernetes.io/projected/494f8e9a-2d4e-4131-b940-cdfa0d817168-kube-api-access-dsdxf\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:19:12.475772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.475694    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:19:12.480615 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.480589    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc545f6f6-2cpf6"]
Jul 17 07:19:12.590900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:12.590861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:14.289654 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.289620    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494f8e9a-2d4e-4131-b940-cdfa0d817168" path="/var/lib/kubelet/pods/494f8e9a-2d4e-4131-b940-cdfa0d817168/volumes"
Jul 17 07:19:14.368984 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.368945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:14.483498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.483468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:14.540643 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.540551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:14.635148 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.635108    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:19:14.635487 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.635469    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494f8e9a-2d4e-4131-b940-cdfa0d817168" containerName="console"
Jul 17 07:19:14.635585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.635489    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="494f8e9a-2d4e-4131-b940-cdfa0d817168" containerName="console"
Jul 17 07:19:14.635641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.635588    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="494f8e9a-2d4e-4131-b940-cdfa0d817168" containerName="console"
Jul 17 07:19:14.639753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.639730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.642304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.642284    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 07:19:14.642421 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.642329    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 07:19:14.642484 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.642332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vqgqw\""
Jul 17 07:19:14.643517 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.643493    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 07:19:14.649183 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.646626    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:19:14.672439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.672417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:19:14.740205 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.740176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntk2j\" (UniqueName: \"kubernetes.io/projected/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-kube-api-access-ntk2j\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.740296 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.740243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-config-file\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.841589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.841487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntk2j\" (UniqueName: \"kubernetes.io/projected/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-kube-api-access-ntk2j\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.841589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.841581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-config-file\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.842270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.842247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-config-file\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.850587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.850565    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntk2j\" (UniqueName: \"kubernetes.io/projected/d3e910ce-7d62-4baa-ad7b-e506ab53ce37-kube-api-access-ntk2j\") pod \"limitador-limitador-54d6d988d5-2sh6l\" (UID: \"d3e910ce-7d62-4baa-ad7b-e506ab53ce37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:14.957170 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:14.957133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:15.088363 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:15.088286    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:19:15.091122 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:19:15.091096    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e910ce_7d62_4baa_ad7b_e506ab53ce37.slice/crio-96e72b199466d9e8dc14f314f86b0dda84ce628e71630376adda80ae247ccbd7 WatchSource:0}: Error finding container 96e72b199466d9e8dc14f314f86b0dda84ce628e71630376adda80ae247ccbd7: Status 404 returned error can't find the container with id 96e72b199466d9e8dc14f314f86b0dda84ce628e71630376adda80ae247ccbd7
Jul 17 07:19:15.172670 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:15.172628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l" event={"ID":"d3e910ce-7d62-4baa-ad7b-e506ab53ce37","Type":"ContainerStarted","Data":"96e72b199466d9e8dc14f314f86b0dda84ce628e71630376adda80ae247ccbd7"}
Jul 17 07:19:15.224976 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:15.224940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:15.306696 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:15.306661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:18.189213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:18.189176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l" event={"ID":"d3e910ce-7d62-4baa-ad7b-e506ab53ce37","Type":"ContainerStarted","Data":"1eee6a4e13015dbb80751b348cec343293bfdfb9bf33a8d1e3eb0884229acaaf"}
Jul 17 07:19:18.189595 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:18.189318    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:18.206876 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:18.206829    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l" podStartSLOduration=1.5380446330000002 podStartE2EDuration="4.206816125s" totalImagesPullingTime="2.668771492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:19:18.205674269 +0000 UTC m=+454.459097094" watchObservedRunningTime="2026-07-17 07:19:18.206816125 +0000 UTC m=+454.460238975"
Jul 17 07:19:29.194833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:29.194799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"
Jul 17 07:19:30.506746 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:30.506707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:30.605957 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:30.605916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:31.038569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:31.038537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:54.845437 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:54.845371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:54.984080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:54.984040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:19:57.790895 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:19:57.790850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:01.412391 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:01.412360    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:01.554509 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:01.554462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:01.672248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:01.672172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:01.762304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:01.762269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:05.141080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:05.141034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:05.242693 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:05.242657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:38.775286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:38.775201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:38.872124 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:38.872083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:39.362354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:39.362310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:39.613576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:39.613501    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:39.962929 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:39.962845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:40.063989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:40.063956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:40.338945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:40.338904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:20:49.802814 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:49.802774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:20:49.903221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:20:49.903182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:05.070118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:05.070082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:05.439644 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:05.439556    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:21:10.125612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.125572    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"]
Jul 17 07:21:10.129273 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.129250    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.133110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.133058    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 07:21:10.133110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.133090    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 07:21:10.133110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.133103    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z7wkz\""
Jul 17 07:21:10.133471 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.133456    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 07:21:10.138508 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.138488    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"]
Jul 17 07:21:10.167326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.167279    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:10.252018 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.251986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.252158 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.252029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96q9l\" (UniqueName: \"kubernetes.io/projected/6c4611d2-bbb8-440f-88a2-b67aea298503-kube-api-access-96q9l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.252158 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.252110    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.252158 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.252126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.252271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.252167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.252271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.252225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4611d2-bbb8-440f-88a2-b67aea298503-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4611d2-bbb8-440f-88a2-b67aea298503-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96q9l\" (UniqueName: \"kubernetes.io/projected/6c4611d2-bbb8-440f-88a2-b67aea298503-kube-api-access-96q9l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353737 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.353804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.353787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.355512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.355491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4611d2-bbb8-440f-88a2-b67aea298503-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.355768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.355752    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4611d2-bbb8-440f-88a2-b67aea298503-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.361173 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.361157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96q9l\" (UniqueName: \"kubernetes.io/projected/6c4611d2-bbb8-440f-88a2-b67aea298503-kube-api-access-96q9l\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w9f5k\" (UID: \"6c4611d2-bbb8-440f-88a2-b67aea298503\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.440233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.440155    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:10.569417 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.569367    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"]
Jul 17 07:21:10.571585 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:21:10.571546    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4611d2_bbb8_440f_88a2_b67aea298503.slice/crio-35c9030dd0e7086678569b5b87cddfae1bb77b034c8ea1fb6b67b3d11820be1b WatchSource:0}: Error finding container 35c9030dd0e7086678569b5b87cddfae1bb77b034c8ea1fb6b67b3d11820be1b: Status 404 returned error can't find the container with id 35c9030dd0e7086678569b5b87cddfae1bb77b034c8ea1fb6b67b3d11820be1b
Jul 17 07:21:10.644618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.644580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k" event={"ID":"6c4611d2-bbb8-440f-88a2-b67aea298503","Type":"ContainerStarted","Data":"35c9030dd0e7086678569b5b87cddfae1bb77b034c8ea1fb6b67b3d11820be1b"}
Jul 17 07:21:10.877174 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:10.877128    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:21:12.653468 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:12.653432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k" event={"ID":"6c4611d2-bbb8-440f-88a2-b67aea298503","Type":"ContainerStarted","Data":"dbd29284df0e5f674ff10eb5feb278e544c61eeebbbd1b8ce66f16525956ee50"}
Jul 17 07:21:12.672541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:12.672491    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k" podStartSLOduration=1.604289744 podStartE2EDuration="2.672476493s" totalImagesPullingTime="1.068186749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:21:12.671137941 +0000 UTC m=+568.924560754" watchObservedRunningTime="2026-07-17 07:21:12.672476493 +0000 UTC m=+568.925899305"
Jul 17 07:21:20.441141 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:20.441083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:20.441141 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:20.441153    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:20.453437 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:20.453409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:20.703583 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:20.703504    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w9f5k"
Jul 17 07:21:28.131821 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.131781    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"]
Jul 17 07:21:28.138257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.138233    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.141043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.141021    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 17 07:21:28.154822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.154797    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"]
Jul 17 07:21:28.226878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.226845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:28.302591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.302591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mp2z\" (UniqueName: \"kubernetes.io/projected/ac641c8b-a596-4366-91b0-cb2a9ad176d3-kube-api-access-8mp2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.302834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.302834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.302834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.302834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.302736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mp2z\" (UniqueName: \"kubernetes.io/projected/ac641c8b-a596-4366-91b0-cb2a9ad176d3-kube-api-access-8mp2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403993 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.403993 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.404079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.403997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.405896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.405877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac641c8b-a596-4366-91b0-cb2a9ad176d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.406156 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.406138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac641c8b-a596-4366-91b0-cb2a9ad176d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.413063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.413040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mp2z\" (UniqueName: \"kubernetes.io/projected/ac641c8b-a596-4366-91b0-cb2a9ad176d3-kube-api-access-8mp2z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4mxnb\" (UID: \"ac641c8b-a596-4366-91b0-cb2a9ad176d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.448875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.448850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:28.580913 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.580886    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"]
Jul 17 07:21:28.583764 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:21:28.583736    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac641c8b_a596_4366_91b0_cb2a9ad176d3.slice/crio-2cdbef4b274f37648c114ee178db188a192fe41407ef6582b6cb3f8f5c7337ba WatchSource:0}: Error finding container 2cdbef4b274f37648c114ee178db188a192fe41407ef6582b6cb3f8f5c7337ba: Status 404 returned error can't find the container with id 2cdbef4b274f37648c114ee178db188a192fe41407ef6582b6cb3f8f5c7337ba
Jul 17 07:21:28.722274 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:28.722192    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb" event={"ID":"ac641c8b-a596-4366-91b0-cb2a9ad176d3","Type":"ContainerStarted","Data":"2cdbef4b274f37648c114ee178db188a192fe41407ef6582b6cb3f8f5c7337ba"}
Jul 17 07:21:29.428224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:29.428188    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:21:29.728032 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:29.727948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb" event={"ID":"ac641c8b-a596-4366-91b0-cb2a9ad176d3","Type":"ContainerStarted","Data":"486220caa5708de7ea5aed4a46c4d06079e76f3ce9b17ad0054f0a7a73cafa5f"}
Jul 17 07:21:29.746283 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:29.746228    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb" podStartSLOduration=1.477503787 podStartE2EDuration="1.746213802s" totalImagesPullingTime="268.710015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:21:29.745939047 +0000 UTC m=+585.999361859" watchObservedRunningTime="2026-07-17 07:21:29.746213802 +0000 UTC m=+585.999636624"
Jul 17 07:21:30.121962 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:30.121922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:37.851787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:37.851726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:38.449605 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:38.449570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:38.449787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:38.449630    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:38.462145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:38.462114    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:38.777003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:38.776924    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4mxnb"
Jul 17 07:21:39.535097 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:39.535063    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:21:44.171192 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:44.171159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:44.207056 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:44.207026    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:21:44.215483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:44.215459    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:21:46.636760 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:46.636718    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:21:48.671900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:48.671867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:21:55.437547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:21:55.437505    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:22:19.588890 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:19.588806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:22:19.687245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:19.687205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:22:37.934656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:37.934619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:22:38.086174 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:38.086133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:22:39.071178 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:39.071144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:22:39.173470 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:22:39.173434    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:23:07.631829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:07.631795    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:23:10.391307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:10.391247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:23:10.455077 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:10.455044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:23:12.054009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:12.053975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:23:17.140806 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:17.140771    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:23:27.834146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:27.834111    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:23:46.179558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:46.179463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:23:46.280126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:23:46.280096    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:24:43.601131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:24:43.601098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:24:44.858724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:24:44.858680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:24:46.534011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:24:46.533975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:07.242957 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:07.242863    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:25:20.736468 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:20.736434    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:25:25.547999 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:25.547963    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:25.643230 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:25.643195    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:25.771293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:25.771260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:31.824189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:31.824155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:31.881577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:31.881538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:31.979523 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:31.979489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:32.078530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:32.078442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:32.185520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:32.185479    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:38.741235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:38.741200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:38.814545 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:38.814514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:40.918752 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:40.918710    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:40.994246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:40.994214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:25:56.243019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:25:56.242982    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:06.931987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:06.931953    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:14.221029 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:14.220981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:26:14.283176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:14.283147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:26:16.035673 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:16.035637    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:26.243528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:26.243494    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:36.330189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:36.330144    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:44.235692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:44.235661    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:26:44.237538 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:44.237518    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:26:47.581126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:47.581092    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:26:59.036521 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:26:59.036491    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:27:08.729162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:27:08.729129    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:27:33.933396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:27:33.933357    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:27:45.128900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:27:45.128850    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:07.837259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:07.837174    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:19.123571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:19.123535    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:28.636111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:28.636074    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:36.841757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:36.841717    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:46.638100 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:46.638066    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:28:56.338108 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:28:56.338069    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:29:05.731857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:29:05.731817    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:29:15.029372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:29:15.029319    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:30:27.443165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:30:27.443130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:31:03.927369 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:03.927322    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:31:13.231079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:13.230990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:31:21.096991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:21.096956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:31:21.162242 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:21.162212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:31:25.437018 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:25.436981    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:31:38.156573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:38.156532    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:31:44.262904 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:44.262879    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:31:44.263372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:44.262879    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:31:50.722582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:31:50.722546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:00.333719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:00.333685    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:08.028293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:08.028256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:13.754874 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:13.754835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:32:19.032657 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:19.032619    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:30.932278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:30.932238    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:39.723253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:39.723162    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:32:51.933352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:32:51.933314    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:33:12.226013 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:12.225976    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:33:21.326389 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:21.326352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:33:22.446655 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:22.446621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:33:22.548240 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:22.548207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:33:31.325153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:31.325119    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:33:43.829680 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:33:43.829643    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:02.342329 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:02.342289    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:11.726998 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:11.726909    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:21.325735 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:21.325701    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:29.433820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:29.433783    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:39.128139 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:39.128102    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:34:48.916203 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:34:48.916166    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:02.822578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:02.822539    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:12.329692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:12.329658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:22.575488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:22.575448    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:34.826167 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:34.826130    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:44.032637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:44.032544    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:35:51.228720 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:35:51.228680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:36:08.124923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:08.124885    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:36:20.338078 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:20.338046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:36:30.323692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:30.323655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:36:36.119983 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:36.119944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:37.651027 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:37.650993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:41.637160 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:41.637106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:41.695519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:41.695487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:41.941364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:41.941276    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:42.042679 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:42.042644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.025186 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.025156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.125241 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.125205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.247047 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.247014    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:36:43.250774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.250749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.254153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.254127    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2dfadf83-serving-cert\""
Jul 17 07:36:43.255184 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.255165    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:36:43.255298 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.255211    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:36:43.256290 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.256273    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:36:43.267142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.267120    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:36:43.319643 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.319608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.376884 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.376846    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz2h\" (UniqueName: \"kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.376884 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.376879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.377133 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.376999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.468304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.468263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.477391 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.477368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz2h\" (UniqueName: \"kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.477480 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.477404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.477480 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.477461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.478029 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.477994    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.479863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.479843    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.485779 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.485754    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz2h\" (UniqueName: \"kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h\") pod \"maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.563104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.563066    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:43.696334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.696249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:36:43.698532 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:36:43.698500    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eb5b15_9a64_4fbd_a3a0_3ed26e47ea8f.slice/crio-c941c3c305f9ccd709f8ab9eb3dfda49375d33823e81d026ea0e2a4aa30cf318 WatchSource:0}: Error finding container c941c3c305f9ccd709f8ab9eb3dfda49375d33823e81d026ea0e2a4aa30cf318: Status 404 returned error can't find the container with id c941c3c305f9ccd709f8ab9eb3dfda49375d33823e81d026ea0e2a4aa30cf318
Jul 17 07:36:43.700040 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.700019    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:36:43.823794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.823718    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:43.924077 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:43.924046    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:44.296042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:44.296011    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:36:44.296737 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:44.296716    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:36:44.504996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:44.504937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerStarted","Data":"c941c3c305f9ccd709f8ab9eb3dfda49375d33823e81d026ea0e2a4aa30cf318"}
Jul 17 07:36:45.485542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:45.485505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:36:45.828838 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:45.828802    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:36:46.514831 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:46.514799    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="08e24578a9492d5441723e72d8b3fcc2e004d790af665fe2c5d177ffe6496ab4" exitCode=1
Jul 17 07:36:46.515276 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:46.514866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"08e24578a9492d5441723e72d8b3fcc2e004d790af665fe2c5d177ffe6496ab4"}
Jul 17 07:36:46.515276 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:46.515080    2584 scope.go:117] "RemoveContainer" containerID="08e24578a9492d5441723e72d8b3fcc2e004d790af665fe2c5d177ffe6496ab4"
Jul 17 07:36:47.520857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:47.520772    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b" exitCode=1
Jul 17 07:36:47.520857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:47.520808    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"}
Jul 17 07:36:47.520857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:47.520853    2584 scope.go:117] "RemoveContainer" containerID="08e24578a9492d5441723e72d8b3fcc2e004d790af665fe2c5d177ffe6496ab4"
Jul 17 07:36:47.521360 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:47.521075    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:36:47.521360 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:36:47.521291    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:36:48.526154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:48.526123    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:36:48.526533 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:36:48.526293    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:36:48.564155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:48.564126    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:49.530564 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:49.530532    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:36:49.530921 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:36:49.530693    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:36:53.563687 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:53.563648    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:36:53.564228 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:36:53.564099    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:36:53.564373 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:36:53.564323    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:03.172860 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:03.172800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:37:05.285314 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:05.285283    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:37:06.602282 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:06.602252    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2" exitCode=1
Jul 17 07:37:06.602704 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:06.602329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"}
Jul 17 07:37:06.602704 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:06.602406    2584 scope.go:117] "RemoveContainer" containerID="a06eb335949d77836206dd17d32bbf9589b1d33d0e79e889dd141f80b5ea003b"
Jul 17 07:37:06.602825 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:06.602788    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:06.602982 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:06.602961    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:08.563444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:08.563400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:37:08.563830 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:08.563765    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:08.563947 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:08.563927    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:13.564083 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:13.564038    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:37:13.564493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:13.564395    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:13.564579 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:13.564560    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:25.285231 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:25.285198    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:25.285714 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:25.285407    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:30.909512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:30.909476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:37:39.285091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:39.285059    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:39.331544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:39.331504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:37:39.738011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:39.737981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerStarted","Data":"f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"}
Jul 17 07:37:39.738238 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:39.738212    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:37:39.757361 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:39.756676    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podStartSLOduration=53.299191339 podStartE2EDuration="56.756656515s" totalImagesPullingTime="3.457465176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:36:43 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:37:39.755112466 +0000 UTC m=+1556.008535280" watchObservedRunningTime="2026-07-17 07:37:39.756656515 +0000 UTC m=+1556.010079328"
Jul 17 07:37:40.743563 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:40.743530    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3" exitCode=1
Jul 17 07:37:40.744005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:40.743617    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"}
Jul 17 07:37:40.744005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:40.743693    2584 scope.go:117] "RemoveContainer" containerID="893060c744f9cc7bc94fed3b0cdbb20eb0e8f1840b5735e251b3f30d2a6914f2"
Jul 17 07:37:40.744005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:40.743930    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:37:40.744171 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:40.744136    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:41.750983 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:41.750950    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:37:41.751383 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:41.751125    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:43.563481 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:43.563441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:37:43.563941 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:43.563905    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:37:43.564122 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:43.564098    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:37:51.090259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:51.090226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:37:57.284723 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:37:57.284690    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:37:57.285212 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:37:57.284863    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:38:11.285148 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:11.285113    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:38:11.285554 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:38:11.285327    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:38:23.284970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:23.284933    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:38:23.927582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:23.927493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerStarted","Data":"3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"}
Jul 17 07:38:23.927745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:23.927712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:38:24.934111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:24.934074    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4" exitCode=1
Jul 17 07:38:24.934530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:24.934160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"}
Jul 17 07:38:24.934530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:24.934224    2584 scope.go:117] "RemoveContainer" containerID="f8879acc02a05e30d02f9697dfbf189db4657a2f2c72ec399ffe2381a63badb3"
Jul 17 07:38:24.934530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:24.934478    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:38:24.934704 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:38:24.934686    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:38:25.940877 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:25.940849    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:38:25.941268 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:38:25.941022    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:38:33.563780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:33.563738    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:38:33.564263 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:33.564122    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:38:33.564305 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:38:33.564266    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:38:46.284717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:38:46.284639    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:38:46.285069 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:38:46.284810    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:00.285018 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:00.284982    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:00.285517 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:00.285182    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:12.290595 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:12.290565    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:12.290978 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:12.290746    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:25.285355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:25.285305    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:25.285739 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:25.285528    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:36.289468 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:36.289437    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:36.289925 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:36.289602    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:48.218104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:48.218068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:48.343541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:48.343509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:51.021946 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:51.021911    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:39:51.285181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:51.285155    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:52.310197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:52.310120    2584 generic.go:356] "Generic (PLEG): container finished" podID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerID="722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2" exitCode=1
Jul 17 07:39:52.310658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:52.310193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2"}
Jul 17 07:39:52.310658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:52.310252    2584 scope.go:117] "RemoveContainer" containerID="3fffe486b4896fd5fd22b696e527a8ad2b171cd3c7dcf8d7dd975a401932a9d4"
Jul 17 07:39:52.310658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:52.310570    2584 scope.go:117] "RemoveContainer" containerID="722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2"
Jul 17 07:39:52.310780 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:52.310752    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:53.563617 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:53.563578    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:39:53.564028 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:53.563972    2584 scope.go:117] "RemoveContainer" containerID="722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2"
Jul 17 07:39:53.564155 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:53.564136    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:53.928259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:53.928226    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:39:53.974568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:53.974537    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:39:53.994992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:53.994952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:54.319548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:54.319524    2584 scope.go:117] "RemoveContainer" containerID="722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2"
Jul 17 07:39:54.319747 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:54.319728    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb_odh-ai-gateway-infra(58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"
Jul 17 07:39:54.428709 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:54.428674    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-serving-cert: secret "maas-api-e2e-ait-2dfadf83-serving-cert" not found
Jul 17 07:39:54.428885 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:54.428759    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls podName:58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f nodeName:}" failed. No retries permitted until 2026-07-17 07:39:54.928739135 +0000 UTC m=+1691.182161934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls") pod "maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" (UID: "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f") : secret "maas-api-e2e-ait-2dfadf83-serving-cert" not found
Jul 17 07:39:54.755723 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:54.755644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:54.855024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:54.854995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:54.933132 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:54.933104    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-serving-cert: secret "maas-api-e2e-ait-2dfadf83-serving-cert" not found
Jul 17 07:39:54.933267 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:54.933162    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls podName:58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f nodeName:}" failed. No retries permitted until 2026-07-17 07:39:55.933148734 +0000 UTC m=+1692.186571524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls") pod "maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" (UID: "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f") : secret "maas-api-e2e-ait-2dfadf83-serving-cert" not found
Jul 17 07:39:55.035201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.035172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:55.138009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.137982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:55.257599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.257567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:55.344746 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.344706    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:55.446832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.446809    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:39:55.537935 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.537900    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsz2h\" (UniqueName: \"kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h\") pod \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") "
Jul 17 07:39:55.538080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.537969    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca\") pod \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") "
Jul 17 07:39:55.538080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.538004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls\") pod \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\" (UID: \"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f\") "
Jul 17 07:39:55.538313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.538288    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" (UID: "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:39:55.540058 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.540036    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" (UID: "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:39:55.540131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.540098    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h" (OuterVolumeSpecName: "kube-api-access-bsz2h") pod "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" (UID: "58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f"). InnerVolumeSpecName "kube-api-access-bsz2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:39:55.639019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.638947    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:39:55.639019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.638975    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:39:55.639019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:55.638984    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsz2h\" (UniqueName: \"kubernetes.io/projected/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f-kube-api-access-bsz2h\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:39:56.328370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.328347    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"
Jul 17 07:39:56.328739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.328331    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb" event={"ID":"58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f","Type":"ContainerDied","Data":"c941c3c305f9ccd709f8ab9eb3dfda49375d33823e81d026ea0e2a4aa30cf318"}
Jul 17 07:39:56.328739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.328471    2584 scope.go:117] "RemoveContainer" containerID="722090842df953ccb28c13f1b9694ff3e6341df841d5ca421aa017b6599bbae2"
Jul 17 07:39:56.351399 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.351375    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:39:56.358297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.358277    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dfadf83-6469cd74b7-nsxsb"]
Jul 17 07:39:56.512663 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.512634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.592792    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593474    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593494    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593554    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593563    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593574    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593583    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593601    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593610    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593621    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593629    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593755    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593764    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593776    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.594433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.593792    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:39:56.599879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.599853    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.602761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.602738    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:39:56.602887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.602784    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:39:56.603094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.603077    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:39:56.603198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.603182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b531d224-serving-cert\""
Jul 17 07:39:56.608125 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.608104    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:39:56.630104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.630075    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:56.648440 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.648413    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.648540 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.648443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.648580 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.648531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5bm\" (UniqueName: \"kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.725875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.725843    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:56.749088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.749051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.749202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.749094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.749202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.749147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5bm\" (UniqueName: \"kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.749765 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.749730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.751538 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.751518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.757176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.757153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5bm\" (UniqueName: \"kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm\") pod \"maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:56.912491 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:56.912405    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:39:57.038650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:57.038624    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:39:57.040830 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:39:57.040805    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7b0b9d_b97e_4b05_86e0_8f64e4708f8f.slice/crio-f5d7b66638f5fa6f3ae6b1ca84209ce059b3022c8589e9864fc5c1f035bedc1b WatchSource:0}: Error finding container f5d7b66638f5fa6f3ae6b1ca84209ce059b3022c8589e9864fc5c1f035bedc1b: Status 404 returned error can't find the container with id f5d7b66638f5fa6f3ae6b1ca84209ce059b3022c8589e9864fc5c1f035bedc1b
Jul 17 07:39:57.126063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:57.126026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:57.229012 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:57.228944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:39:57.333388 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:57.333354    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerStarted","Data":"f5d7b66638f5fa6f3ae6b1ca84209ce059b3022c8589e9864fc5c1f035bedc1b"}
Jul 17 07:39:58.290316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:58.290283    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" path="/var/lib/kubelet/pods/58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f/volumes"
Jul 17 07:39:58.339909 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:58.339879    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerID="f05f1dbf9ef5e16e0430318cbc791b5194d5df410b219b99e1ecbab4c9c53571" exitCode=1
Jul 17 07:39:58.340303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:58.339932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerDied","Data":"f05f1dbf9ef5e16e0430318cbc791b5194d5df410b219b99e1ecbab4c9c53571"}
Jul 17 07:39:58.340303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:58.340156    2584 scope.go:117] "RemoveContainer" containerID="f05f1dbf9ef5e16e0430318cbc791b5194d5df410b219b99e1ecbab4c9c53571"
Jul 17 07:39:59.345982 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:59.345950    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1" exitCode=1
Jul 17 07:39:59.346438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:59.346042    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerDied","Data":"cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"}
Jul 17 07:39:59.346438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:59.346125    2584 scope.go:117] "RemoveContainer" containerID="f05f1dbf9ef5e16e0430318cbc791b5194d5df410b219b99e1ecbab4c9c53571"
Jul 17 07:39:59.346438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:59.346301    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:39:59.346577 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:39:59.346557    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:39:59.704992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:39:59.704958    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:40:00.351793 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:00.351764    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:40:00.352150 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:00.351921    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:01.912933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:01.912893    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:01.913321 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:01.913306    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:40:01.913507 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:01.913487    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:02.836453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:02.836418    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:40:06.913138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:06.913048    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:06.913523 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:06.913499    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:40:06.913683 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:06.913664    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:07.254555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:07.254464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:40:07.354572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:07.354538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:40:17.894177 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:17.894141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:40:17.994499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:17.994466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:40:21.284958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:21.284927    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:40:22.440974 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:22.440885    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2" exitCode=1
Jul 17 07:40:22.441364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:22.440963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerDied","Data":"5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"}
Jul 17 07:40:22.441364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:22.441027    2584 scope.go:117] "RemoveContainer" containerID="cb820df03ba18d3de28bfd3f056c067dd50e3b8507a7548137d804a272ea8ad1"
Jul 17 07:40:22.441364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:22.441334    2584 scope.go:117] "RemoveContainer" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"
Jul 17 07:40:22.441582 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:22.441564    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:26.912700 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:26.912664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:26.913288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:26.912721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:26.913288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:26.913125    2584 scope.go:117] "RemoveContainer" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"
Jul 17 07:40:26.913456 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:26.913369    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:38.284891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:38.284855    2584 scope.go:117] "RemoveContainer" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"
Jul 17 07:40:38.285261 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:38.285079    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:51.284711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:51.284674    2584 scope.go:117] "RemoveContainer" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"
Jul 17 07:40:52.574255 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:52.574221    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerID="e1487012fd541f8d5a2ace35b4921b3608a286d661806cd7cd8035d01466f727" exitCode=1
Jul 17 07:40:52.574716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:52.574298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerDied","Data":"e1487012fd541f8d5a2ace35b4921b3608a286d661806cd7cd8035d01466f727"}
Jul 17 07:40:52.574716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:52.574377    2584 scope.go:117] "RemoveContainer" containerID="5978d0a632e0e08816bc17cc24c3f12678c987c445e60e81c7117439fc4ddfe2"
Jul 17 07:40:52.574716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:52.574692    2584 scope.go:117] "RemoveContainer" containerID="e1487012fd541f8d5a2ace35b4921b3608a286d661806cd7cd8035d01466f727"
Jul 17 07:40:52.574925 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:52.574904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:40:56.913524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:56.913488    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:56.913910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:56.913547    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:40:56.913910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:40:56.913864    2584 scope.go:117] "RemoveContainer" containerID="e1487012fd541f8d5a2ace35b4921b3608a286d661806cd7cd8035d01466f727"
Jul 17 07:40:56.914068 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:40:56.914038    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck_odh-ai-gateway-infra(ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"
Jul 17 07:41:01.271788 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:01.271752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:01.359983 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:01.359952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:04.058709 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:04.058665    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:41:07.367589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.367556    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:41:07.390760 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.390723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:07.505417 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.505384    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:41:07.638380 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.638278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck" event={"ID":"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f","Type":"ContainerDied","Data":"f5d7b66638f5fa6f3ae6b1ca84209ce059b3022c8589e9864fc5c1f035bedc1b"}
Jul 17 07:41:07.638380 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.638306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"
Jul 17 07:41:07.638590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.638379    2584 scope.go:117] "RemoveContainer" containerID="e1487012fd541f8d5a2ace35b4921b3608a286d661806cd7cd8035d01466f727"
Jul 17 07:41:07.643939 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.643914    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv5bm\" (UniqueName: \"kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm\") pod \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") "
Jul 17 07:41:07.644046 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.643984    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls\") pod \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") "
Jul 17 07:41:07.644096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.644079    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca\") pod \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\" (UID: \"ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f\") "
Jul 17 07:41:07.644506 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.644481    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" (UID: "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:41:07.646386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.646355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" (UID: "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:41:07.646513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.646457    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm" (OuterVolumeSpecName: "kube-api-access-rv5bm") pod "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" (UID: "ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f"). InnerVolumeSpecName "kube-api-access-rv5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:41:07.745312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.745282    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5bm\" (UniqueName: \"kubernetes.io/projected/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-kube-api-access-rv5bm\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:41:07.745312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.745308    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:41:07.745312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.745319    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:41:07.827656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.827623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:07.960270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.960241    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:41:07.962369 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:07.962325    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b531d224-5b597c4b47-mhpck"]
Jul 17 07:41:08.289596 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.289560    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" path="/var/lib/kubelet/pods/ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f/volumes"
Jul 17 07:41:08.339848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.339819    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:08.340111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340100    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340112    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340127    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340132    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340143    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340148    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340164    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340170    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340216    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340223    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340234    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:41:08.340467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.340240    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eb5b15-9a64-4fbd-a3a0-3ed26e47ea8f" containerName="maas-api"
Jul 17 07:41:08.344442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.344425    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:08.348493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.348470    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:41:08.349172 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.349155    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:41:08.356438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.356418    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:08.449936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.449897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk\" (UID: \"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:08.458948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.458912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:08.550743 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.550662    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk\" (UID: \"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:08.551426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.551403    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk\" (UID: \"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:08.552418 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.552393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:08.650869 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.650836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:08.654845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.654826    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:08.786219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:08.786193    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:08.787793 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:41:08.787760    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313aabe2_4230_4b9e_b9c5_40b1ee2fb97a.slice/crio-e6214febd279807784c7a68f6e750c42d04b7d8d102d93b6ba83cbdf358404fc WatchSource:0}: Error finding container e6214febd279807784c7a68f6e750c42d04b7d8d102d93b6ba83cbdf358404fc: Status 404 returned error can't find the container with id e6214febd279807784c7a68f6e750c42d04b7d8d102d93b6ba83cbdf358404fc
Jul 17 07:41:09.647840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.647804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerStarted","Data":"e6214febd279807784c7a68f6e750c42d04b7d8d102d93b6ba83cbdf358404fc"}
Jul 17 07:41:09.731919 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.731891    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:09.838489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.838408    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:09.846416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.846385    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:41:09.846889 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.846854    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:09.846889 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.846872    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:09.847069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.846972    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:09.847069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.846984    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7b0b9d-b97e-4b05-86e0-8f64e4708f8f" containerName="maas-api"
Jul 17 07:41:09.850228 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.850032    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:09.854997 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.854973    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:41:09.855445 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.855417    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-370a9d5a-serving-cert\""
Jul 17 07:41:09.856008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.855986    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:41:09.863769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.863747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:41:09.931778 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.931668    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:09.963930 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.963895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:09.964094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.963971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sbr\" (UniqueName: \"kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:09.964094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:09.964055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.065313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.065275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.065526 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.065452    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sbr\" (UniqueName: \"kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.065639 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.065621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.065765 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:10.065744    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-serving-cert: secret "maas-api-e2e-ait-370a9d5a-serving-cert" not found
Jul 17 07:41:10.065844 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:10.065832    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls podName:295b56da-a428-4877-a54d-c3ceb861e79b nodeName:}" failed. No retries permitted until 2026-07-17 07:41:10.565813586 +0000 UTC m=+1766.819236401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls") pod "maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" (UID: "295b56da-a428-4877-a54d-c3ceb861e79b") : secret "maas-api-e2e-ait-370a9d5a-serving-cert" not found
Jul 17 07:41:10.066103 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.066055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.078650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.078622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sbr\" (UniqueName: \"kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.504312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.504272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:10.570308 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.570272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:10.570518 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:10.570452    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-serving-cert: secret "maas-api-e2e-ait-370a9d5a-serving-cert" not found
Jul 17 07:41:10.570596 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:10.570532    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls podName:295b56da-a428-4877-a54d-c3ceb861e79b nodeName:}" failed. No retries permitted until 2026-07-17 07:41:11.570513954 +0000 UTC m=+1767.823936755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls") pod "maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" (UID: "295b56da-a428-4877-a54d-c3ceb861e79b") : secret "maas-api-e2e-ait-370a9d5a-serving-cert" not found
Jul 17 07:41:10.617812 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.617766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:10.751996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:10.751945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:11.578994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.578953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:11.581381 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.581361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:11.659186 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.659152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerStarted","Data":"0a00468f6812b4ef93458a6d9fbf6532ed9810855c728e65691831ad166a7f6c"}
Jul 17 07:41:11.663474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.663451    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:11.694176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.694131    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" podStartSLOduration=1.38321604 podStartE2EDuration="3.694115135s" totalImagesPullingTime="2.310899095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:41:11.692590751 +0000 UTC m=+1767.946013563" watchObservedRunningTime="2026-07-17 07:41:11.694115135 +0000 UTC m=+1767.947537947"
Jul 17 07:41:11.812525 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:11.812502    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:41:11.813797 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:41:11.813768    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295b56da_a428_4877_a54d_c3ceb861e79b.slice/crio-d44feafb744ef47ce95490302633f44b0df1c8496a4cd1ec386f3a1d83ce35ef WatchSource:0}: Error finding container d44feafb744ef47ce95490302633f44b0df1c8496a4cd1ec386f3a1d83ce35ef: Status 404 returned error can't find the container with id d44feafb744ef47ce95490302633f44b0df1c8496a4cd1ec386f3a1d83ce35ef
Jul 17 07:41:12.645537 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.645503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:12.664354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.664317    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="cbed05a80f6e31928696fd4ae48f23b1feb17e66e111a1e565215f66179aa437" exitCode=1
Jul 17 07:41:12.664521 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.664380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"cbed05a80f6e31928696fd4ae48f23b1feb17e66e111a1e565215f66179aa437"}
Jul 17 07:41:12.664521 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.664433    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerStarted","Data":"d44feafb744ef47ce95490302633f44b0df1c8496a4cd1ec386f3a1d83ce35ef"}
Jul 17 07:41:12.664693 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.664676    2584 scope.go:117] "RemoveContainer" containerID="cbed05a80f6e31928696fd4ae48f23b1feb17e66e111a1e565215f66179aa437"
Jul 17 07:41:12.666001 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.665978    2584 generic.go:356] "Generic (PLEG): container finished" podID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerID="0a00468f6812b4ef93458a6d9fbf6532ed9810855c728e65691831ad166a7f6c" exitCode=6
Jul 17 07:41:12.666090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.666013    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerDied","Data":"0a00468f6812b4ef93458a6d9fbf6532ed9810855c728e65691831ad166a7f6c"}
Jul 17 07:41:12.666275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:12.666258    2584 scope.go:117] "RemoveContainer" containerID="0a00468f6812b4ef93458a6d9fbf6532ed9810855c728e65691831ad166a7f6c"
Jul 17 07:41:13.672079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.671994    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931" exitCode=1
Jul 17 07:41:13.672558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.672078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"}
Jul 17 07:41:13.672558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.672135    2584 scope.go:117] "RemoveContainer" containerID="cbed05a80f6e31928696fd4ae48f23b1feb17e66e111a1e565215f66179aa437"
Jul 17 07:41:13.672558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.672426    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:13.672712 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:13.672638    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:13.673808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.673789    2584 generic.go:356] "Generic (PLEG): container finished" podID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerID="eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb" exitCode=6
Jul 17 07:41:13.673868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.673848    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerDied","Data":"eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb"}
Jul 17 07:41:13.674096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.674081    2584 scope.go:117] "RemoveContainer" containerID="eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb"
Jul 17 07:41:13.674258 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:13.674239    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk_odh-ai-gateway-infra(313aabe2-4230-4b9e-b9c5-40b1ee2fb97a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a"
Jul 17 07:41:13.682719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:13.682701    2584 scope.go:117] "RemoveContainer" containerID="0a00468f6812b4ef93458a6d9fbf6532ed9810855c728e65691831ad166a7f6c"
Jul 17 07:41:14.680281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:14.680250    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:14.680745 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:14.680466    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:14.681606 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:14.681586    2584 scope.go:117] "RemoveContainer" containerID="eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb"
Jul 17 07:41:14.681751 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:14.681738    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk_odh-ai-gateway-infra(313aabe2-4230-4b9e-b9c5-40b1ee2fb97a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a"
Jul 17 07:41:16.663583 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:16.663548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:16.664055 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:16.663915    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:16.664097 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:16.664074    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:16.878430 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:16.878392    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:41:21.663574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:21.663531    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:21.663989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:21.663923    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:21.664097 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:21.664078    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:28.285545 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:28.285510    2584 scope.go:117] "RemoveContainer" containerID="eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb"
Jul 17 07:41:28.744044 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:28.743955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerStarted","Data":"2fe909dea74a09180e5244de29f37ba901786ef3587a428caeabbbe45bfa6c1c"}
Jul 17 07:41:29.750128 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:29.750092    2584 generic.go:356] "Generic (PLEG): container finished" podID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerID="2fe909dea74a09180e5244de29f37ba901786ef3587a428caeabbbe45bfa6c1c" exitCode=6
Jul 17 07:41:29.750623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:29.750169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerDied","Data":"2fe909dea74a09180e5244de29f37ba901786ef3587a428caeabbbe45bfa6c1c"}
Jul 17 07:41:29.750623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:29.750228    2584 scope.go:117] "RemoveContainer" containerID="eb014a0f8291712ca62c5d8df18fe9478f49babced31e160d06de59b03082beb"
Jul 17 07:41:29.750623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:29.750532    2584 scope.go:117] "RemoveContainer" containerID="2fe909dea74a09180e5244de29f37ba901786ef3587a428caeabbbe45bfa6c1c"
Jul 17 07:41:29.750798 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:29.750743    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk_odh-ai-gateway-infra(313aabe2-4230-4b9e-b9c5-40b1ee2fb97a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a"
Jul 17 07:41:29.781266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:29.781238    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:30.887501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:30.887478    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:31.048963 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.048934    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca\") pod \"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a\" (UID: \"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a\") "
Jul 17 07:41:31.049292 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.049273    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" (UID: "313aabe2-4230-4b9e-b9c5-40b1ee2fb97a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:41:31.150015 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.149980    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:41:31.761102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.761068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk" event={"ID":"313aabe2-4230-4b9e-b9c5-40b1ee2fb97a","Type":"ContainerDied","Data":"e6214febd279807784c7a68f6e750c42d04b7d8d102d93b6ba83cbdf358404fc"}
Jul 17 07:41:31.761102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.761101    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"
Jul 17 07:41:31.761365 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.761132    2584 scope.go:117] "RemoveContainer" containerID="2fe909dea74a09180e5244de29f37ba901786ef3587a428caeabbbe45bfa6c1c"
Jul 17 07:41:31.784308 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.784270    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:31.792014 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:31.791990    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-wfksk"]
Jul 17 07:41:32.290245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:32.290208    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" path="/var/lib/kubelet/pods/313aabe2-4230-4b9e-b9c5-40b1ee2fb97a/volumes"
Jul 17 07:41:34.286970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:34.286940    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:34.781689 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:34.781656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerStarted","Data":"ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"}
Jul 17 07:41:34.781889 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:34.781871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:35.788090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:35.788057    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e" exitCode=1
Jul 17 07:41:35.788547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:35.788111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"}
Jul 17 07:41:35.788547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:35.788174    2584 scope.go:117] "RemoveContainer" containerID="c3c11830f3fa6e7289a5181a077202697a52c60c6efad0b466c3c30b5b29d931"
Jul 17 07:41:35.788547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:35.788363    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:41:35.788672 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:35.788561    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:36.794496 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:36.794420    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:41:36.794847 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:36.794591    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:41.664003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:41.663953    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:41:41.664453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:41.664369    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:41:41.664555 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:41.664536    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:44.319325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:44.319291    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:41:44.319751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:44.319568    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:41:52.285234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:52.285201    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:41:52.285669 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:41:52.285418    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:41:53.277821 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:53.277776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:41:54.546803 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:41:54.546769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:42:01.183226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183191    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183513    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183527    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183532    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183589    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.183613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.183600    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:01.186544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.186526    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:01.189086 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.189064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:42:01.195733 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.195709    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:01.290082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.290049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nw25t\" (UID: \"13b796f7-194e-4b59-9829-d1b89815a94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:01.391191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.391154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nw25t\" (UID: \"13b796f7-194e-4b59-9829-d1b89815a94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:01.391794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.391774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nw25t\" (UID: \"13b796f7-194e-4b59-9829-d1b89815a94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:01.497941 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.497868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:01.627728 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.627702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:01.630137 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:42:01.630109    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b796f7_194e_4b59_9829_d1b89815a94b.slice/crio-aa3f3e4a36c69c2cbced072c5f5ed58046de4cd89d45188ead204711fece4dcb WatchSource:0}: Error finding container aa3f3e4a36c69c2cbced072c5f5ed58046de4cd89d45188ead204711fece4dcb: Status 404 returned error can't find the container with id aa3f3e4a36c69c2cbced072c5f5ed58046de4cd89d45188ead204711fece4dcb
Jul 17 07:42:01.897006 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:01.896967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" event={"ID":"13b796f7-194e-4b59-9829-d1b89815a94b","Type":"ContainerStarted","Data":"aa3f3e4a36c69c2cbced072c5f5ed58046de4cd89d45188ead204711fece4dcb"}
Jul 17 07:42:02.902425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:02.902389    2584 generic.go:356] "Generic (PLEG): container finished" podID="13b796f7-194e-4b59-9829-d1b89815a94b" containerID="9864bdbc04d298b9c08a8fa43e7bc5ab57011f23d32c411ac1b4933fff4ceb04" exitCode=6
Jul 17 07:42:02.902804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:02.902470    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" event={"ID":"13b796f7-194e-4b59-9829-d1b89815a94b","Type":"ContainerDied","Data":"9864bdbc04d298b9c08a8fa43e7bc5ab57011f23d32c411ac1b4933fff4ceb04"}
Jul 17 07:42:02.902804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:02.902668    2584 scope.go:117] "RemoveContainer" containerID="9864bdbc04d298b9c08a8fa43e7bc5ab57011f23d32c411ac1b4933fff4ceb04"
Jul 17 07:42:03.908327 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:03.908290    2584 generic.go:356] "Generic (PLEG): container finished" podID="13b796f7-194e-4b59-9829-d1b89815a94b" containerID="ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339" exitCode=6
Jul 17 07:42:03.908807 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:03.908377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" event={"ID":"13b796f7-194e-4b59-9829-d1b89815a94b","Type":"ContainerDied","Data":"ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339"}
Jul 17 07:42:03.908807 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:03.908432    2584 scope.go:117] "RemoveContainer" containerID="9864bdbc04d298b9c08a8fa43e7bc5ab57011f23d32c411ac1b4933fff4ceb04"
Jul 17 07:42:03.908807 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:03.908590    2584 scope.go:117] "RemoveContainer" containerID="ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339"
Jul 17 07:42:03.908940 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:03.908814    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nw25t_odh-ai-gateway-infra(13b796f7-194e-4b59-9829-d1b89815a94b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" podUID="13b796f7-194e-4b59-9829-d1b89815a94b"
Jul 17 07:42:04.286479 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.286451    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:42:04.286750 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.286737    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:42:04.913928 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.913900    2584 scope.go:117] "RemoveContainer" containerID="ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339"
Jul 17 07:42:04.914420 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:04.914128    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nw25t_odh-ai-gateway-infra(13b796f7-194e-4b59-9829-d1b89815a94b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" podUID="13b796f7-194e-4b59-9829-d1b89815a94b"
Jul 17 07:42:04.915495 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.915466    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerStarted","Data":"b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"}
Jul 17 07:42:04.915703 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.915688    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:42:04.952131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:04.952076    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podStartSLOduration=54.341690394 podStartE2EDuration="55.952059379s" totalImagesPullingTime="1.610368985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:41:09 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:41:34.811821146 +0000 UTC m=+1791.065243959" watchObservedRunningTime="2026-07-17 07:42:04.952059379 +0000 UTC m=+1821.205482191"
Jul 17 07:42:05.921637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:05.921603    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c" exitCode=1
Jul 17 07:42:05.922051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:05.921663    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"}
Jul 17 07:42:05.922051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:05.921706    2584 scope.go:117] "RemoveContainer" containerID="ef8d14402bf7bc232c77da7e36e374f9428a79532abf2e915181c26785f0a73e"
Jul 17 07:42:05.922051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:05.921965    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:05.922239 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:05.922144    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:06.927313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:06.927288    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:06.927707 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:06.927465    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:08.413357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.413309    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:08.413736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.413642    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:08.413736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.413655    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:08.413736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.413721    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="313aabe2-4230-4b9e-b9c5-40b1ee2fb97a" containerName="revoke-keys"
Jul 17 07:42:08.417831 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.417812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:08.426828 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.426802    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:08.550586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.550539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5\" (UID: \"b252f65b-937f-4e6b-9269-0ae12fa44e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:08.651681 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.651650    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5\" (UID: \"b252f65b-937f-4e6b-9269-0ae12fa44e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:08.652234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.652215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5\" (UID: \"b252f65b-937f-4e6b-9269-0ae12fa44e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:08.728387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.728262    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:08.856478 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.856451    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:08.858928 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:42:08.858897    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb252f65b_937f_4e6b_9269_0ae12fa44e5a.slice/crio-70674c603ae886882e7481bdaafb0afa5e3a0668be1f737d76e0bec75d165caf WatchSource:0}: Error finding container 70674c603ae886882e7481bdaafb0afa5e3a0668be1f737d76e0bec75d165caf: Status 404 returned error can't find the container with id 70674c603ae886882e7481bdaafb0afa5e3a0668be1f737d76e0bec75d165caf
Jul 17 07:42:08.936387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:08.936331    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" event={"ID":"b252f65b-937f-4e6b-9269-0ae12fa44e5a","Type":"ContainerStarted","Data":"70674c603ae886882e7481bdaafb0afa5e3a0668be1f737d76e0bec75d165caf"}
Jul 17 07:42:09.942545 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:09.942512    2584 generic.go:356] "Generic (PLEG): container finished" podID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerID="8d3dfc021d0756958b6a2249be77efaae8cf01931a4e6f78f199f7ea758f5ee7" exitCode=6
Jul 17 07:42:09.943008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:09.942613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" event={"ID":"b252f65b-937f-4e6b-9269-0ae12fa44e5a","Type":"ContainerDied","Data":"8d3dfc021d0756958b6a2249be77efaae8cf01931a4e6f78f199f7ea758f5ee7"}
Jul 17 07:42:09.943008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:09.942808    2584 scope.go:117] "RemoveContainer" containerID="8d3dfc021d0756958b6a2249be77efaae8cf01931a4e6f78f199f7ea758f5ee7"
Jul 17 07:42:10.948438 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:10.948407    2584 generic.go:356] "Generic (PLEG): container finished" podID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerID="1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20" exitCode=6
Jul 17 07:42:10.948849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:10.948493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" event={"ID":"b252f65b-937f-4e6b-9269-0ae12fa44e5a","Type":"ContainerDied","Data":"1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20"}
Jul 17 07:42:10.948849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:10.948551    2584 scope.go:117] "RemoveContainer" containerID="8d3dfc021d0756958b6a2249be77efaae8cf01931a4e6f78f199f7ea758f5ee7"
Jul 17 07:42:10.948849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:10.948771    2584 scope.go:117] "RemoveContainer" containerID="1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20"
Jul 17 07:42:10.949019 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:10.948984    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5_odh-ai-gateway-infra(b252f65b-937f-4e6b-9269-0ae12fa44e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a"
Jul 17 07:42:11.663837 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:11.663800    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:42:11.664195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:11.664180    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:11.664381 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:11.664363    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:11.954010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:11.953928    2584 scope.go:117] "RemoveContainer" containerID="1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20"
Jul 17 07:42:11.954394 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:11.954104    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5_odh-ai-gateway-infra(b252f65b-937f-4e6b-9269-0ae12fa44e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a"
Jul 17 07:42:15.284725 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:15.284694    2584 scope.go:117] "RemoveContainer" containerID="ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339"
Jul 17 07:42:15.971069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:15.971035    2584 generic.go:356] "Generic (PLEG): container finished" podID="13b796f7-194e-4b59-9829-d1b89815a94b" containerID="dd92d23209c1cc178bb00ea01f99247336ac7a87d095dd2548c385135c4aa41e" exitCode=6
Jul 17 07:42:15.971219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:15.971095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" event={"ID":"13b796f7-194e-4b59-9829-d1b89815a94b","Type":"ContainerDied","Data":"dd92d23209c1cc178bb00ea01f99247336ac7a87d095dd2548c385135c4aa41e"}
Jul 17 07:42:15.971219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:15.971147    2584 scope.go:117] "RemoveContainer" containerID="ae60cd15ce0b0c040f8a2b72b08b855c0632c1b23f38d0eeacd2af4847971339"
Jul 17 07:42:15.971483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:15.971462    2584 scope.go:117] "RemoveContainer" containerID="dd92d23209c1cc178bb00ea01f99247336ac7a87d095dd2548c385135c4aa41e"
Jul 17 07:42:15.971700 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:15.971683    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nw25t_odh-ai-gateway-infra(13b796f7-194e-4b59-9829-d1b89815a94b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" podUID="13b796f7-194e-4b59-9829-d1b89815a94b"
Jul 17 07:42:17.003195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.003159    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:17.139516 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.139491    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:17.216712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.216672    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca\") pod \"13b796f7-194e-4b59-9829-d1b89815a94b\" (UID: \"13b796f7-194e-4b59-9829-d1b89815a94b\") "
Jul 17 07:42:17.217072 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.217045    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13b796f7-194e-4b59-9829-d1b89815a94b" (UID: "13b796f7-194e-4b59-9829-d1b89815a94b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:42:17.317950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.317918    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13b796f7-194e-4b59-9829-d1b89815a94b-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:42:17.984895 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.984861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t" event={"ID":"13b796f7-194e-4b59-9829-d1b89815a94b","Type":"ContainerDied","Data":"aa3f3e4a36c69c2cbced072c5f5ed58046de4cd89d45188ead204711fece4dcb"}
Jul 17 07:42:17.985080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.984900    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"
Jul 17 07:42:17.985080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:17.984921    2584 scope.go:117] "RemoveContainer" containerID="dd92d23209c1cc178bb00ea01f99247336ac7a87d095dd2548c385135c4aa41e"
Jul 17 07:42:18.006903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:18.006870    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:18.009038 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:18.009010    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nw25t"]
Jul 17 07:42:18.289903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:18.289873    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" path="/var/lib/kubelet/pods/13b796f7-194e-4b59-9829-d1b89815a94b/volumes"
Jul 17 07:42:27.285064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:27.285034    2584 scope.go:117] "RemoveContainer" containerID="1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20"
Jul 17 07:42:27.285572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:27.285135    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:27.285572 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:27.285278    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:28.032738 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:28.032706    2584 generic.go:356] "Generic (PLEG): container finished" podID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerID="121cb5304ce42235d79b4fb900336de263ad6a47c3904b66d36863335e7b673a" exitCode=6
Jul 17 07:42:28.032880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:28.032790    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" event={"ID":"b252f65b-937f-4e6b-9269-0ae12fa44e5a","Type":"ContainerDied","Data":"121cb5304ce42235d79b4fb900336de263ad6a47c3904b66d36863335e7b673a"}
Jul 17 07:42:28.032880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:28.032861    2584 scope.go:117] "RemoveContainer" containerID="1a8930497463643a615124dd83014e674c0bbd3e3daa5fd735da0284c9693d20"
Jul 17 07:42:28.033180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:28.033161    2584 scope.go:117] "RemoveContainer" containerID="121cb5304ce42235d79b4fb900336de263ad6a47c3904b66d36863335e7b673a"
Jul 17 07:42:28.033410 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:28.033388    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5_odh-ai-gateway-infra(b252f65b-937f-4e6b-9269-0ae12fa44e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a"
Jul 17 07:42:29.063555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:29.063520    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:29.195780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:29.195757    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:29.321489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:29.321409    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca\") pod \"b252f65b-937f-4e6b-9269-0ae12fa44e5a\" (UID: \"b252f65b-937f-4e6b-9269-0ae12fa44e5a\") "
Jul 17 07:42:29.321756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:29.321733    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b252f65b-937f-4e6b-9269-0ae12fa44e5a" (UID: "b252f65b-937f-4e6b-9269-0ae12fa44e5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:42:29.422714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:29.422678    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b252f65b-937f-4e6b-9269-0ae12fa44e5a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:42:30.043780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.043750    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"
Jul 17 07:42:30.044040 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.043787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5" event={"ID":"b252f65b-937f-4e6b-9269-0ae12fa44e5a","Type":"ContainerDied","Data":"70674c603ae886882e7481bdaafb0afa5e3a0668be1f737d76e0bec75d165caf"}
Jul 17 07:42:30.044040 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.043844    2584 scope.go:117] "RemoveContainer" containerID="121cb5304ce42235d79b4fb900336de263ad6a47c3904b66d36863335e7b673a"
Jul 17 07:42:30.068759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.068731    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:30.075695 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.075671    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-bbwk5"]
Jul 17 07:42:30.289312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:30.289281    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" path="/var/lib/kubelet/pods/b252f65b-937f-4e6b-9269-0ae12fa44e5a/volumes"
Jul 17 07:42:40.284831 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:40.284802    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:40.285202 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:40.284984    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:46.044072 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:46.044034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:42:51.227403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227367    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227697    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227711    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227721    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227727    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227736    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227742    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227747    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227753    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227760    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227765    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227771    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.227811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227776    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.228207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227830    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.228207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227837    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.228207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227846    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.228207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227852    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b796f7-194e-4b59-9829-d1b89815a94b" containerName="revoke-keys"
Jul 17 07:42:51.228207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.227859    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:42:51.230795 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.230777    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:42:51.233367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.233327    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:42:51.240166 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.240139    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:42:51.304223 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.304176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw\" (UID: \"10b70083-0ec6-4996-a2a5-91ab190af85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:42:51.404699 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.404652    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw\" (UID: \"10b70083-0ec6-4996-a2a5-91ab190af85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:42:51.405304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.405281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw\" (UID: \"10b70083-0ec6-4996-a2a5-91ab190af85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:42:51.542863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.542836    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:42:51.669502 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:51.669443    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:42:51.672023 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:42:51.671996    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b70083_0ec6_4996_a2a5_91ab190af85b.slice/crio-b1cd2f10d1bf536560b4494f89fa60cc171b89fc8663844f92385037a6efc5e0 WatchSource:0}: Error finding container b1cd2f10d1bf536560b4494f89fa60cc171b89fc8663844f92385037a6efc5e0: Status 404 returned error can't find the container with id b1cd2f10d1bf536560b4494f89fa60cc171b89fc8663844f92385037a6efc5e0
Jul 17 07:42:52.134325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:52.134245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerStarted","Data":"ca7f6977de1af6cc502af92e50981fea6b6a8a3dc83d88a934d115ccfb2439cd"}
Jul 17 07:42:52.134325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:52.134302    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerStarted","Data":"b1cd2f10d1bf536560b4494f89fa60cc171b89fc8663844f92385037a6efc5e0"}
Jul 17 07:42:52.152188 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:52.152139    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" podStartSLOduration=1.15212455 podStartE2EDuration="1.15212455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:42:52.150619301 +0000 UTC m=+1868.404042112" watchObservedRunningTime="2026-07-17 07:42:52.15212455 +0000 UTC m=+1868.405547407"
Jul 17 07:42:53.139507 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:53.139473    2584 generic.go:356] "Generic (PLEG): container finished" podID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerID="ca7f6977de1af6cc502af92e50981fea6b6a8a3dc83d88a934d115ccfb2439cd" exitCode=6
Jul 17 07:42:53.139899 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:53.139546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerDied","Data":"ca7f6977de1af6cc502af92e50981fea6b6a8a3dc83d88a934d115ccfb2439cd"}
Jul 17 07:42:53.139899 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:53.139716    2584 scope.go:117] "RemoveContainer" containerID="ca7f6977de1af6cc502af92e50981fea6b6a8a3dc83d88a934d115ccfb2439cd"
Jul 17 07:42:54.145616 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:54.145570    2584 generic.go:356] "Generic (PLEG): container finished" podID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerID="6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67" exitCode=6
Jul 17 07:42:54.146059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:54.145632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerDied","Data":"6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67"}
Jul 17 07:42:54.146059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:54.145689    2584 scope.go:117] "RemoveContainer" containerID="ca7f6977de1af6cc502af92e50981fea6b6a8a3dc83d88a934d115ccfb2439cd"
Jul 17 07:42:54.146059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:54.145877    2584 scope.go:117] "RemoveContainer" containerID="6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67"
Jul 17 07:42:54.146171 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:54.146086    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw_odh-ai-gateway-infra(10b70083-0ec6-4996-a2a5-91ab190af85b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b"
Jul 17 07:42:55.151626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:55.151596    2584 scope.go:117] "RemoveContainer" containerID="6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67"
Jul 17 07:42:55.152021 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:55.151768    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw_odh-ai-gateway-infra(10b70083-0ec6-4996-a2a5-91ab190af85b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b"
Jul 17 07:42:55.285916 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:55.285880    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:56.159476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:56.159443    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189" exitCode=1
Jul 17 07:42:56.159476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:56.159489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"}
Jul 17 07:42:56.160046 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:56.159545    2584 scope.go:117] "RemoveContainer" containerID="b7a7e27d5fa8386c74b8b4ba706116e1d6c760583249894989f652734ed2455c"
Jul 17 07:42:56.160046 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:56.159901    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:42:56.160154 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:56.160133    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:42:59.916453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:59.916413    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:42:59.916951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:42:59.916775    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:42:59.916951 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:42:59.916927    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:01.664358 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:01.664308    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:43:01.664754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:01.664740    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:43:01.664925 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:01.664905    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:08.285261 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.285189    2584 scope.go:117] "RemoveContainer" containerID="6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67"
Jul 17 07:43:08.465963 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.465941    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:08.466299 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.466287    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b252f65b-937f-4e6b-9269-0ae12fa44e5a" containerName="revoke-keys"
Jul 17 07:43:08.469236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.469221    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:08.476639 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.476610    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:08.551757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.551698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr\" (UID: \"cda60597-0789-434c-86c7-33dc4a6f3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:08.652712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.652683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr\" (UID: \"cda60597-0789-434c-86c7-33dc4a6f3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:08.653353 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.653310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr\" (UID: \"cda60597-0789-434c-86c7-33dc4a6f3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:08.779851 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:08.779820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:09.115492 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.115413    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:09.118283 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:43:09.118250    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda60597_0789_434c_86c7_33dc4a6f3f80.slice/crio-8f848bfa5417260b6ce12f3060cb1f1f7016b1158f12b75e812dd620fb216902 WatchSource:0}: Error finding container 8f848bfa5417260b6ce12f3060cb1f1f7016b1158f12b75e812dd620fb216902: Status 404 returned error can't find the container with id 8f848bfa5417260b6ce12f3060cb1f1f7016b1158f12b75e812dd620fb216902
Jul 17 07:43:09.217001 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.216964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" event={"ID":"cda60597-0789-434c-86c7-33dc4a6f3f80","Type":"ContainerStarted","Data":"8f848bfa5417260b6ce12f3060cb1f1f7016b1158f12b75e812dd620fb216902"}
Jul 17 07:43:09.218777 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.218749    2584 generic.go:356] "Generic (PLEG): container finished" podID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerID="a0f51a4a2969671b44207868a81b74348a1cdd15b2d54ea4b20e2c7c0005f7bf" exitCode=6
Jul 17 07:43:09.218904 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.218788    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerDied","Data":"a0f51a4a2969671b44207868a81b74348a1cdd15b2d54ea4b20e2c7c0005f7bf"}
Jul 17 07:43:09.218904 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.218828    2584 scope.go:117] "RemoveContainer" containerID="6fd08be8f6bd37407f2fef9771fed5bb1910c428e7efb19ee704243ad5777d67"
Jul 17 07:43:09.219128 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:09.219106    2584 scope.go:117] "RemoveContainer" containerID="a0f51a4a2969671b44207868a81b74348a1cdd15b2d54ea4b20e2c7c0005f7bf"
Jul 17 07:43:09.219385 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:09.219318    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw_odh-ai-gateway-infra(10b70083-0ec6-4996-a2a5-91ab190af85b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b"
Jul 17 07:43:10.224262 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.224227    2584 generic.go:356] "Generic (PLEG): container finished" podID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerID="67a5e57eefc84c71fec458a9176f41311061a1a5455a730325e38e52bfb81ffd" exitCode=6
Jul 17 07:43:10.224749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.224317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" event={"ID":"cda60597-0789-434c-86c7-33dc4a6f3f80","Type":"ContainerDied","Data":"67a5e57eefc84c71fec458a9176f41311061a1a5455a730325e38e52bfb81ffd"}
Jul 17 07:43:10.224749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.224550    2584 scope.go:117] "RemoveContainer" containerID="67a5e57eefc84c71fec458a9176f41311061a1a5455a730325e38e52bfb81ffd"
Jul 17 07:43:10.245428 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.245401    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:43:10.377151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.377126    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:43:10.468182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.468154    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca\") pod \"10b70083-0ec6-4996-a2a5-91ab190af85b\" (UID: \"10b70083-0ec6-4996-a2a5-91ab190af85b\") "
Jul 17 07:43:10.468501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.468482    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10b70083-0ec6-4996-a2a5-91ab190af85b" (UID: "10b70083-0ec6-4996-a2a5-91ab190af85b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:43:10.569152 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:10.569132    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b70083-0ec6-4996-a2a5-91ab190af85b-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:43:11.231933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.231842    2584 generic.go:356] "Generic (PLEG): container finished" podID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerID="9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e" exitCode=6
Jul 17 07:43:11.231933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.231896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" event={"ID":"cda60597-0789-434c-86c7-33dc4a6f3f80","Type":"ContainerDied","Data":"9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e"}
Jul 17 07:43:11.232441 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.231965    2584 scope.go:117] "RemoveContainer" containerID="67a5e57eefc84c71fec458a9176f41311061a1a5455a730325e38e52bfb81ffd"
Jul 17 07:43:11.232441 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.232161    2584 scope.go:117] "RemoveContainer" containerID="9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e"
Jul 17 07:43:11.232441 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:11.232410    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr_odh-ai-gateway-infra(cda60597-0789-434c-86c7-33dc4a6f3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80"
Jul 17 07:43:11.233153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.233133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw" event={"ID":"10b70083-0ec6-4996-a2a5-91ab190af85b","Type":"ContainerDied","Data":"b1cd2f10d1bf536560b4494f89fa60cc171b89fc8663844f92385037a6efc5e0"}
Jul 17 07:43:11.233153 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.233147    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"
Jul 17 07:43:11.242717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.242701    2584 scope.go:117] "RemoveContainer" containerID="a0f51a4a2969671b44207868a81b74348a1cdd15b2d54ea4b20e2c7c0005f7bf"
Jul 17 07:43:11.263189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.263166    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:43:11.269127 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:11.269107    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-xm7gw"]
Jul 17 07:43:12.239582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:12.239553    2584 scope.go:117] "RemoveContainer" containerID="9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e"
Jul 17 07:43:12.240010 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:12.239771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr_odh-ai-gateway-infra(cda60597-0789-434c-86c7-33dc4a6f3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80"
Jul 17 07:43:12.290284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:12.290248    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" path="/var/lib/kubelet/pods/10b70083-0ec6-4996-a2a5-91ab190af85b/volumes"
Jul 17 07:43:15.284811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:15.284783    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:43:15.285163 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:15.284924    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:25.285221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:25.285193    2584 scope.go:117] "RemoveContainer" containerID="9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e"
Jul 17 07:43:26.297242 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:26.297208    2584 generic.go:356] "Generic (PLEG): container finished" podID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerID="dd37186eea7de28b60b104d8fe37be614ede05eb7a917a86b1a0558084ff05ae" exitCode=6
Jul 17 07:43:26.297706 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:26.297260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" event={"ID":"cda60597-0789-434c-86c7-33dc4a6f3f80","Type":"ContainerDied","Data":"dd37186eea7de28b60b104d8fe37be614ede05eb7a917a86b1a0558084ff05ae"}
Jul 17 07:43:26.297706 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:26.297305    2584 scope.go:117] "RemoveContainer" containerID="9622b3d14490ea644b5fdb09e3b8c37aa468ec1d688201d2b5ce118dfd53d85e"
Jul 17 07:43:26.297706 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:26.297619    2584 scope.go:117] "RemoveContainer" containerID="dd37186eea7de28b60b104d8fe37be614ede05eb7a917a86b1a0558084ff05ae"
Jul 17 07:43:26.297844 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:26.297823    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr_odh-ai-gateway-infra(cda60597-0789-434c-86c7-33dc4a6f3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80"
Jul 17 07:43:27.323317 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:27.323284    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:27.449540 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:27.449517    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:27.616041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:27.615958    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca\") pod \"cda60597-0789-434c-86c7-33dc4a6f3f80\" (UID: \"cda60597-0789-434c-86c7-33dc4a6f3f80\") "
Jul 17 07:43:27.616326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:27.616306    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cda60597-0789-434c-86c7-33dc4a6f3f80" (UID: "cda60597-0789-434c-86c7-33dc4a6f3f80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:43:27.716732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:27.716704    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda60597-0789-434c-86c7-33dc4a6f3f80-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:43:28.307226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:28.307200    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"
Jul 17 07:43:28.307372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:28.307240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr" event={"ID":"cda60597-0789-434c-86c7-33dc4a6f3f80","Type":"ContainerDied","Data":"8f848bfa5417260b6ce12f3060cb1f1f7016b1158f12b75e812dd620fb216902"}
Jul 17 07:43:28.307372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:28.307294    2584 scope.go:117] "RemoveContainer" containerID="dd37186eea7de28b60b104d8fe37be614ede05eb7a917a86b1a0558084ff05ae"
Jul 17 07:43:28.325113 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:28.325089    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:28.334222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:28.334197    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-b6mnr"]
Jul 17 07:43:29.285250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:29.285221    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:43:29.285427 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:29.285407    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:30.289676 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:30.289642    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" path="/var/lib/kubelet/pods/cda60597-0789-434c-86c7-33dc4a6f3f80/volumes"
Jul 17 07:43:43.285154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:43.285118    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:43:43.285645 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:43.285303    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:54.286784 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:54.286747    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:43:54.287159 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:43:54.286959    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:43:58.508122 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508086    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508419    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508430    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508442    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508448    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508456    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508462    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508469    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508474    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508484    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508490    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508537    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508545    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:43:58.508808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508551    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508557    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.508808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.508561    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b70083-0ec6-4996-a2a5-91ab190af85b" containerName="revoke-keys"
Jul 17 07:43:58.511484 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.511467    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:43:58.514022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.514005    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:43:58.518602 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.518162    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:43:58.561216 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.561185    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-kthln\" (UID: \"586e6aa0-b405-49a6-a075-a36a8d65852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:43:58.662402 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.662329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-kthln\" (UID: \"586e6aa0-b405-49a6-a075-a36a8d65852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:43:58.662992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.662971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-kthln\" (UID: \"586e6aa0-b405-49a6-a075-a36a8d65852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:43:58.822904 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.822869    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:43:58.960191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:58.960172    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:43:58.965120 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:43:58.965067    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586e6aa0_b405_49a6_a075_a36a8d65852a.slice/crio-2b7c1bbae244e0f6defbf1dd1d62a601e013c3c6492248ab6aa99ca90d7c32fd WatchSource:0}: Error finding container 2b7c1bbae244e0f6defbf1dd1d62a601e013c3c6492248ab6aa99ca90d7c32fd: Status 404 returned error can't find the container with id 2b7c1bbae244e0f6defbf1dd1d62a601e013c3c6492248ab6aa99ca90d7c32fd
Jul 17 07:43:59.448705 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:59.448607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerStarted","Data":"f531047913488f5c3d8ba5c5e717447a548c8799727411020ad728413507cff3"}
Jul 17 07:43:59.448705 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:59.448661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerStarted","Data":"2b7c1bbae244e0f6defbf1dd1d62a601e013c3c6492248ab6aa99ca90d7c32fd"}
Jul 17 07:43:59.464873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:43:59.464825    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" podStartSLOduration=1.464809827 podStartE2EDuration="1.464809827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:43:59.464114899 +0000 UTC m=+1935.717537711" watchObservedRunningTime="2026-07-17 07:43:59.464809827 +0000 UTC m=+1935.718232636"
Jul 17 07:44:00.454206 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:00.454173    2584 generic.go:356] "Generic (PLEG): container finished" podID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerID="f531047913488f5c3d8ba5c5e717447a548c8799727411020ad728413507cff3" exitCode=6
Jul 17 07:44:00.454660 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:00.454259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerDied","Data":"f531047913488f5c3d8ba5c5e717447a548c8799727411020ad728413507cff3"}
Jul 17 07:44:00.454660 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:00.454433    2584 scope.go:117] "RemoveContainer" containerID="f531047913488f5c3d8ba5c5e717447a548c8799727411020ad728413507cff3"
Jul 17 07:44:01.460041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:01.460004    2584 generic.go:356] "Generic (PLEG): container finished" podID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerID="5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105" exitCode=6
Jul 17 07:44:01.460041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:01.460041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerDied","Data":"5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105"}
Jul 17 07:44:01.460502 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:01.460096    2584 scope.go:117] "RemoveContainer" containerID="f531047913488f5c3d8ba5c5e717447a548c8799727411020ad728413507cff3"
Jul 17 07:44:01.460502 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:01.460258    2584 scope.go:117] "RemoveContainer" containerID="5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105"
Jul 17 07:44:01.460502 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:01.460486    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-kthln_odh-ai-gateway-infra(586e6aa0-b405-49a6-a075-a36a8d65852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a"
Jul 17 07:44:02.465751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:02.465721    2584 scope.go:117] "RemoveContainer" containerID="5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105"
Jul 17 07:44:02.466229 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:02.465961    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-kthln_odh-ai-gateway-infra(586e6aa0-b405-49a6-a075-a36a8d65852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a"
Jul 17 07:44:08.284868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:08.284831    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:44:08.285363 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:08.285006    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:15.103554 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.103511    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:15.104136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.103978    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:44:15.104136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.103998    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:44:15.104136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.104111    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda60597-0789-434c-86c7-33dc4a6f3f80" containerName="revoke-keys"
Jul 17 07:44:15.108416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.108395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:15.122006 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.121979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:15.197484 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.197448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q\" (UID: \"d2601e8b-fe47-4be7-8dca-6283d44379bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:15.298666 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.298631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q\" (UID: \"d2601e8b-fe47-4be7-8dca-6283d44379bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:15.299232 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.299213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q\" (UID: \"d2601e8b-fe47-4be7-8dca-6283d44379bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:15.417433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.417354    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:15.544880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:15.544845    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:15.547828 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:44:15.547802    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2601e8b_fe47_4be7_8dca_6283d44379bc.slice/crio-9e6939c11e0251309aed69cf1752490fc821252f62e2f73a8818427d1a7fa7d0 WatchSource:0}: Error finding container 9e6939c11e0251309aed69cf1752490fc821252f62e2f73a8818427d1a7fa7d0: Status 404 returned error can't find the container with id 9e6939c11e0251309aed69cf1752490fc821252f62e2f73a8818427d1a7fa7d0
Jul 17 07:44:16.285605 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:16.285567    2584 scope.go:117] "RemoveContainer" containerID="5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105"
Jul 17 07:44:16.524149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:16.524114    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerID="519b0b50320e6e51b854d8770c46e54e1f931f2c2c4d1cf4d9bc87c83346d01a" exitCode=7
Jul 17 07:44:16.524325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:16.524182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerDied","Data":"519b0b50320e6e51b854d8770c46e54e1f931f2c2c4d1cf4d9bc87c83346d01a"}
Jul 17 07:44:16.524325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:16.524225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerStarted","Data":"9e6939c11e0251309aed69cf1752490fc821252f62e2f73a8818427d1a7fa7d0"}
Jul 17 07:44:16.524426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:16.524414    2584 scope.go:117] "RemoveContainer" containerID="519b0b50320e6e51b854d8770c46e54e1f931f2c2c4d1cf4d9bc87c83346d01a"
Jul 17 07:44:17.530324 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.530290    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerID="57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e" exitCode=7
Jul 17 07:44:17.530819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.530371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerDied","Data":"57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e"}
Jul 17 07:44:17.530819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.530466    2584 scope.go:117] "RemoveContainer" containerID="519b0b50320e6e51b854d8770c46e54e1f931f2c2c4d1cf4d9bc87c83346d01a"
Jul 17 07:44:17.530819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.530675    2584 scope.go:117] "RemoveContainer" containerID="57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e"
Jul 17 07:44:17.530984 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:17.530882    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q_odh-ai-gateway-infra(d2601e8b-fe47-4be7-8dca-6283d44379bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc"
Jul 17 07:44:17.532175 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.532158    2584 generic.go:356] "Generic (PLEG): container finished" podID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerID="7c485ec69c1b50c3f6acc8dce30363afaea81a0fde90d578b360415dbed299a1" exitCode=6
Jul 17 07:44:17.532257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.532234    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerDied","Data":"7c485ec69c1b50c3f6acc8dce30363afaea81a0fde90d578b360415dbed299a1"}
Jul 17 07:44:17.532519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.532506    2584 scope.go:117] "RemoveContainer" containerID="7c485ec69c1b50c3f6acc8dce30363afaea81a0fde90d578b360415dbed299a1"
Jul 17 07:44:17.532655 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:17.532639    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-kthln_odh-ai-gateway-infra(586e6aa0-b405-49a6-a075-a36a8d65852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a"
Jul 17 07:44:17.540741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:17.540724    2584 scope.go:117] "RemoveContainer" containerID="5884b8c1ee0ed16c933dfb3745fea018a4f9581b93ea68f7fd66ca1b4e075105"
Jul 17 07:44:18.539626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.539598    2584 scope.go:117] "RemoveContainer" containerID="57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e"
Jul 17 07:44:18.540063 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:18.539798    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q_odh-ai-gateway-infra(d2601e8b-fe47-4be7-8dca-6283d44379bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc"
Jul 17 07:44:18.571220 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.571193    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:44:18.699936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.699912    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:44:18.831451 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.831361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca\") pod \"586e6aa0-b405-49a6-a075-a36a8d65852a\" (UID: \"586e6aa0-b405-49a6-a075-a36a8d65852a\") "
Jul 17 07:44:18.831713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.831691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "586e6aa0-b405-49a6-a075-a36a8d65852a" (UID: "586e6aa0-b405-49a6-a075-a36a8d65852a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:44:18.932906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:18.932879    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/586e6aa0-b405-49a6-a075-a36a8d65852a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:44:19.546054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:19.546021    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"
Jul 17 07:44:19.546054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:19.546034    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln" event={"ID":"586e6aa0-b405-49a6-a075-a36a8d65852a","Type":"ContainerDied","Data":"2b7c1bbae244e0f6defbf1dd1d62a601e013c3c6492248ab6aa99ca90d7c32fd"}
Jul 17 07:44:19.546577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:19.546087    2584 scope.go:117] "RemoveContainer" containerID="7c485ec69c1b50c3f6acc8dce30363afaea81a0fde90d578b360415dbed299a1"
Jul 17 07:44:19.567421 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:19.567393    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:44:19.572926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:19.572906    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-kthln"]
Jul 17 07:44:20.290170 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:20.290138    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" path="/var/lib/kubelet/pods/586e6aa0-b405-49a6-a075-a36a8d65852a/volumes"
Jul 17 07:44:21.284859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:21.284829    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:44:22.561257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:22.561226    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8" exitCode=1
Jul 17 07:44:22.561757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:22.561272    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"}
Jul 17 07:44:22.561757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:22.561317    2584 scope.go:117] "RemoveContainer" containerID="f731e4c3073f34495c37a027fd23f858271fbb9417e2977db0f72a09ba5b4189"
Jul 17 07:44:22.561757 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:22.561682    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:44:22.561939 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:22.561916    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:24.916722 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:24.916688    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:44:24.917080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:24.917036    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:44:24.917221 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:24.917202    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:31.285043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:31.285013    2584 scope.go:117] "RemoveContainer" containerID="57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e"
Jul 17 07:44:31.603547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:31.603460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerStarted","Data":"6fb9d359601d5e26aa031715ff71c16ba208b02eb51e33f909d3ba180b277843"}
Jul 17 07:44:31.620673 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:31.620624    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" podStartSLOduration=16.620611985 podStartE2EDuration="16.620611985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:44:31.619602953 +0000 UTC m=+1967.873025767" watchObservedRunningTime="2026-07-17 07:44:31.620611985 +0000 UTC m=+1967.874034795"
Jul 17 07:44:31.664284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:31.664257    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:44:31.664614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:31.664601    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:44:31.664769 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:31.664753    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:32.608790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:32.608755    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerID="6fb9d359601d5e26aa031715ff71c16ba208b02eb51e33f909d3ba180b277843" exitCode=7
Jul 17 07:44:32.609205 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:32.608828    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerDied","Data":"6fb9d359601d5e26aa031715ff71c16ba208b02eb51e33f909d3ba180b277843"}
Jul 17 07:44:32.609205 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:32.608888    2584 scope.go:117] "RemoveContainer" containerID="57b3120e9ce0cad4b03c0a54c444d073b4067028b289290d3ff337f0e43efe1e"
Jul 17 07:44:32.609205 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:32.609190    2584 scope.go:117] "RemoveContainer" containerID="6fb9d359601d5e26aa031715ff71c16ba208b02eb51e33f909d3ba180b277843"
Jul 17 07:44:32.609436 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:32.609415    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q_odh-ai-gateway-infra(d2601e8b-fe47-4be7-8dca-6283d44379bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc"
Jul 17 07:44:32.635724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:32.635697    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:33.738365 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:33.738325    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:33.845505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:33.845478    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca\") pod \"d2601e8b-fe47-4be7-8dca-6283d44379bc\" (UID: \"d2601e8b-fe47-4be7-8dca-6283d44379bc\") "
Jul 17 07:44:33.845794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:33.845773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2601e8b-fe47-4be7-8dca-6283d44379bc" (UID: "d2601e8b-fe47-4be7-8dca-6283d44379bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:44:33.946962 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:33.946878    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2601e8b-fe47-4be7-8dca-6283d44379bc-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:44:34.619038 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:34.618951    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"
Jul 17 07:44:34.619038 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:34.619004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q" event={"ID":"d2601e8b-fe47-4be7-8dca-6283d44379bc","Type":"ContainerDied","Data":"9e6939c11e0251309aed69cf1752490fc821252f62e2f73a8818427d1a7fa7d0"}
Jul 17 07:44:34.619289 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:34.619073    2584 scope.go:117] "RemoveContainer" containerID="6fb9d359601d5e26aa031715ff71c16ba208b02eb51e33f909d3ba180b277843"
Jul 17 07:44:34.636069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:34.636049    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:34.642511 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:34.642476    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-82r2q"]
Jul 17 07:44:36.289932 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:36.289890    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" path="/var/lib/kubelet/pods/d2601e8b-fe47-4be7-8dca-6283d44379bc/volumes"
Jul 17 07:44:44.287177 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:44.287143    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:44:44.287691 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:44.287317    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:51.322744 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.322677    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:44:51.323222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323191    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323210    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323228    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323237    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323248    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323257    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323282    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323290    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323299    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323308    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323320    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323329    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323432    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323448    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.323833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323457    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323467    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:51.323833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.323477    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="586e6aa0-b405-49a6-a075-a36a8d65852a" containerName="revoke-keys"
Jul 17 07:44:51.326536 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.326512    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:44:51.329132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.329109    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:44:51.332824 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.332800    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:44:51.391125 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.391077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8\" (UID: \"f7576b71-d3fb-485f-8e29-c1eb403a67a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:44:51.491846 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.491804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8\" (UID: \"f7576b71-d3fb-485f-8e29-c1eb403a67a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:44:51.492461 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.492440    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8\" (UID: \"f7576b71-d3fb-485f-8e29-c1eb403a67a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:44:51.637908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.637821    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:44:51.775332 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:51.775293    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:44:51.777701 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:44:51.777678    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7576b71_d3fb_485f_8e29_c1eb403a67a5.slice/crio-ff6f7e2c8e477c429fcf566d17a0c9684755ac53069458069bfed3a1a4fc14e5 WatchSource:0}: Error finding container ff6f7e2c8e477c429fcf566d17a0c9684755ac53069458069bfed3a1a4fc14e5: Status 404 returned error can't find the container with id ff6f7e2c8e477c429fcf566d17a0c9684755ac53069458069bfed3a1a4fc14e5
Jul 17 07:44:52.697221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:52.697120    2584 generic.go:356] "Generic (PLEG): container finished" podID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerID="05067ee383e55bf2eb15866b465923ded97c8e98248adde2399357132d0324c9" exitCode=6
Jul 17 07:44:52.697633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:52.697206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerDied","Data":"05067ee383e55bf2eb15866b465923ded97c8e98248adde2399357132d0324c9"}
Jul 17 07:44:52.697633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:52.697271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerStarted","Data":"ff6f7e2c8e477c429fcf566d17a0c9684755ac53069458069bfed3a1a4fc14e5"}
Jul 17 07:44:52.697633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:52.697390    2584 scope.go:117] "RemoveContainer" containerID="05067ee383e55bf2eb15866b465923ded97c8e98248adde2399357132d0324c9"
Jul 17 07:44:53.704020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:53.703933    2584 generic.go:356] "Generic (PLEG): container finished" podID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerID="38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137" exitCode=6
Jul 17 07:44:53.704020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:53.704001    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerDied","Data":"38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137"}
Jul 17 07:44:53.704492 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:53.704069    2584 scope.go:117] "RemoveContainer" containerID="05067ee383e55bf2eb15866b465923ded97c8e98248adde2399357132d0324c9"
Jul 17 07:44:53.704492 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:53.704253    2584 scope.go:117] "RemoveContainer" containerID="38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137"
Jul 17 07:44:53.704582 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:53.704502    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8_odh-ai-gateway-infra(f7576b71-d3fb-485f-8e29-c1eb403a67a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5"
Jul 17 07:44:54.710293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:54.710265    2584 scope.go:117] "RemoveContainer" containerID="38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137"
Jul 17 07:44:54.710807 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:54.710530    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8_odh-ai-gateway-infra(f7576b71-d3fb-485f-8e29-c1eb403a67a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5"
Jul 17 07:44:58.285387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.285332    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:44:58.285761 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:44:58.285523    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:44:58.554584 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.554502    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:44:58.554911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.554898    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2601e8b-fe47-4be7-8dca-6283d44379bc" containerName="revoke-keys"
Jul 17 07:44:58.559088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.559064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:44:58.565082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.565047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:44:58.653784 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.653744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c\" (UID: \"838f9182-2911-4fa0-bec7-efbe4d37c8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:44:58.754786 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.754751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c\" (UID: \"838f9182-2911-4fa0-bec7-efbe4d37c8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:44:58.755358 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.755317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c\" (UID: \"838f9182-2911-4fa0-bec7-efbe4d37c8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:44:58.872076 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:58.871982    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:44:59.005109 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:59.005078    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:44:59.007459 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:44:59.007429    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f9182_2911_4fa0_bec7_efbe4d37c8ee.slice/crio-3d409efe5efc8c9c31bfdfedf8eaa5f60206297956a565ec4bdd11398185ce5d WatchSource:0}: Error finding container 3d409efe5efc8c9c31bfdfedf8eaa5f60206297956a565ec4bdd11398185ce5d: Status 404 returned error can't find the container with id 3d409efe5efc8c9c31bfdfedf8eaa5f60206297956a565ec4bdd11398185ce5d
Jul 17 07:44:59.737849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:59.737813    2584 generic.go:356] "Generic (PLEG): container finished" podID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerID="670848f1d95abe225ceac0749826d247c7a8cc926127698f861e0c770afee15b" exitCode=6
Jul 17 07:44:59.738236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:59.737897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerDied","Data":"670848f1d95abe225ceac0749826d247c7a8cc926127698f861e0c770afee15b"}
Jul 17 07:44:59.738236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:59.737951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerStarted","Data":"3d409efe5efc8c9c31bfdfedf8eaa5f60206297956a565ec4bdd11398185ce5d"}
Jul 17 07:44:59.738236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:44:59.738078    2584 scope.go:117] "RemoveContainer" containerID="670848f1d95abe225ceac0749826d247c7a8cc926127698f861e0c770afee15b"
Jul 17 07:45:00.189132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.189057    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"]
Jul 17 07:45:00.192327 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.192303    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"
Jul 17 07:45:00.201563 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.201540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"
Jul 17 07:45:00.232204 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.232167    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"]
Jul 17 07:45:00.354326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.354296    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"]
Jul 17 07:45:00.389280 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:45:00.389239    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ec5d82_16ae_449d_b61b_f00aa11d829b.slice/crio-fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da WatchSource:0}: Error finding container fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da: Status 404 returned error can't find the container with id fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da
Jul 17 07:45:00.744036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.744003    2584 generic.go:356] "Generic (PLEG): container finished" podID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerID="de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690" exitCode=6
Jul 17 07:45:00.744511 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.744075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerDied","Data":"de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690"}
Jul 17 07:45:00.744511 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.744127    2584 scope.go:117] "RemoveContainer" containerID="670848f1d95abe225ceac0749826d247c7a8cc926127698f861e0c770afee15b"
Jul 17 07:45:00.744511 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.744328    2584 scope.go:117] "RemoveContainer" containerID="de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690"
Jul 17 07:45:00.744646 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:00.744582    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c_odh-ai-gateway-infra(838f9182-2911-4fa0-bec7-efbe4d37c8ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee"
Jul 17 07:45:00.745713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.745690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n" event={"ID":"a0ec5d82-16ae-449d-b61b-f00aa11d829b","Type":"ContainerStarted","Data":"51e74823b033081153e4c20bb8a7939c870d65df69d26f5cba7f644521d5da45"}
Jul 17 07:45:00.745811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.745742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n" event={"ID":"a0ec5d82-16ae-449d-b61b-f00aa11d829b","Type":"ContainerStarted","Data":"fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da"}
Jul 17 07:45:00.801462 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:00.801407    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n" podStartSLOduration=0.801390916 podStartE2EDuration="801.390916ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:45:00.800042396 +0000 UTC m=+1997.053465208" watchObservedRunningTime="2026-07-17 07:45:00.801390916 +0000 UTC m=+1997.054813706"
Jul 17 07:45:01.751891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:01.751855    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0ec5d82-16ae-449d-b61b-f00aa11d829b" containerID="51e74823b033081153e4c20bb8a7939c870d65df69d26f5cba7f644521d5da45" exitCode=0
Jul 17 07:45:01.752326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:01.751940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n" event={"ID":"a0ec5d82-16ae-449d-b61b-f00aa11d829b","Type":"ContainerDied","Data":"51e74823b033081153e4c20bb8a7939c870d65df69d26f5cba7f644521d5da45"}
Jul 17 07:45:01.753571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:01.753553    2584 scope.go:117] "RemoveContainer" containerID="de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690"
Jul 17 07:45:01.753739 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:01.753723    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c_odh-ai-gateway-infra(838f9182-2911-4fa0-bec7-efbe4d37c8ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee"
Jul 17 07:45:02.886800 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:02.886769    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"
Jul 17 07:45:03.763745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:03.763710    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n"
Jul 17 07:45:03.763745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:03.763736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737905-b544n" event={"ID":"a0ec5d82-16ae-449d-b61b-f00aa11d829b","Type":"ContainerDied","Data":"fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da"}
Jul 17 07:45:03.763966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:03.763788    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb37c4422614401644f34ccff30e3bdb7d1e94db1ea2c00c4e2ec30e3116f7da"
Jul 17 07:45:05.131240 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.131210    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:05.131625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.131550    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ec5d82-16ae-449d-b61b-f00aa11d829b" containerName="cleanup"
Jul 17 07:45:05.131625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.131562    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ec5d82-16ae-449d-b61b-f00aa11d829b" containerName="cleanup"
Jul 17 07:45:05.131700 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.131629    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ec5d82-16ae-449d-b61b-f00aa11d829b" containerName="cleanup"
Jul 17 07:45:05.134605 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.134589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:05.143840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.143812    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:05.213524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.213476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd\" (UID: \"4cc72686-f4b3-4ae0-9f2c-52529144e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:05.314857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.314799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd\" (UID: \"4cc72686-f4b3-4ae0-9f2c-52529144e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:05.315476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.315455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd\" (UID: \"4cc72686-f4b3-4ae0-9f2c-52529144e810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:05.445664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.445574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:05.583959 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.583926    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:05.587420 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:45:05.587392    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc72686_f4b3_4ae0_9f2c_52529144e810.slice/crio-2839620d6b7221a8184e15eea9a0e92531799db8434fa366e401cfc2d1791862 WatchSource:0}: Error finding container 2839620d6b7221a8184e15eea9a0e92531799db8434fa366e401cfc2d1791862: Status 404 returned error can't find the container with id 2839620d6b7221a8184e15eea9a0e92531799db8434fa366e401cfc2d1791862
Jul 17 07:45:05.773688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:05.773600    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerStarted","Data":"2839620d6b7221a8184e15eea9a0e92531799db8434fa366e401cfc2d1791862"}
Jul 17 07:45:06.285591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.285562    2584 scope.go:117] "RemoveContainer" containerID="38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137"
Jul 17 07:45:06.780016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.779933    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerID="9dcd9a75adcfa586f7fa631ec124f92e9d515f59acad8f6815d3d507c0ffd946" exitCode=7
Jul 17 07:45:06.780178 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.780019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerDied","Data":"9dcd9a75adcfa586f7fa631ec124f92e9d515f59acad8f6815d3d507c0ffd946"}
Jul 17 07:45:06.780226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.780207    2584 scope.go:117] "RemoveContainer" containerID="9dcd9a75adcfa586f7fa631ec124f92e9d515f59acad8f6815d3d507c0ffd946"
Jul 17 07:45:06.781921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.781898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerStarted","Data":"0a5a72308a19e2022d6214bcccd26b39ee5d3df513bfd7034b9fa45124dc7049"}
Jul 17 07:45:06.813243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:06.813185    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" podStartSLOduration=15.813167009 podStartE2EDuration="15.813167009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:45:06.81098805 +0000 UTC m=+2003.064410862" watchObservedRunningTime="2026-07-17 07:45:06.813167009 +0000 UTC m=+2003.066589851"
Jul 17 07:45:07.788267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.788232    2584 generic.go:356] "Generic (PLEG): container finished" podID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerID="0a5a72308a19e2022d6214bcccd26b39ee5d3df513bfd7034b9fa45124dc7049" exitCode=6
Jul 17 07:45:07.788749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.788300    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerDied","Data":"0a5a72308a19e2022d6214bcccd26b39ee5d3df513bfd7034b9fa45124dc7049"}
Jul 17 07:45:07.788749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.788375    2584 scope.go:117] "RemoveContainer" containerID="38168839970ddb0e6f72dcaaeb45a8fbb973cdff7ed35ae97bbcd48916ef8137"
Jul 17 07:45:07.788749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.788684    2584 scope.go:117] "RemoveContainer" containerID="0a5a72308a19e2022d6214bcccd26b39ee5d3df513bfd7034b9fa45124dc7049"
Jul 17 07:45:07.788923 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:07.788902    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8_odh-ai-gateway-infra(f7576b71-d3fb-485f-8e29-c1eb403a67a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5"
Jul 17 07:45:07.790263 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.790241    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerID="cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af" exitCode=7
Jul 17 07:45:07.790379 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.790284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerDied","Data":"cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af"}
Jul 17 07:45:07.790541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.790527    2584 scope.go:117] "RemoveContainer" containerID="cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af"
Jul 17 07:45:07.790678 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:07.790661    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd_odh-ai-gateway-infra(4cc72686-f4b3-4ae0-9f2c-52529144e810)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810"
Jul 17 07:45:07.798975 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.798956    2584 scope.go:117] "RemoveContainer" containerID="9dcd9a75adcfa586f7fa631ec124f92e9d515f59acad8f6815d3d507c0ffd946"
Jul 17 07:45:07.828597 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:07.828571    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:45:08.254761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:08.254674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:45:08.342951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:08.342914    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:45:08.797987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:08.797967    2584 scope.go:117] "RemoveContainer" containerID="cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af"
Jul 17 07:45:08.798291 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:08.798130    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd_odh-ai-gateway-infra(4cc72686-f4b3-4ae0-9f2c-52529144e810)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810"
Jul 17 07:45:08.930410 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:08.930383    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:45:09.043760 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.043729    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca\") pod \"f7576b71-d3fb-485f-8e29-c1eb403a67a5\" (UID: \"f7576b71-d3fb-485f-8e29-c1eb403a67a5\") "
Jul 17 07:45:09.044094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.044074    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7576b71-d3fb-485f-8e29-c1eb403a67a5" (UID: "f7576b71-d3fb-485f-8e29-c1eb403a67a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:45:09.144711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.144624    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7576b71-d3fb-485f-8e29-c1eb403a67a5-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:45:09.803260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.803233    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"
Jul 17 07:45:09.803260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.803251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8" event={"ID":"f7576b71-d3fb-485f-8e29-c1eb403a67a5","Type":"ContainerDied","Data":"ff6f7e2c8e477c429fcf566d17a0c9684755ac53069458069bfed3a1a4fc14e5"}
Jul 17 07:45:09.803748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.803306    2584 scope.go:117] "RemoveContainer" containerID="0a5a72308a19e2022d6214bcccd26b39ee5d3df513bfd7034b9fa45124dc7049"
Jul 17 07:45:09.837280 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.837248    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:45:09.840376 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:09.840351    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-fpnj8"]
Jul 17 07:45:10.289829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:10.289801    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" path="/var/lib/kubelet/pods/f7576b71-d3fb-485f-8e29-c1eb403a67a5/volumes"
Jul 17 07:45:11.285034 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:11.284999    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:45:11.285425 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:11.285138    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:45:13.284688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:13.284659    2584 scope.go:117] "RemoveContainer" containerID="de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690"
Jul 17 07:45:13.823556 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:13.823365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerStarted","Data":"8ae309b29be69b21eba1efb1f4db528dcb3d39c6209d667ad7e426dcb9facc9f"}
Jul 17 07:45:13.840636 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:13.840592    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" podStartSLOduration=15.840578155 podStartE2EDuration="15.840578155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:45:13.840099781 +0000 UTC m=+2010.093522615" watchObservedRunningTime="2026-07-17 07:45:13.840578155 +0000 UTC m=+2010.094000966"
Jul 17 07:45:14.828710 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:14.828681    2584 generic.go:356] "Generic (PLEG): container finished" podID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerID="8ae309b29be69b21eba1efb1f4db528dcb3d39c6209d667ad7e426dcb9facc9f" exitCode=6
Jul 17 07:45:14.829190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:14.828736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerDied","Data":"8ae309b29be69b21eba1efb1f4db528dcb3d39c6209d667ad7e426dcb9facc9f"}
Jul 17 07:45:14.829190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:14.828781    2584 scope.go:117] "RemoveContainer" containerID="de5f8411a8f3037ec9f380da7025fdd0190fe38ee6544e9d1a0066f256ba8690"
Jul 17 07:45:14.829190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:14.829036    2584 scope.go:117] "RemoveContainer" containerID="8ae309b29be69b21eba1efb1f4db528dcb3d39c6209d667ad7e426dcb9facc9f"
Jul 17 07:45:14.829369 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:14.829225    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c_odh-ai-gateway-infra(838f9182-2911-4fa0-bec7-efbe4d37c8ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee"
Jul 17 07:45:14.863361 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:14.860762    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:45:15.969769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:15.969744    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:45:16.104693 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.104591    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca\") pod \"838f9182-2911-4fa0-bec7-efbe4d37c8ee\" (UID: \"838f9182-2911-4fa0-bec7-efbe4d37c8ee\") "
Jul 17 07:45:16.104958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.104933    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "838f9182-2911-4fa0-bec7-efbe4d37c8ee" (UID: "838f9182-2911-4fa0-bec7-efbe4d37c8ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:45:16.205698 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.205662    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/838f9182-2911-4fa0-bec7-efbe4d37c8ee-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:45:16.696261 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.696226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:45:16.794393 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.794334    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:45:16.842222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.842191    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"
Jul 17 07:45:16.842413 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.842194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c" event={"ID":"838f9182-2911-4fa0-bec7-efbe4d37c8ee","Type":"ContainerDied","Data":"3d409efe5efc8c9c31bfdfedf8eaa5f60206297956a565ec4bdd11398185ce5d"}
Jul 17 07:45:16.842413 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.842328    2584 scope.go:117] "RemoveContainer" containerID="8ae309b29be69b21eba1efb1f4db528dcb3d39c6209d667ad7e426dcb9facc9f"
Jul 17 07:45:16.859653 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.859627    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:45:16.865951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:16.865927    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-w2z2c"]
Jul 17 07:45:18.290175 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:18.290145    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" path="/var/lib/kubelet/pods/838f9182-2911-4fa0-bec7-efbe4d37c8ee/volumes"
Jul 17 07:45:24.289991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:24.289957    2584 scope.go:117] "RemoveContainer" containerID="cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af"
Jul 17 07:45:24.878390 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:24.878287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerStarted","Data":"3054ecbc663de1e1f475a267f1961cde84689b25d93708225a548a4127100a86"}
Jul 17 07:45:24.900459 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:24.900409    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" podStartSLOduration=19.900395032 podStartE2EDuration="19.900395032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:45:24.89976082 +0000 UTC m=+2021.153183632" watchObservedRunningTime="2026-07-17 07:45:24.900395032 +0000 UTC m=+2021.153817843"
Jul 17 07:45:25.884748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:25.884714    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerID="3054ecbc663de1e1f475a267f1961cde84689b25d93708225a548a4127100a86" exitCode=7
Jul 17 07:45:25.885165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:25.884774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerDied","Data":"3054ecbc663de1e1f475a267f1961cde84689b25d93708225a548a4127100a86"}
Jul 17 07:45:25.885165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:25.884820    2584 scope.go:117] "RemoveContainer" containerID="cf99cfb01752b198a0fe873beea8b23a60edd846e2a28d33f470a894dc9c82af"
Jul 17 07:45:25.885165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:25.885087    2584 scope.go:117] "RemoveContainer" containerID="3054ecbc663de1e1f475a267f1961cde84689b25d93708225a548a4127100a86"
Jul 17 07:45:25.885363 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:25.885288    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd_odh-ai-gateway-infra(4cc72686-f4b3-4ae0-9f2c-52529144e810)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810"
Jul 17 07:45:25.909453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:25.909429    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:26.285200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:26.285171    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:45:26.285422 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:26.285330    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:45:27.024981 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.024949    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:27.198244 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.198152    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca\") pod \"4cc72686-f4b3-4ae0-9f2c-52529144e810\" (UID: \"4cc72686-f4b3-4ae0-9f2c-52529144e810\") "
Jul 17 07:45:27.198568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.198544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cc72686-f4b3-4ae0-9f2c-52529144e810" (UID: "4cc72686-f4b3-4ae0-9f2c-52529144e810"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:45:27.298669 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.298634    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc72686-f4b3-4ae0-9f2c-52529144e810-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:45:27.895634 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.895602    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"
Jul 17 07:45:27.895808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.895602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd" event={"ID":"4cc72686-f4b3-4ae0-9f2c-52529144e810","Type":"ContainerDied","Data":"2839620d6b7221a8184e15eea9a0e92531799db8434fa366e401cfc2d1791862"}
Jul 17 07:45:27.895808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.895739    2584 scope.go:117] "RemoveContainer" containerID="3054ecbc663de1e1f475a267f1961cde84689b25d93708225a548a4127100a86"
Jul 17 07:45:27.917678 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.917649    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:27.920733 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:27.920711    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-mkdpd"]
Jul 17 07:45:28.289879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:28.289847    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" path="/var/lib/kubelet/pods/4cc72686-f4b3-4ae0-9f2c-52529144e810/volumes"
Jul 17 07:45:40.284921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:40.284890    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:45:40.285321 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:40.285097    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:45:41.369685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369964    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369975    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369983    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369989    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.369996    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370001    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370011    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370017    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370024    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370030    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370037    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370042    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370048    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370053    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370099    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370105    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370111    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370117    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370122    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370128    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370134    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370140    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc72686-f4b3-4ae0-9f2c-52529144e810" containerName="revoke-keys"
Jul 17 07:45:41.370524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.370145    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:45:41.374165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.374147    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:41.376702 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.376682    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:45:41.381249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.381067    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:45:41.412297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.412263    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz\" (UID: \"b76a611a-628c-41a3-b6e1-d8ae26d06c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:41.513316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.513283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz\" (UID: \"b76a611a-628c-41a3-b6e1-d8ae26d06c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:41.513923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.513905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz\" (UID: \"b76a611a-628c-41a3-b6e1-d8ae26d06c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:41.685997 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.685908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:41.810252 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.810228    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:45:41.812665 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:45:41.812634    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76a611a_628c_41a3_b6e1_d8ae26d06c92.slice/crio-476b30b269ec140d56af802d8685c65c6bdd6ac35b15117e77d2d1c452777b81 WatchSource:0}: Error finding container 476b30b269ec140d56af802d8685c65c6bdd6ac35b15117e77d2d1c452777b81: Status 404 returned error can't find the container with id 476b30b269ec140d56af802d8685c65c6bdd6ac35b15117e77d2d1c452777b81
Jul 17 07:45:41.956572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:41.956489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" event={"ID":"b76a611a-628c-41a3-b6e1-d8ae26d06c92","Type":"ContainerStarted","Data":"476b30b269ec140d56af802d8685c65c6bdd6ac35b15117e77d2d1c452777b81"}
Jul 17 07:45:42.961936 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:42.961897    2584 generic.go:356] "Generic (PLEG): container finished" podID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerID="e33cf866389113777280cbce431eff8cfea8eee74816df9eab6b171bf03d62f8" exitCode=6
Jul 17 07:45:42.962448 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:42.961989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" event={"ID":"b76a611a-628c-41a3-b6e1-d8ae26d06c92","Type":"ContainerDied","Data":"e33cf866389113777280cbce431eff8cfea8eee74816df9eab6b171bf03d62f8"}
Jul 17 07:45:42.962448 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:42.962176    2584 scope.go:117] "RemoveContainer" containerID="e33cf866389113777280cbce431eff8cfea8eee74816df9eab6b171bf03d62f8"
Jul 17 07:45:43.968100 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:43.968067    2584 generic.go:356] "Generic (PLEG): container finished" podID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerID="efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a" exitCode=6
Jul 17 07:45:43.968585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:43.968128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" event={"ID":"b76a611a-628c-41a3-b6e1-d8ae26d06c92","Type":"ContainerDied","Data":"efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a"}
Jul 17 07:45:43.968585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:43.968186    2584 scope.go:117] "RemoveContainer" containerID="e33cf866389113777280cbce431eff8cfea8eee74816df9eab6b171bf03d62f8"
Jul 17 07:45:43.968585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:43.968385    2584 scope.go:117] "RemoveContainer" containerID="efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a"
Jul 17 07:45:43.968694 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:43.968641    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz_odh-ai-gateway-infra(b76a611a-628c-41a3-b6e1-d8ae26d06c92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92"
Jul 17 07:45:44.974352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:44.974311    2584 scope.go:117] "RemoveContainer" containerID="efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a"
Jul 17 07:45:44.974706 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:44.974534    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz_odh-ai-gateway-infra(b76a611a-628c-41a3-b6e1-d8ae26d06c92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92"
Jul 17 07:45:54.286632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:54.286593    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:45:54.287012 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:54.286777    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:45:57.284656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:57.284622    2584 scope.go:117] "RemoveContainer" containerID="efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a"
Jul 17 07:45:58.029446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:58.029409    2584 generic.go:356] "Generic (PLEG): container finished" podID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerID="0ddd08046ca1c90c0b169a74a169496c8b59d29754c87388916430dbc76f0884" exitCode=6
Jul 17 07:45:58.029607 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:58.029482    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" event={"ID":"b76a611a-628c-41a3-b6e1-d8ae26d06c92","Type":"ContainerDied","Data":"0ddd08046ca1c90c0b169a74a169496c8b59d29754c87388916430dbc76f0884"}
Jul 17 07:45:58.029607 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:58.029546    2584 scope.go:117] "RemoveContainer" containerID="efdb8680b701ab91c23ecbe1397cb748a2f29cf269f61a11298a12050018964a"
Jul 17 07:45:58.029836 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:58.029822    2584 scope.go:117] "RemoveContainer" containerID="0ddd08046ca1c90c0b169a74a169496c8b59d29754c87388916430dbc76f0884"
Jul 17 07:45:58.030063 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:45:58.030042    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz_odh-ai-gateway-infra(b76a611a-628c-41a3-b6e1-d8ae26d06c92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92"
Jul 17 07:45:59.056572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.056538    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:45:59.121759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.121716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:45:59.194127 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.194102    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:45:59.253528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.253499    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca\") pod \"b76a611a-628c-41a3-b6e1-d8ae26d06c92\" (UID: \"b76a611a-628c-41a3-b6e1-d8ae26d06c92\") "
Jul 17 07:45:59.253858 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.253830    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b76a611a-628c-41a3-b6e1-d8ae26d06c92" (UID: "b76a611a-628c-41a3-b6e1-d8ae26d06c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:45:59.354724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:45:59.354645    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76a611a-628c-41a3-b6e1-d8ae26d06c92-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:46:00.041251 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.041219    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"
Jul 17 07:46:00.041421 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.041220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz" event={"ID":"b76a611a-628c-41a3-b6e1-d8ae26d06c92","Type":"ContainerDied","Data":"476b30b269ec140d56af802d8685c65c6bdd6ac35b15117e77d2d1c452777b81"}
Jul 17 07:46:00.041421 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.041377    2584 scope.go:117] "RemoveContainer" containerID="0ddd08046ca1c90c0b169a74a169496c8b59d29754c87388916430dbc76f0884"
Jul 17 07:46:00.063017 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.062987    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:46:00.066781 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.066754    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rvnqz"]
Jul 17 07:46:00.294621 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:00.294536    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" path="/var/lib/kubelet/pods/b76a611a-628c-41a3-b6e1-d8ae26d06c92/volumes"
Jul 17 07:46:05.179718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.179676    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.179974    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.179985    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.179992    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.179998    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180008    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180015    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="838f9182-2911-4fa0-bec7-efbe4d37c8ee" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180023    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180028    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7576b71-d3fb-485f-8e29-c1eb403a67a5" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180082    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.180106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.180091    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:05.184414 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.184397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:05.187052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.187030    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:46:05.191513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.191450    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:05.308614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.308582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s\" (UID: \"e5f62986-0a80-4108-ad7d-173584284d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:05.409870 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.409821    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s\" (UID: \"e5f62986-0a80-4108-ad7d-173584284d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:05.410470 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.410452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s\" (UID: \"e5f62986-0a80-4108-ad7d-173584284d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:05.496138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.496062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:05.626130 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:05.626107    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:05.628671 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:46:05.628640    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f62986_0a80_4108_ad7d_173584284d16.slice/crio-790a662daf490507baaea2cd1cdbf774439ecad6e97c2b9702f4019fb581e5b1 WatchSource:0}: Error finding container 790a662daf490507baaea2cd1cdbf774439ecad6e97c2b9702f4019fb581e5b1: Status 404 returned error can't find the container with id 790a662daf490507baaea2cd1cdbf774439ecad6e97c2b9702f4019fb581e5b1
Jul 17 07:46:06.068989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:06.068953    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerStarted","Data":"55b3171acc8dfb79e98daa2312fb412a2e9030cfefcb6f85af5623fb9609ba0a"}
Jul 17 07:46:06.069162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:06.069005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerStarted","Data":"790a662daf490507baaea2cd1cdbf774439ecad6e97c2b9702f4019fb581e5b1"}
Jul 17 07:46:06.086739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:06.086642    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" podStartSLOduration=1.086626752 podStartE2EDuration="1.086626752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:46:06.084568261 +0000 UTC m=+2062.337991084" watchObservedRunningTime="2026-07-17 07:46:06.086626752 +0000 UTC m=+2062.340049564"
Jul 17 07:46:07.075443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:07.075367    2584 generic.go:356] "Generic (PLEG): container finished" podID="e5f62986-0a80-4108-ad7d-173584284d16" containerID="55b3171acc8dfb79e98daa2312fb412a2e9030cfefcb6f85af5623fb9609ba0a" exitCode=7
Jul 17 07:46:07.075808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:07.075453    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerDied","Data":"55b3171acc8dfb79e98daa2312fb412a2e9030cfefcb6f85af5623fb9609ba0a"}
Jul 17 07:46:07.075808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:07.075696    2584 scope.go:117] "RemoveContainer" containerID="55b3171acc8dfb79e98daa2312fb412a2e9030cfefcb6f85af5623fb9609ba0a"
Jul 17 07:46:08.082185 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:08.082147    2584 generic.go:356] "Generic (PLEG): container finished" podID="e5f62986-0a80-4108-ad7d-173584284d16" containerID="13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305" exitCode=7
Jul 17 07:46:08.082713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:08.082232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerDied","Data":"13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305"}
Jul 17 07:46:08.082713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:08.082301    2584 scope.go:117] "RemoveContainer" containerID="55b3171acc8dfb79e98daa2312fb412a2e9030cfefcb6f85af5623fb9609ba0a"
Jul 17 07:46:08.082713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:08.082497    2584 scope.go:117] "RemoveContainer" containerID="13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305"
Jul 17 07:46:08.082858 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:08.082748    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s_odh-ai-gateway-infra(e5f62986-0a80-4108-ad7d-173584284d16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" podUID="e5f62986-0a80-4108-ad7d-173584284d16"
Jul 17 07:46:08.285557 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:08.285524    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:46:08.285765 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:08.285742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:46:09.088728 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:09.088700    2584 scope.go:117] "RemoveContainer" containerID="13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305"
Jul 17 07:46:09.089115 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:09.088899    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s_odh-ai-gateway-infra(e5f62986-0a80-4108-ad7d-173584284d16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" podUID="e5f62986-0a80-4108-ad7d-173584284d16"
Jul 17 07:46:22.284614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:22.284580    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:46:22.285037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:22.284732    2584 scope.go:117] "RemoveContainer" containerID="13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305"
Jul 17 07:46:22.285037 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:22.284767    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:46:23.152035 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:23.152001    2584 generic.go:356] "Generic (PLEG): container finished" podID="e5f62986-0a80-4108-ad7d-173584284d16" containerID="fd2c366c2319ce63bc6e091da93ced307b4884c302fe15dc004db83a1009b1c1" exitCode=7
Jul 17 07:46:23.152239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:23.152081    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerDied","Data":"fd2c366c2319ce63bc6e091da93ced307b4884c302fe15dc004db83a1009b1c1"}
Jul 17 07:46:23.152239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:23.152155    2584 scope.go:117] "RemoveContainer" containerID="13bc2df10108087536d6fb00441f3190faef54e90676e2e33e13796b5116a305"
Jul 17 07:46:23.152524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:23.152508    2584 scope.go:117] "RemoveContainer" containerID="fd2c366c2319ce63bc6e091da93ced307b4884c302fe15dc004db83a1009b1c1"
Jul 17 07:46:23.152736 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:23.152717    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s_odh-ai-gateway-infra(e5f62986-0a80-4108-ad7d-173584284d16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" podUID="e5f62986-0a80-4108-ad7d-173584284d16"
Jul 17 07:46:24.185686 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:24.185655    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:24.321950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:24.321926    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:24.474469 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:24.474391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca\") pod \"e5f62986-0a80-4108-ad7d-173584284d16\" (UID: \"e5f62986-0a80-4108-ad7d-173584284d16\") "
Jul 17 07:46:24.474734 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:24.474712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5f62986-0a80-4108-ad7d-173584284d16" (UID: "e5f62986-0a80-4108-ad7d-173584284d16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:46:24.575132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:24.575102    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f62986-0a80-4108-ad7d-173584284d16-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:46:25.162871 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:25.162839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"
Jul 17 07:46:25.162871 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:25.162855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s" event={"ID":"e5f62986-0a80-4108-ad7d-173584284d16","Type":"ContainerDied","Data":"790a662daf490507baaea2cd1cdbf774439ecad6e97c2b9702f4019fb581e5b1"}
Jul 17 07:46:25.163132 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:25.162915    2584 scope.go:117] "RemoveContainer" containerID="fd2c366c2319ce63bc6e091da93ced307b4884c302fe15dc004db83a1009b1c1"
Jul 17 07:46:25.187288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:25.187260    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:25.192064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:25.192040    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-97f8s"]
Jul 17 07:46:26.290395 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:26.290364    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f62986-0a80-4108-ad7d-173584284d16" path="/var/lib/kubelet/pods/e5f62986-0a80-4108-ad7d-173584284d16/volumes"
Jul 17 07:46:35.284732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:35.284699    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:46:35.285202 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:35.284857    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:46:38.652542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652504    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652805    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652816    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652823    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652829    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652841    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652847    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.652908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652906    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76a611a-628c-41a3-b6e1-d8ae26d06c92" containerName="revoke-keys"
Jul 17 07:46:38.653130 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652913    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.653130 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.652923    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:46:38.657230 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.657213    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:38.659809 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.659786    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:46:38.663716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.663692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:46:38.693138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.693107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4\" (UID: \"fd9bed99-0700-4af4-b0b1-ea5141402f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:38.794330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.794289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4\" (UID: \"fd9bed99-0700-4af4-b0b1-ea5141402f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:38.794924 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.794901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4\" (UID: \"fd9bed99-0700-4af4-b0b1-ea5141402f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:38.968129 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:38.968043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:39.100078 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:39.100053    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:46:39.102091 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:46:39.102063    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9bed99_0700_4af4_b0b1_ea5141402f05.slice/crio-bc63eb5d508d18b25582a30ffa8605c77c3b75001233885795229d5092c53603 WatchSource:0}: Error finding container bc63eb5d508d18b25582a30ffa8605c77c3b75001233885795229d5092c53603: Status 404 returned error can't find the container with id bc63eb5d508d18b25582a30ffa8605c77c3b75001233885795229d5092c53603
Jul 17 07:46:39.224817 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:39.224726    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" event={"ID":"fd9bed99-0700-4af4-b0b1-ea5141402f05","Type":"ContainerStarted","Data":"bc63eb5d508d18b25582a30ffa8605c77c3b75001233885795229d5092c53603"}
Jul 17 07:46:40.229933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:40.229897    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerID="98f179c2079e208d32cbb0e18a611bee6916e00eb9d6992cc64c3a0289db692a" exitCode=6
Jul 17 07:46:40.230352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:40.229939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" event={"ID":"fd9bed99-0700-4af4-b0b1-ea5141402f05","Type":"ContainerDied","Data":"98f179c2079e208d32cbb0e18a611bee6916e00eb9d6992cc64c3a0289db692a"}
Jul 17 07:46:40.230352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:40.230109    2584 scope.go:117] "RemoveContainer" containerID="98f179c2079e208d32cbb0e18a611bee6916e00eb9d6992cc64c3a0289db692a"
Jul 17 07:46:41.236486 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:41.236397    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerID="7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7" exitCode=6
Jul 17 07:46:41.236911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:41.236490    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" event={"ID":"fd9bed99-0700-4af4-b0b1-ea5141402f05","Type":"ContainerDied","Data":"7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7"}
Jul 17 07:46:41.236911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:41.236561    2584 scope.go:117] "RemoveContainer" containerID="98f179c2079e208d32cbb0e18a611bee6916e00eb9d6992cc64c3a0289db692a"
Jul 17 07:46:41.236911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:41.236658    2584 scope.go:117] "RemoveContainer" containerID="7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7"
Jul 17 07:46:41.236911 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:41.236846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4_odh-ai-gateway-infra(fd9bed99-0700-4af4-b0b1-ea5141402f05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05"
Jul 17 07:46:42.242873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:42.242848    2584 scope.go:117] "RemoveContainer" containerID="7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7"
Jul 17 07:46:42.243244 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:42.243032    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4_odh-ai-gateway-infra(fd9bed99-0700-4af4-b0b1-ea5141402f05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05"
Jul 17 07:46:44.349034 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:44.349002    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:46:44.349513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:44.349005    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:46:47.284864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:47.284838    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:46:47.285253 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:47.285025    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:46:55.285226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:55.285192    2584 scope.go:117] "RemoveContainer" containerID="7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7"
Jul 17 07:46:56.311913 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:56.311874    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerID="7500665e2a163eb6ef039985f5da2b2c4f72ce5474bbd1266cb0df33eaeb2260" exitCode=6
Jul 17 07:46:56.312423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:56.311952    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" event={"ID":"fd9bed99-0700-4af4-b0b1-ea5141402f05","Type":"ContainerDied","Data":"7500665e2a163eb6ef039985f5da2b2c4f72ce5474bbd1266cb0df33eaeb2260"}
Jul 17 07:46:56.312423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:56.312064    2584 scope.go:117] "RemoveContainer" containerID="7cd99ac97d527aa6e2d1a161cf149c8aef069c3f389d3f1d1c1bd511bc5187e7"
Jul 17 07:46:56.312423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:56.312376    2584 scope.go:117] "RemoveContainer" containerID="7500665e2a163eb6ef039985f5da2b2c4f72ce5474bbd1266cb0df33eaeb2260"
Jul 17 07:46:56.312646 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:46:56.312625    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4_odh-ai-gateway-infra(fd9bed99-0700-4af4-b0b1-ea5141402f05)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05"
Jul 17 07:46:57.339534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:57.339502    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:46:57.473698 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:57.473670    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:57.550119 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:57.550079    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca\") pod \"fd9bed99-0700-4af4-b0b1-ea5141402f05\" (UID: \"fd9bed99-0700-4af4-b0b1-ea5141402f05\") "
Jul 17 07:46:57.550483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:57.550460    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd9bed99-0700-4af4-b0b1-ea5141402f05" (UID: "fd9bed99-0700-4af4-b0b1-ea5141402f05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:46:57.650735 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:57.650647    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd9bed99-0700-4af4-b0b1-ea5141402f05-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:46:58.322897 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:58.322870    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"
Jul 17 07:46:58.323073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:58.322872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4" event={"ID":"fd9bed99-0700-4af4-b0b1-ea5141402f05","Type":"ContainerDied","Data":"bc63eb5d508d18b25582a30ffa8605c77c3b75001233885795229d5092c53603"}
Jul 17 07:46:58.323073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:58.323005    2584 scope.go:117] "RemoveContainer" containerID="7500665e2a163eb6ef039985f5da2b2c4f72ce5474bbd1266cb0df33eaeb2260"
Jul 17 07:46:58.340435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:58.340391    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:46:58.342405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:46:58.342374    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pdhm4"]
Jul 17 07:47:00.290440 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:00.290407    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" path="/var/lib/kubelet/pods/fd9bed99-0700-4af4-b0b1-ea5141402f05/volumes"
Jul 17 07:47:01.284822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:01.284786    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:47:01.285013 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:01.284945    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:47:05.239611 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239574    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239879    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239890    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239901    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239907    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239920    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239926    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239940    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239945    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.239993    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f62986-0a80-4108-ad7d-173584284d16" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.240002    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.240008    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:05.242955 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.242939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:05.246056 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.246036    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:47:05.267678 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.267648    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:05.321196 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.321157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk\" (UID: \"26910a4a-4bee-4e38-abc8-67ffe72e79f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:05.421612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.421577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk\" (UID: \"26910a4a-4bee-4e38-abc8-67ffe72e79f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:05.422233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.422214    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk\" (UID: \"26910a4a-4bee-4e38-abc8-67ffe72e79f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:05.552925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.552894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:05.685299 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:05.685261    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:05.688259 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:05.688223    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26910a4a_4bee_4e38_abc8_67ffe72e79f7.slice/crio-9a356a4e41f71fa913af43221e8e19f04a677ad6db7f1e1cea9586c5827d90f7 WatchSource:0}: Error finding container 9a356a4e41f71fa913af43221e8e19f04a677ad6db7f1e1cea9586c5827d90f7: Status 404 returned error can't find the container with id 9a356a4e41f71fa913af43221e8e19f04a677ad6db7f1e1cea9586c5827d90f7
Jul 17 07:47:06.358187 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:06.358155    2584 generic.go:356] "Generic (PLEG): container finished" podID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerID="78d690cfbffbc2c0e55715d527bddefb9472213030d59e137d0ecd135b84f254" exitCode=7
Jul 17 07:47:06.358632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:06.358203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" event={"ID":"26910a4a-4bee-4e38-abc8-67ffe72e79f7","Type":"ContainerDied","Data":"78d690cfbffbc2c0e55715d527bddefb9472213030d59e137d0ecd135b84f254"}
Jul 17 07:47:06.358632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:06.358274    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" event={"ID":"26910a4a-4bee-4e38-abc8-67ffe72e79f7","Type":"ContainerStarted","Data":"9a356a4e41f71fa913af43221e8e19f04a677ad6db7f1e1cea9586c5827d90f7"}
Jul 17 07:47:06.358632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:06.358514    2584 scope.go:117] "RemoveContainer" containerID="78d690cfbffbc2c0e55715d527bddefb9472213030d59e137d0ecd135b84f254"
Jul 17 07:47:07.363970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:07.363936    2584 generic.go:356] "Generic (PLEG): container finished" podID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerID="fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5" exitCode=7
Jul 17 07:47:07.364435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:07.363981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" event={"ID":"26910a4a-4bee-4e38-abc8-67ffe72e79f7","Type":"ContainerDied","Data":"fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5"}
Jul 17 07:47:07.364435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:07.364030    2584 scope.go:117] "RemoveContainer" containerID="78d690cfbffbc2c0e55715d527bddefb9472213030d59e137d0ecd135b84f254"
Jul 17 07:47:07.364435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:07.364235    2584 scope.go:117] "RemoveContainer" containerID="fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5"
Jul 17 07:47:07.364606 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:07.364478    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk_odh-ai-gateway-infra(26910a4a-4bee-4e38-abc8-67ffe72e79f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7"
Jul 17 07:47:08.369979 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:08.369954    2584 scope.go:117] "RemoveContainer" containerID="fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5"
Jul 17 07:47:08.370387 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:08.370123    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk_odh-ai-gateway-infra(26910a4a-4bee-4e38-abc8-67ffe72e79f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7"
Jul 17 07:47:14.287617 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:14.287584    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:47:14.288141 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:14.287858    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:47:15.170179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:15.170135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:15.400461 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:15.400427    2584 generic.go:356] "Generic (PLEG): container finished" podID="295b56da-a428-4877-a54d-c3ceb861e79b" containerID="656a1806f12e3ac500768b2d7beef007ba53777dd38257fc62d572fc9a5d2f57" exitCode=1
Jul 17 07:47:15.400896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:15.400493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"656a1806f12e3ac500768b2d7beef007ba53777dd38257fc62d572fc9a5d2f57"}
Jul 17 07:47:15.400896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:15.400553    2584 scope.go:117] "RemoveContainer" containerID="8e04ab406f4a566186f4daf550ccd6bb62aa48eaf5ee51b656063ae0c01021f8"
Jul 17 07:47:15.400896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:15.400884    2584 scope.go:117] "RemoveContainer" containerID="656a1806f12e3ac500768b2d7beef007ba53777dd38257fc62d572fc9a5d2f57"
Jul 17 07:47:15.401108 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:15.401088    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:47:16.230071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:16.230028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:18.127996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:18.127963    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:47:19.285310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:19.285281    2584 scope.go:117] "RemoveContainer" containerID="fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5"
Jul 17 07:47:19.916148 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:19.916111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:47:19.916523 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:19.916509    2584 scope.go:117] "RemoveContainer" containerID="656a1806f12e3ac500768b2d7beef007ba53777dd38257fc62d572fc9a5d2f57"
Jul 17 07:47:19.916692 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:19.916674    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf_odh-ai-gateway-infra(295b56da-a428-4877-a54d-c3ceb861e79b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" podUID="295b56da-a428-4877-a54d-c3ceb861e79b"
Jul 17 07:47:20.428100 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:20.428068    2584 generic.go:356] "Generic (PLEG): container finished" podID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerID="72dce1e7a4b65e89d61c6bb45516fd433a3842fbb2dfd89a8a3bda7b08753ae4" exitCode=7
Jul 17 07:47:20.428573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:20.428141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" event={"ID":"26910a4a-4bee-4e38-abc8-67ffe72e79f7","Type":"ContainerDied","Data":"72dce1e7a4b65e89d61c6bb45516fd433a3842fbb2dfd89a8a3bda7b08753ae4"}
Jul 17 07:47:20.428573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:20.428218    2584 scope.go:117] "RemoveContainer" containerID="fbb7753c1f1ef41a27b61adc7bb8d2120d16c2c163cedca24b287e5f2a50c9f5"
Jul 17 07:47:20.428665 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:20.428575    2584 scope.go:117] "RemoveContainer" containerID="72dce1e7a4b65e89d61c6bb45516fd433a3842fbb2dfd89a8a3bda7b08753ae4"
Jul 17 07:47:20.428800 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:20.428780    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk_odh-ai-gateway-infra(26910a4a-4bee-4e38-abc8-67ffe72e79f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7"
Jul 17 07:47:21.386923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.386887    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:47:21.415184 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.415155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:21.454423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.454393    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:21.531530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.531505    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:47:21.544919 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.544876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:21.599469 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.599442    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:21.655154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.655051    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca\") pod \"295b56da-a428-4877-a54d-c3ceb861e79b\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") "
Jul 17 07:47:21.655154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.655128    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2sbr\" (UniqueName: \"kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr\") pod \"295b56da-a428-4877-a54d-c3ceb861e79b\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") "
Jul 17 07:47:21.655372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.655187    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") pod \"295b56da-a428-4877-a54d-c3ceb861e79b\" (UID: \"295b56da-a428-4877-a54d-c3ceb861e79b\") "
Jul 17 07:47:21.655505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.655475    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "295b56da-a428-4877-a54d-c3ceb861e79b" (UID: "295b56da-a428-4877-a54d-c3ceb861e79b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:47:21.657195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.657168    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "295b56da-a428-4877-a54d-c3ceb861e79b" (UID: "295b56da-a428-4877-a54d-c3ceb861e79b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:47:21.657310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.657290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr" (OuterVolumeSpecName: "kube-api-access-p2sbr") pod "295b56da-a428-4877-a54d-c3ceb861e79b" (UID: "295b56da-a428-4877-a54d-c3ceb861e79b"). InnerVolumeSpecName "kube-api-access-p2sbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:47:21.756406 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.756366    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca\") pod \"26910a4a-4bee-4e38-abc8-67ffe72e79f7\" (UID: \"26910a4a-4bee-4e38-abc8-67ffe72e79f7\") "
Jul 17 07:47:21.756589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.756520    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/295b56da-a428-4877-a54d-c3ceb861e79b-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:21.756589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.756532    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2sbr\" (UniqueName: \"kubernetes.io/projected/295b56da-a428-4877-a54d-c3ceb861e79b-kube-api-access-p2sbr\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:21.756589 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.756541    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/295b56da-a428-4877-a54d-c3ceb861e79b-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:21.756772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.756746    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26910a4a-4bee-4e38-abc8-67ffe72e79f7" (UID: "26910a4a-4bee-4e38-abc8-67ffe72e79f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:47:21.857497 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:21.857463    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26910a4a-4bee-4e38-abc8-67ffe72e79f7-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:22.441282 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.441251    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"
Jul 17 07:47:22.441507 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.441293    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk" event={"ID":"26910a4a-4bee-4e38-abc8-67ffe72e79f7","Type":"ContainerDied","Data":"9a356a4e41f71fa913af43221e8e19f04a677ad6db7f1e1cea9586c5827d90f7"}
Jul 17 07:47:22.441507 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.441395    2584 scope.go:117] "RemoveContainer" containerID="72dce1e7a4b65e89d61c6bb45516fd433a3842fbb2dfd89a8a3bda7b08753ae4"
Jul 17 07:47:22.446378 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.446325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf" event={"ID":"295b56da-a428-4877-a54d-c3ceb861e79b","Type":"ContainerDied","Data":"d44feafb744ef47ce95490302633f44b0df1c8496a4cd1ec386f3a1d83ce35ef"}
Jul 17 07:47:22.446490 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.446380    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"
Jul 17 07:47:22.451277 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.451257    2584 scope.go:117] "RemoveContainer" containerID="656a1806f12e3ac500768b2d7beef007ba53777dd38257fc62d572fc9a5d2f57"
Jul 17 07:47:22.458605 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.458579    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:22.466019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.465992    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-sk8zk"]
Jul 17 07:47:22.475766 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.475731    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:47:22.479361 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:22.479324    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-370a9d5a-7594cdc97c-7bjqf"]
Jul 17 07:47:24.289938 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:24.289895    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" path="/var/lib/kubelet/pods/26910a4a-4bee-4e38-abc8-67ffe72e79f7/volumes"
Jul 17 07:47:24.290493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:24.290474    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" path="/var/lib/kubelet/pods/295b56da-a428-4877-a54d-c3ceb861e79b/volumes"
Jul 17 07:47:27.421838 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:27.421803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:27.482739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:27.482708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:27.687851 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:27.687760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:28.694748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.694715    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695008    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695018    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695029    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695035    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695042    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695047    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695054    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695059    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695064    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695069    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695079    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695084    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695090    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695095    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695101    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695105    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695112    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695117    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695170    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9bed99-0700-4af4-b0b1-ea5141402f05" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695178    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695183    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695190    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695195    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695199    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695204    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695210    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695215    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.695202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.695221    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.699413 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.699395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:28.702207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.702184    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:47:28.702835 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.702813    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:47:28.710081 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.710055    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:28.718279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.718255    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v\" (UID: \"ff470ebf-d24f-4346-a234-c42a018de0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:28.817027 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.816989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:28.819167 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.819134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v\" (UID: \"ff470ebf-d24f-4346-a234-c42a018de0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:28.819743 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.819725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v\" (UID: \"ff470ebf-d24f-4346-a234-c42a018de0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:28.924453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.924418    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:28.925360 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.925317    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:47:28.925664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.925650    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.925711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.925667    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26910a4a-4bee-4e38-abc8-67ffe72e79f7" containerName="revoke-keys"
Jul 17 07:47:28.925754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.925729    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="295b56da-a428-4877-a54d-c3ceb861e79b" containerName="maas-api"
Jul 17 07:47:28.928564 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.928547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:28.931301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.931246    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:47:28.931301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.931266    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f44a26e0-serving-cert\""
Jul 17 07:47:28.931301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.931292    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:47:28.941379 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:28.941333    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:47:29.009793 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.009704    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:29.016863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.016833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:29.020590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.020564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.021005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.020977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.021466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.021446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hkc\" (UniqueName: \"kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.118161 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.118076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:29.122730 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.122706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.122845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.122744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.122920 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.122854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hkc\" (UniqueName: \"kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.123565 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.123540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.125403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.125351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.132570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.132544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hkc\" (UniqueName: \"kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc\") pod \"maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.147826 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.147807    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:29.150077 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:29.150051    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff470ebf_d24f_4346_a234_c42a018de0c1.slice/crio-0e47ec5e378a8d6f2675e0179d8804a38fabf9763ac0abe5c38514508ef8b474 WatchSource:0}: Error finding container 0e47ec5e378a8d6f2675e0179d8804a38fabf9763ac0abe5c38514508ef8b474: Status 404 returned error can't find the container with id 0e47ec5e378a8d6f2675e0179d8804a38fabf9763ac0abe5c38514508ef8b474
Jul 17 07:47:29.239555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.239525    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:29.378923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.378886    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:47:29.382069 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:29.382038    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a013de_b564_438b_95bb_b60ca4585f22.slice/crio-31d93ad287876099d4add22e4599074d5002220e7ed8e5eb03fa686b0f1a2de0 WatchSource:0}: Error finding container 31d93ad287876099d4add22e4599074d5002220e7ed8e5eb03fa686b0f1a2de0: Status 404 returned error can't find the container with id 31d93ad287876099d4add22e4599074d5002220e7ed8e5eb03fa686b0f1a2de0
Jul 17 07:47:29.479966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.479928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerStarted","Data":"553000b0d02d28b5caf2b09e5459fa4310e65b2f2e557a313c3fd96321c941d1"}
Jul 17 07:47:29.480170 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.479983    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerStarted","Data":"0e47ec5e378a8d6f2675e0179d8804a38fabf9763ac0abe5c38514508ef8b474"}
Jul 17 07:47:29.481049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.481024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerStarted","Data":"31d93ad287876099d4add22e4599074d5002220e7ed8e5eb03fa686b0f1a2de0"}
Jul 17 07:47:29.516383 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.516313    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" podStartSLOduration=1.51629696 podStartE2EDuration="1.51629696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:47:29.514754509 +0000 UTC m=+2145.768177321" watchObservedRunningTime="2026-07-17 07:47:29.51629696 +0000 UTC m=+2145.769719772"
Jul 17 07:47:29.593260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.593167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:29.689847 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:29.689813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:30.486604 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.486509    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="f6be44290147856a8096c9f979f9d2ae3b92ad5e89f709dc2daa54bbbbc601eb" exitCode=1
Jul 17 07:47:30.486996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.486599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"f6be44290147856a8096c9f979f9d2ae3b92ad5e89f709dc2daa54bbbbc601eb"}
Jul 17 07:47:30.486996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.486827    2584 scope.go:117] "RemoveContainer" containerID="f6be44290147856a8096c9f979f9d2ae3b92ad5e89f709dc2daa54bbbbc601eb"
Jul 17 07:47:30.488369 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.488328    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerID="553000b0d02d28b5caf2b09e5459fa4310e65b2f2e557a313c3fd96321c941d1" exitCode=6
Jul 17 07:47:30.488501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.488376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerDied","Data":"553000b0d02d28b5caf2b09e5459fa4310e65b2f2e557a313c3fd96321c941d1"}
Jul 17 07:47:30.488616 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:30.488601    2584 scope.go:117] "RemoveContainer" containerID="553000b0d02d28b5caf2b09e5459fa4310e65b2f2e557a313c3fd96321c941d1"
Jul 17 07:47:31.326902 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.326864    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:47:31.459325 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.459243    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:31.462457 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.462440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:31.470224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.470198    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:31.495967 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.495936    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerID="58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6" exitCode=6
Jul 17 07:47:31.496411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.496025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerDied","Data":"58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6"}
Jul 17 07:47:31.496411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.496086    2584 scope.go:117] "RemoveContainer" containerID="553000b0d02d28b5caf2b09e5459fa4310e65b2f2e557a313c3fd96321c941d1"
Jul 17 07:47:31.496411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.496229    2584 scope.go:117] "RemoveContainer" containerID="58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6"
Jul 17 07:47:31.496572 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:31.496469    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v_odh-ai-gateway-infra(ff470ebf-d24f-4346-a234-c42a018de0c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1"
Jul 17 07:47:31.498074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.498054    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a" exitCode=1
Jul 17 07:47:31.498200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.498088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"}
Jul 17 07:47:31.498410 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.498392    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:31.498649 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:31.498628    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:47:31.506840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.506821    2584 scope.go:117] "RemoveContainer" containerID="f6be44290147856a8096c9f979f9d2ae3b92ad5e89f709dc2daa54bbbbc601eb"
Jul 17 07:47:31.544959 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.544920    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-j2thb\" (UID: \"0fde507b-dde1-4dfb-923b-918647bfdf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:31.646004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.645969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-j2thb\" (UID: \"0fde507b-dde1-4dfb-923b-918647bfdf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:31.646669 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.646643    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-j2thb\" (UID: \"0fde507b-dde1-4dfb-923b-918647bfdf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:31.773390 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.773263    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:31.904621 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:31.904593    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:31.906351 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:31.906307    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fde507b_dde1_4dfb_923b_918647bfdf7d.slice/crio-467f4ec44a4b4b01d4a111f249f0419e117c5d7d4dd0a9177c961f83fcbe04fc WatchSource:0}: Error finding container 467f4ec44a4b4b01d4a111f249f0419e117c5d7d4dd0a9177c961f83fcbe04fc: Status 404 returned error can't find the container with id 467f4ec44a4b4b01d4a111f249f0419e117c5d7d4dd0a9177c961f83fcbe04fc
Jul 17 07:47:32.504749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:32.504651    2584 scope.go:117] "RemoveContainer" containerID="58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6"
Jul 17 07:47:32.505198 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:32.504886    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v_odh-ai-gateway-infra(ff470ebf-d24f-4346-a234-c42a018de0c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1"
Jul 17 07:47:32.506273 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:32.506251    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:32.506453 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:32.506434    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:47:32.507443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:32.507419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerStarted","Data":"b149509b4a6264f5efad7536572e278aca3a48fd3b864dd006d64d1ef1e17d0b"}
Jul 17 07:47:32.507528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:32.507463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerStarted","Data":"467f4ec44a4b4b01d4a111f249f0419e117c5d7d4dd0a9177c961f83fcbe04fc"}
Jul 17 07:47:32.557270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:32.557220    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" podStartSLOduration=1.557204203 podStartE2EDuration="1.557204203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:47:32.553929182 +0000 UTC m=+2148.807352004" watchObservedRunningTime="2026-07-17 07:47:32.557204203 +0000 UTC m=+2148.810627015"
Jul 17 07:47:33.251942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.251902    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:33.255214 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.255190    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:33.262094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.262063    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:33.357326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.357289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h\" (UID: \"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:33.458378 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.458321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h\" (UID: \"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:33.458973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.458951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h\" (UID: \"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:33.512833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.512742    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerID="b149509b4a6264f5efad7536572e278aca3a48fd3b864dd006d64d1ef1e17d0b" exitCode=6
Jul 17 07:47:33.512833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.512797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerDied","Data":"b149509b4a6264f5efad7536572e278aca3a48fd3b864dd006d64d1ef1e17d0b"}
Jul 17 07:47:33.513226 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.513045    2584 scope.go:117] "RemoveContainer" containerID="b149509b4a6264f5efad7536572e278aca3a48fd3b864dd006d64d1ef1e17d0b"
Jul 17 07:47:33.567014 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.566985    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:33.741447 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:33.741423    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:33.819391 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:33.819355    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840ba9e3_c2b5_4c9c_8879_ecd809b11b3e.slice/crio-6aef052a7c0014f41bfcb9f84a5c46832e432221bc991476844f506e880f65a0 WatchSource:0}: Error finding container 6aef052a7c0014f41bfcb9f84a5c46832e432221bc991476844f506e880f65a0: Status 404 returned error can't find the container with id 6aef052a7c0014f41bfcb9f84a5c46832e432221bc991476844f506e880f65a0
Jul 17 07:47:34.240253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.240220    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:34.240692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.240675    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:34.240898 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:34.240877    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:47:34.520025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.519598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerStarted","Data":"4a015ec156098df2deb46cc59556ce1ee61b7494a88513bfc1289e0d41850778"}
Jul 17 07:47:34.520025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.519659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerStarted","Data":"6aef052a7c0014f41bfcb9f84a5c46832e432221bc991476844f506e880f65a0"}
Jul 17 07:47:34.521576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.521549    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerID="2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac" exitCode=6
Jul 17 07:47:34.521682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.521583    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerDied","Data":"2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac"}
Jul 17 07:47:34.521682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.521627    2584 scope.go:117] "RemoveContainer" containerID="b149509b4a6264f5efad7536572e278aca3a48fd3b864dd006d64d1ef1e17d0b"
Jul 17 07:47:34.521829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.521808    2584 scope.go:117] "RemoveContainer" containerID="2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac"
Jul 17 07:47:34.522061 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:34.522038    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-j2thb_odh-ai-gateway-infra(0fde507b-dde1-4dfb-923b-918647bfdf7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d"
Jul 17 07:47:34.536690 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:34.536635    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" podStartSLOduration=1.536618193 podStartE2EDuration="1.536618193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:47:34.535537189 +0000 UTC m=+2150.788960002" watchObservedRunningTime="2026-07-17 07:47:34.536618193 +0000 UTC m=+2150.790041004"
Jul 17 07:47:35.528112 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:35.528078    2584 generic.go:356] "Generic (PLEG): container finished" podID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerID="4a015ec156098df2deb46cc59556ce1ee61b7494a88513bfc1289e0d41850778" exitCode=7
Jul 17 07:47:35.528531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:35.528168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerDied","Data":"4a015ec156098df2deb46cc59556ce1ee61b7494a88513bfc1289e0d41850778"}
Jul 17 07:47:35.528531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:35.528359    2584 scope.go:117] "RemoveContainer" containerID="4a015ec156098df2deb46cc59556ce1ee61b7494a88513bfc1289e0d41850778"
Jul 17 07:47:35.529951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:35.529934    2584 scope.go:117] "RemoveContainer" containerID="2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac"
Jul 17 07:47:35.530112 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:35.530095    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-j2thb_odh-ai-gateway-infra(0fde507b-dde1-4dfb-923b-918647bfdf7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d"
Jul 17 07:47:36.472966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.472930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:36.536218 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.536184    2584 generic.go:356] "Generic (PLEG): container finished" podID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerID="97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e" exitCode=7
Jul 17 07:47:36.536675 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.536268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerDied","Data":"97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e"}
Jul 17 07:47:36.536675 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.536327    2584 scope.go:117] "RemoveContainer" containerID="4a015ec156098df2deb46cc59556ce1ee61b7494a88513bfc1289e0d41850778"
Jul 17 07:47:36.536675 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.536560    2584 scope.go:117] "RemoveContainer" containerID="97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e"
Jul 17 07:47:36.536833 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:36.536809    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h_odh-ai-gateway-infra(840ba9e3-c2b5-4c9c-8879-ecd809b11b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e"
Jul 17 07:47:36.557117 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:36.557088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:37.543066 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:37.543039    2584 scope.go:117] "RemoveContainer" containerID="97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e"
Jul 17 07:47:37.543548 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:37.543224    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h_odh-ai-gateway-infra(840ba9e3-c2b5-4c9c-8879-ecd809b11b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e"
Jul 17 07:47:39.239862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:39.239832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:39.240248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:39.240233    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:39.240448 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:39.240427    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:47:40.283314 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:40.283286    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:47:47.284917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:47.284879    2584 scope.go:117] "RemoveContainer" containerID="58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6"
Jul 17 07:47:47.589195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:47.589106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerStarted","Data":"3a520924138238bc74aeeb890416c41cd874940536b8c86276f461d5771775d2"}
Jul 17 07:47:48.595562 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:48.595530    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerID="3a520924138238bc74aeeb890416c41cd874940536b8c86276f461d5771775d2" exitCode=6
Jul 17 07:47:48.595989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:48.595602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerDied","Data":"3a520924138238bc74aeeb890416c41cd874940536b8c86276f461d5771775d2"}
Jul 17 07:47:48.595989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:48.595662    2584 scope.go:117] "RemoveContainer" containerID="58bb8f4f1e601cf2742344fe5276f469cb3315a0f0cdf4e25ee0c2957e8b99d6"
Jul 17 07:47:48.595989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:48.595985    2584 scope.go:117] "RemoveContainer" containerID="3a520924138238bc74aeeb890416c41cd874940536b8c86276f461d5771775d2"
Jul 17 07:47:48.596197 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:48.596179    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v_odh-ai-gateway-infra(ff470ebf-d24f-4346-a234-c42a018de0c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1"
Jul 17 07:47:48.625495 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:48.625458    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:49.736138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:49.736110    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:49.802737 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:49.802704    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca\") pod \"ff470ebf-d24f-4346-a234-c42a018de0c1\" (UID: \"ff470ebf-d24f-4346-a234-c42a018de0c1\") "
Jul 17 07:47:49.803067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:49.803045    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff470ebf-d24f-4346-a234-c42a018de0c1" (UID: "ff470ebf-d24f-4346-a234-c42a018de0c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:47:49.904189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:49.904106    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff470ebf-d24f-4346-a234-c42a018de0c1-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:50.607396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:50.607306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"
Jul 17 07:47:50.607396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:50.607355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v" event={"ID":"ff470ebf-d24f-4346-a234-c42a018de0c1","Type":"ContainerDied","Data":"0e47ec5e378a8d6f2675e0179d8804a38fabf9763ac0abe5c38514508ef8b474"}
Jul 17 07:47:50.607624 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:50.607417    2584 scope.go:117] "RemoveContainer" containerID="3a520924138238bc74aeeb890416c41cd874940536b8c86276f461d5771775d2"
Jul 17 07:47:50.624307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:50.624282    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:50.626475 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:50.626450    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rnp7v"]
Jul 17 07:47:51.284963 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:51.284934    2584 scope.go:117] "RemoveContainer" containerID="2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac"
Jul 17 07:47:51.614640 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:51.614549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerStarted","Data":"6f9e23a926a47fa8b1b7e530a8714868c6bbc9198e0466d2e78f54a3066b29c1"}
Jul 17 07:47:52.284861 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.284826    2584 scope.go:117] "RemoveContainer" containerID="97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e"
Jul 17 07:47:52.291834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.291807    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" path="/var/lib/kubelet/pods/ff470ebf-d24f-4346-a234-c42a018de0c1/volumes"
Jul 17 07:47:52.621612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.621527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerStarted","Data":"52381688de25b4f8da72f0b28a2062f582c63f6e5748ea9b1e4bd2c81c7dfaf4"}
Jul 17 07:47:52.623234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.623211    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerID="6f9e23a926a47fa8b1b7e530a8714868c6bbc9198e0466d2e78f54a3066b29c1" exitCode=6
Jul 17 07:47:52.623392 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.623248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerDied","Data":"6f9e23a926a47fa8b1b7e530a8714868c6bbc9198e0466d2e78f54a3066b29c1"}
Jul 17 07:47:52.623392 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.623287    2584 scope.go:117] "RemoveContainer" containerID="2d5eb2fbf0c7c6847395cd8818bdd433699fecaa1e7d2f8c5be4deeaaa92e9ac"
Jul 17 07:47:52.623586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.623569    2584 scope.go:117] "RemoveContainer" containerID="6f9e23a926a47fa8b1b7e530a8714868c6bbc9198e0466d2e78f54a3066b29c1"
Jul 17 07:47:52.623784 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:52.623765    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-j2thb_odh-ai-gateway-infra(0fde507b-dde1-4dfb-923b-918647bfdf7d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d"
Jul 17 07:47:52.645408 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:52.645384    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:53.285261 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.285233    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:53.631931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.631900    2584 generic.go:356] "Generic (PLEG): container finished" podID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerID="52381688de25b4f8da72f0b28a2062f582c63f6e5748ea9b1e4bd2c81c7dfaf4" exitCode=7
Jul 17 07:47:53.632309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.631965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerDied","Data":"52381688de25b4f8da72f0b28a2062f582c63f6e5748ea9b1e4bd2c81c7dfaf4"}
Jul 17 07:47:53.632309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.632024    2584 scope.go:117] "RemoveContainer" containerID="97ee28ac3091d0a31af844f9e9d8940ca1fa0d11b8599ed126bde674a960414e"
Jul 17 07:47:53.632456 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.632351    2584 scope.go:117] "RemoveContainer" containerID="52381688de25b4f8da72f0b28a2062f582c63f6e5748ea9b1e4bd2c81c7dfaf4"
Jul 17 07:47:53.632595 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:53.632570    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h_odh-ai-gateway-infra(840ba9e3-c2b5-4c9c-8879-ecd809b11b3e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e"
Jul 17 07:47:53.652880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.652634    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:53.772289 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.772263    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:53.835452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.835421    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca\") pod \"0fde507b-dde1-4dfb-923b-918647bfdf7d\" (UID: \"0fde507b-dde1-4dfb-923b-918647bfdf7d\") "
Jul 17 07:47:53.835748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.835725    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fde507b-dde1-4dfb-923b-918647bfdf7d" (UID: "0fde507b-dde1-4dfb-923b-918647bfdf7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:47:53.936442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:53.936327    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fde507b-dde1-4dfb-923b-918647bfdf7d-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:54.640284 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.640205    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636" exitCode=1
Jul 17 07:47:54.640768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.640280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"}
Jul 17 07:47:54.640768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.640363    2584 scope.go:117] "RemoveContainer" containerID="282a1cd512475c86c1cce427a228ab2e9a991efa1e418098c52eb2e8d54be43a"
Jul 17 07:47:54.640768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.640691    2584 scope.go:117] "RemoveContainer" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"
Jul 17 07:47:54.640928 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:54.640908    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:47:54.641747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.641617    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"
Jul 17 07:47:54.641747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.641657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb" event={"ID":"0fde507b-dde1-4dfb-923b-918647bfdf7d","Type":"ContainerDied","Data":"467f4ec44a4b4b01d4a111f249f0419e117c5d7d4dd0a9177c961f83fcbe04fc"}
Jul 17 07:47:54.651646 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.651623    2584 scope.go:117] "RemoveContainer" containerID="6f9e23a926a47fa8b1b7e530a8714868c6bbc9198e0466d2e78f54a3066b29c1"
Jul 17 07:47:54.690496 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.690457    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:54.697415 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.697386    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-j2thb"]
Jul 17 07:47:54.775416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.775390    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:54.843751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.843717    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca\") pod \"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e\" (UID: \"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e\") "
Jul 17 07:47:54.844098 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.844075    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" (UID: "840ba9e3-c2b5-4c9c-8879-ecd809b11b3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:47:54.944942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:54.944855    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:47:55.280789 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.280708    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:47:55.281019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281007    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281020    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281030    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281036    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281049    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281061 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281055    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281066    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281071    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281077    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281083    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281088    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281093    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281141    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281149    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281154    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281160    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281165    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281171    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:47:55.281212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.281177    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:47:55.283976 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.283960    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:47:55.291772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.291747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:47:55.347920 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.347884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x\" (UID: \"907b0e11-11fc-4c1f-b706-f60cafe70d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:47:55.448568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.448526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x\" (UID: \"907b0e11-11fc-4c1f-b706-f60cafe70d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:47:55.449186 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.449150    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x\" (UID: \"907b0e11-11fc-4c1f-b706-f60cafe70d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:47:55.594239 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.594210    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:47:55.650832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.650761    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"
Jul 17 07:47:55.650832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.650771    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h" event={"ID":"840ba9e3-c2b5-4c9c-8879-ecd809b11b3e","Type":"ContainerDied","Data":"6aef052a7c0014f41bfcb9f84a5c46832e432221bc991476844f506e880f65a0"}
Jul 17 07:47:55.650832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.650826    2584 scope.go:117] "RemoveContainer" containerID="52381688de25b4f8da72f0b28a2062f582c63f6e5748ea9b1e4bd2c81c7dfaf4"
Jul 17 07:47:55.673832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.673804    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:55.678576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.678551    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vvs5h"]
Jul 17 07:47:55.728308 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:55.728234    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:47:55.730531 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:47:55.730495    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907b0e11_11fc_4c1f_b706_f60cafe70d29.slice/crio-450a008b0d3e7ac9d64302703271c064bcc58c566e94d73b40396429d8511e94 WatchSource:0}: Error finding container 450a008b0d3e7ac9d64302703271c064bcc58c566e94d73b40396429d8511e94: Status 404 returned error can't find the container with id 450a008b0d3e7ac9d64302703271c064bcc58c566e94d73b40396429d8511e94
Jul 17 07:47:56.290527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.290490    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" path="/var/lib/kubelet/pods/0fde507b-dde1-4dfb-923b-918647bfdf7d/volumes"
Jul 17 07:47:56.290829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.290816    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" path="/var/lib/kubelet/pods/840ba9e3-c2b5-4c9c-8879-ecd809b11b3e/volumes"
Jul 17 07:47:56.660940 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.660849    2584 generic.go:356] "Generic (PLEG): container finished" podID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerID="3249a116bb487de183e3febaa6883a905d8123d872acf42433046b514d92cdff" exitCode=6
Jul 17 07:47:56.661426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.660934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerDied","Data":"3249a116bb487de183e3febaa6883a905d8123d872acf42433046b514d92cdff"}
Jul 17 07:47:56.661426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.660987    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerStarted","Data":"450a008b0d3e7ac9d64302703271c064bcc58c566e94d73b40396429d8511e94"}
Jul 17 07:47:56.661426 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:56.661161    2584 scope.go:117] "RemoveContainer" containerID="3249a116bb487de183e3febaa6883a905d8123d872acf42433046b514d92cdff"
Jul 17 07:47:57.670156 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:57.670124    2584 generic.go:356] "Generic (PLEG): container finished" podID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerID="97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c" exitCode=6
Jul 17 07:47:57.670627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:57.670175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerDied","Data":"97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c"}
Jul 17 07:47:57.670627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:57.670245    2584 scope.go:117] "RemoveContainer" containerID="3249a116bb487de183e3febaa6883a905d8123d872acf42433046b514d92cdff"
Jul 17 07:47:57.670627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:57.670485    2584 scope.go:117] "RemoveContainer" containerID="97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c"
Jul 17 07:47:57.670735 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:57.670687    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x_odh-ai-gateway-infra(907b0e11-11fc-4c1f-b706-f60cafe70d29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29"
Jul 17 07:47:58.676067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:58.676040    2584 scope.go:117] "RemoveContainer" containerID="97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c"
Jul 17 07:47:58.676451 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:58.676201    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x_odh-ai-gateway-infra(907b0e11-11fc-4c1f-b706-f60cafe70d29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29"
Jul 17 07:47:59.239863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:59.239834    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:59.240020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:59.239887    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:47:59.240237 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:47:59.240217    2584 scope.go:117] "RemoveContainer" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"
Jul 17 07:47:59.240433 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:47:59.240415    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:48:10.284647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:10.284613    2584 scope.go:117] "RemoveContainer" containerID="97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c"
Jul 17 07:48:10.729145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:10.729055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerStarted","Data":"9d1e2d37c210e31ee25ba088b2a1452614e39f452482e70944531505c167eec8"}
Jul 17 07:48:10.747606 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:10.747552    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" podStartSLOduration=15.747535658 podStartE2EDuration="15.747535658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:48:10.745717007 +0000 UTC m=+2186.999139819" watchObservedRunningTime="2026-07-17 07:48:10.747535658 +0000 UTC m=+2187.000958469"
Jul 17 07:48:11.284956 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.284915    2584 scope.go:117] "RemoveContainer" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"
Jul 17 07:48:11.285373 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:11.285126    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:48:11.735969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.735886    2584 generic.go:356] "Generic (PLEG): container finished" podID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerID="9d1e2d37c210e31ee25ba088b2a1452614e39f452482e70944531505c167eec8" exitCode=6
Jul 17 07:48:11.735969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.735950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerDied","Data":"9d1e2d37c210e31ee25ba088b2a1452614e39f452482e70944531505c167eec8"}
Jul 17 07:48:11.736195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.735996    2584 scope.go:117] "RemoveContainer" containerID="97c0b3b868bcc0f716fb204adc2db19ff3f8620ed75070fcd1175e09c00ead7c"
Jul 17 07:48:11.736262 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.736248    2584 scope.go:117] "RemoveContainer" containerID="9d1e2d37c210e31ee25ba088b2a1452614e39f452482e70944531505c167eec8"
Jul 17 07:48:11.736491 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:11.736473    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x_odh-ai-gateway-infra(907b0e11-11fc-4c1f-b706-f60cafe70d29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29"
Jul 17 07:48:11.765330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:11.765291    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:48:12.879966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:12.879939    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:48:12.999766 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:12.999670    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca\") pod \"907b0e11-11fc-4c1f-b706-f60cafe70d29\" (UID: \"907b0e11-11fc-4c1f-b706-f60cafe70d29\") "
Jul 17 07:48:13.000064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.000041    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "907b0e11-11fc-4c1f-b706-f60cafe70d29" (UID: "907b0e11-11fc-4c1f-b706-f60cafe70d29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:48:13.100661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.100621    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/907b0e11-11fc-4c1f-b706-f60cafe70d29-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:48:13.747711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.747679    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"
Jul 17 07:48:13.747911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.747712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x" event={"ID":"907b0e11-11fc-4c1f-b706-f60cafe70d29","Type":"ContainerDied","Data":"450a008b0d3e7ac9d64302703271c064bcc58c566e94d73b40396429d8511e94"}
Jul 17 07:48:13.747911 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.747770    2584 scope.go:117] "RemoveContainer" containerID="9d1e2d37c210e31ee25ba088b2a1452614e39f452482e70944531505c167eec8"
Jul 17 07:48:13.771501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.771461    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:48:13.776735 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:13.776704    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2th9x"]
Jul 17 07:48:14.290593 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:14.290562    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" path="/var/lib/kubelet/pods/907b0e11-11fc-4c1f-b706-f60cafe70d29/volumes"
Jul 17 07:48:25.285446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:25.285411    2584 scope.go:117] "RemoveContainer" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"
Jul 17 07:48:26.806593 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:26.806559    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1" exitCode=1
Jul 17 07:48:26.807002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:26.806603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"}
Jul 17 07:48:26.807002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:26.806658    2584 scope.go:117] "RemoveContainer" containerID="519fa22d602f840f122f6261820e108d8a5326fbba50055dda23b023de7c2636"
Jul 17 07:48:26.807002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:26.806976    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:48:26.807186 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:26.807163    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:48:29.240397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:29.240369    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:48:29.240829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:29.240419    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:48:29.240829 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:29.240766    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:48:29.240971 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:29.240952    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:48:31.507410 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507374    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507684    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507695    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507711    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507717    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507738    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507744    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff470ebf-d24f-4346-a234-c42a018de0c1" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507752    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507757    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507766    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:48:31.507782 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507771    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:48:31.508073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507820    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.508073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507830    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="840ba9e3-c2b5-4c9c-8879-ecd809b11b3e" containerName="revoke-keys"
Jul 17 07:48:31.508073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507837    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fde507b-dde1-4dfb-923b-918647bfdf7d" containerName="revoke-keys"
Jul 17 07:48:31.508073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.507842    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:31.511972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.511955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:31.514590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.514568    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:48:31.517931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.517907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:31.556026 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.555997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-m65cp\" (UID: \"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:31.657031 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.656990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-m65cp\" (UID: \"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:31.657666 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.657643    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-m65cp\" (UID: \"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:31.823397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.823365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:31.953234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:31.953210    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:31.955149 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:48:31.955118    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b5a1ee_c35e_455f_9881_d1ef6ed0f5ca.slice/crio-5552328e0e72d435986e13885a885bafff765be9873fd511386c48dd7a8bc664 WatchSource:0}: Error finding container 5552328e0e72d435986e13885a885bafff765be9873fd511386c48dd7a8bc664: Status 404 returned error can't find the container with id 5552328e0e72d435986e13885a885bafff765be9873fd511386c48dd7a8bc664
Jul 17 07:48:32.836819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:32.836784    2584 generic.go:356] "Generic (PLEG): container finished" podID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerID="dc5def633af53c27e9675c49e3bf001e7613cdd768ab0cc70e9261327c8e9962" exitCode=6
Jul 17 07:48:32.837180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:32.836869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" event={"ID":"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca","Type":"ContainerDied","Data":"dc5def633af53c27e9675c49e3bf001e7613cdd768ab0cc70e9261327c8e9962"}
Jul 17 07:48:32.837180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:32.836922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" event={"ID":"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca","Type":"ContainerStarted","Data":"5552328e0e72d435986e13885a885bafff765be9873fd511386c48dd7a8bc664"}
Jul 17 07:48:32.837180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:32.837021    2584 scope.go:117] "RemoveContainer" containerID="dc5def633af53c27e9675c49e3bf001e7613cdd768ab0cc70e9261327c8e9962"
Jul 17 07:48:33.297329 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.297297    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:33.297650 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.297637    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:33.297707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.297652    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:33.297707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.297700    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="907b0e11-11fc-4c1f-b706-f60cafe70d29" containerName="revoke-keys"
Jul 17 07:48:33.300585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.300571    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:33.309890 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.309767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:33.369463 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.369420    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl\" (UID: \"972f82cf-1f6d-4508-8339-4d742dfa568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:33.470719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.470674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl\" (UID: \"972f82cf-1f6d-4508-8339-4d742dfa568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:33.471374 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.471321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl\" (UID: \"972f82cf-1f6d-4508-8339-4d742dfa568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:33.611870 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.611781    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:33.746025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.745990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:33.749088 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:48:33.749042    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f82cf_1f6d_4508_8339_4d742dfa568d.slice/crio-3ed65fca98efc6b074680330d97ad190fbe8f9e70a11d41b977eecb340d8f85b WatchSource:0}: Error finding container 3ed65fca98efc6b074680330d97ad190fbe8f9e70a11d41b977eecb340d8f85b: Status 404 returned error can't find the container with id 3ed65fca98efc6b074680330d97ad190fbe8f9e70a11d41b977eecb340d8f85b
Jul 17 07:48:33.842189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.842150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" event={"ID":"972f82cf-1f6d-4508-8339-4d742dfa568d","Type":"ContainerStarted","Data":"3ed65fca98efc6b074680330d97ad190fbe8f9e70a11d41b977eecb340d8f85b"}
Jul 17 07:48:33.843897 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.843870    2584 generic.go:356] "Generic (PLEG): container finished" podID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerID="180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27" exitCode=6
Jul 17 07:48:33.844049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.843945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" event={"ID":"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca","Type":"ContainerDied","Data":"180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27"}
Jul 17 07:48:33.844049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.843998    2584 scope.go:117] "RemoveContainer" containerID="dc5def633af53c27e9675c49e3bf001e7613cdd768ab0cc70e9261327c8e9962"
Jul 17 07:48:33.844200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:33.844184    2584 scope.go:117] "RemoveContainer" containerID="180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27"
Jul 17 07:48:33.844456 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:33.844437    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-m65cp_odh-ai-gateway-infra(26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca"
Jul 17 07:48:34.849659 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:34.849624    2584 scope.go:117] "RemoveContainer" containerID="180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27"
Jul 17 07:48:34.850097 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:34.849846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-m65cp_odh-ai-gateway-infra(26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca"
Jul 17 07:48:34.850915 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:34.850891    2584 generic.go:356] "Generic (PLEG): container finished" podID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerID="01ebf54bb67283e3d401a1e728e796582a7b4f41808fd7086e58258512e346c1" exitCode=7
Jul 17 07:48:34.851023 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:34.850947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" event={"ID":"972f82cf-1f6d-4508-8339-4d742dfa568d","Type":"ContainerDied","Data":"01ebf54bb67283e3d401a1e728e796582a7b4f41808fd7086e58258512e346c1"}
Jul 17 07:48:34.851110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:34.851098    2584 scope.go:117] "RemoveContainer" containerID="01ebf54bb67283e3d401a1e728e796582a7b4f41808fd7086e58258512e346c1"
Jul 17 07:48:35.857465 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:35.857424    2584 generic.go:356] "Generic (PLEG): container finished" podID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerID="ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e" exitCode=7
Jul 17 07:48:35.857863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:35.857471    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" event={"ID":"972f82cf-1f6d-4508-8339-4d742dfa568d","Type":"ContainerDied","Data":"ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e"}
Jul 17 07:48:35.857863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:35.857547    2584 scope.go:117] "RemoveContainer" containerID="01ebf54bb67283e3d401a1e728e796582a7b4f41808fd7086e58258512e346c1"
Jul 17 07:48:35.857863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:35.857713    2584 scope.go:117] "RemoveContainer" containerID="ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e"
Jul 17 07:48:35.858048 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:35.857947    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl_odh-ai-gateway-infra(972f82cf-1f6d-4508-8339-4d742dfa568d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d"
Jul 17 07:48:36.863330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:36.863303    2584 scope.go:117] "RemoveContainer" containerID="ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e"
Jul 17 07:48:36.863820 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:36.863590    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl_odh-ai-gateway-infra(972f82cf-1f6d-4508-8339-4d742dfa568d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d"
Jul 17 07:48:43.285570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:43.285533    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:48:43.286038 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:43.285736    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:48:45.325689 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.325651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:48:45.330748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.330728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:48:45.335819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.335790    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:48:45.477771 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.477740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7\" (UID: \"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:48:45.578888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.578808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7\" (UID: \"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:48:45.579466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.579448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7\" (UID: \"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:48:45.643059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.643033    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:48:45.780775 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.780666    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:48:45.783416 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:48:45.783383    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136a507e_aa2d_4ca9_ae3f_48d3b47fb4fb.slice/crio-5ef02f0f06720ab5db788890c296577b558cd3ff8e35ba411448cbabd9e66478 WatchSource:0}: Error finding container 5ef02f0f06720ab5db788890c296577b558cd3ff8e35ba411448cbabd9e66478: Status 404 returned error can't find the container with id 5ef02f0f06720ab5db788890c296577b558cd3ff8e35ba411448cbabd9e66478
Jul 17 07:48:45.904333 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:45.904252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" event={"ID":"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb","Type":"ContainerStarted","Data":"5ef02f0f06720ab5db788890c296577b558cd3ff8e35ba411448cbabd9e66478"}
Jul 17 07:48:46.910588 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:46.910558    2584 generic.go:356] "Generic (PLEG): container finished" podID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerID="e67296dbf3f8fe304e522e52df76e56e41aee3e616f6798543d3f35eaaa11ab8" exitCode=6
Jul 17 07:48:46.910960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:46.910635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" event={"ID":"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb","Type":"ContainerDied","Data":"e67296dbf3f8fe304e522e52df76e56e41aee3e616f6798543d3f35eaaa11ab8"}
Jul 17 07:48:46.910960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:46.910788    2584 scope.go:117] "RemoveContainer" containerID="e67296dbf3f8fe304e522e52df76e56e41aee3e616f6798543d3f35eaaa11ab8"
Jul 17 07:48:47.916243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:47.916214    2584 generic.go:356] "Generic (PLEG): container finished" podID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerID="79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093" exitCode=6
Jul 17 07:48:47.916641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:47.916300    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" event={"ID":"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb","Type":"ContainerDied","Data":"79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093"}
Jul 17 07:48:47.916641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:47.916384    2584 scope.go:117] "RemoveContainer" containerID="e67296dbf3f8fe304e522e52df76e56e41aee3e616f6798543d3f35eaaa11ab8"
Jul 17 07:48:47.916641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:47.916519    2584 scope.go:117] "RemoveContainer" containerID="79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093"
Jul 17 07:48:47.916800 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:47.916719    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7_odh-ai-gateway-infra(136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb"
Jul 17 07:48:48.284567 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.284544    2584 scope.go:117] "RemoveContainer" containerID="180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27"
Jul 17 07:48:48.922849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.922820    2584 generic.go:356] "Generic (PLEG): container finished" podID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerID="36fbede99c380ac86d9e49b457faeadedf20572cca7ca715df3894af64be5683" exitCode=6
Jul 17 07:48:48.923250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.922894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" event={"ID":"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca","Type":"ContainerDied","Data":"36fbede99c380ac86d9e49b457faeadedf20572cca7ca715df3894af64be5683"}
Jul 17 07:48:48.923250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.922956    2584 scope.go:117] "RemoveContainer" containerID="180eac4f6f0512a41657025503fb168a54f0e90dae60fc74d1ca138560948d27"
Jul 17 07:48:48.923386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.923273    2584 scope.go:117] "RemoveContainer" containerID="36fbede99c380ac86d9e49b457faeadedf20572cca7ca715df3894af64be5683"
Jul 17 07:48:48.923523 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:48.923504    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-m65cp_odh-ai-gateway-infra(26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca"
Jul 17 07:48:48.924657 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:48.924642    2584 scope.go:117] "RemoveContainer" containerID="79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093"
Jul 17 07:48:48.924825 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:48.924808    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7_odh-ai-gateway-infra(136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb"
Jul 17 07:48:49.286715 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.286677    2584 scope.go:117] "RemoveContainer" containerID="ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e"
Jul 17 07:48:49.931034 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.931001    2584 generic.go:356] "Generic (PLEG): container finished" podID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerID="03c19fc4f79e10689f4bc832a6c0e4d6b49796ca7662ca817ca4520f52dc0125" exitCode=7
Jul 17 07:48:49.931533 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.931077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" event={"ID":"972f82cf-1f6d-4508-8339-4d742dfa568d","Type":"ContainerDied","Data":"03c19fc4f79e10689f4bc832a6c0e4d6b49796ca7662ca817ca4520f52dc0125"}
Jul 17 07:48:49.931533 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.931167    2584 scope.go:117] "RemoveContainer" containerID="ac6f9d73faa4a62ef898e1f97a912b70525f4fff327f3c8c1ec6509d1bcdd52e"
Jul 17 07:48:49.931533 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.931488    2584 scope.go:117] "RemoveContainer" containerID="03c19fc4f79e10689f4bc832a6c0e4d6b49796ca7662ca817ca4520f52dc0125"
Jul 17 07:48:49.931749 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:49.931725    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl_odh-ai-gateway-infra(972f82cf-1f6d-4508-8339-4d742dfa568d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d"
Jul 17 07:48:49.952622 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:49.952597    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:50.090628 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.090604    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:50.215818 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.215789    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca\") pod \"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca\" (UID: \"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca\") "
Jul 17 07:48:50.216136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.216113    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" (UID: "26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:48:50.317093 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.317069    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:48:50.938376 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.938331    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"
Jul 17 07:48:50.938868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.938377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp" event={"ID":"26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca","Type":"ContainerDied","Data":"5552328e0e72d435986e13885a885bafff765be9873fd511386c48dd7a8bc664"}
Jul 17 07:48:50.938868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.938442    2584 scope.go:117] "RemoveContainer" containerID="36fbede99c380ac86d9e49b457faeadedf20572cca7ca715df3894af64be5683"
Jul 17 07:48:50.955711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.955654    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:50.957373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.957348    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-m65cp"]
Jul 17 07:48:50.960201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:50.960179    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:51.091579 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.091549    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:51.225320 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.225238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca\") pod \"972f82cf-1f6d-4508-8339-4d742dfa568d\" (UID: \"972f82cf-1f6d-4508-8339-4d742dfa568d\") "
Jul 17 07:48:51.225625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.225603    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "972f82cf-1f6d-4508-8339-4d742dfa568d" (UID: "972f82cf-1f6d-4508-8339-4d742dfa568d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:48:51.326761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.326712    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972f82cf-1f6d-4508-8339-4d742dfa568d-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:48:51.946072 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.946044    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"
Jul 17 07:48:51.946538 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.946077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl" event={"ID":"972f82cf-1f6d-4508-8339-4d742dfa568d","Type":"ContainerDied","Data":"3ed65fca98efc6b074680330d97ad190fbe8f9e70a11d41b977eecb340d8f85b"}
Jul 17 07:48:51.946538 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.946138    2584 scope.go:117] "RemoveContainer" containerID="03c19fc4f79e10689f4bc832a6c0e4d6b49796ca7662ca817ca4520f52dc0125"
Jul 17 07:48:51.967293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.967259    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:51.972455 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:51.972429    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-qznkl"]
Jul 17 07:48:52.290635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:52.290603    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" path="/var/lib/kubelet/pods/26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca/volumes"
Jul 17 07:48:52.290933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:52.290920    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" path="/var/lib/kubelet/pods/972f82cf-1f6d-4508-8339-4d742dfa568d/volumes"
Jul 17 07:48:58.285297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:48:58.285260    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:48:58.285731 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:48:58.285449    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:03.285251 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:03.285219    2584 scope.go:117] "RemoveContainer" containerID="79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093"
Jul 17 07:49:04.005166 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:04.005130    2584 generic.go:356] "Generic (PLEG): container finished" podID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerID="885798d92eb78ebeeaeefd1443593e254d960eed83dde8e4778197985858703f" exitCode=6
Jul 17 07:49:04.005331 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:04.005181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" event={"ID":"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb","Type":"ContainerDied","Data":"885798d92eb78ebeeaeefd1443593e254d960eed83dde8e4778197985858703f"}
Jul 17 07:49:04.005331 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:04.005260    2584 scope.go:117] "RemoveContainer" containerID="79f9a9a07a066da6877023541fa5fba70a11ebe34470ac236dba5d3c32986093"
Jul 17 07:49:04.005612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:04.005594    2584 scope.go:117] "RemoveContainer" containerID="885798d92eb78ebeeaeefd1443593e254d960eed83dde8e4778197985858703f"
Jul 17 07:49:04.005802 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:04.005783    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7_odh-ai-gateway-infra(136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb"
Jul 17 07:49:05.031259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:05.031227    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:49:05.159524 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:05.159499    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:49:05.238401 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:05.238369    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca\") pod \"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb\" (UID: \"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb\") "
Jul 17 07:49:05.238721 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:05.238698    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" (UID: "136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:49:05.339136 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:05.339097    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:49:06.016489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.016460    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"
Jul 17 07:49:06.016489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.016477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7" event={"ID":"136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb","Type":"ContainerDied","Data":"5ef02f0f06720ab5db788890c296577b558cd3ff8e35ba411448cbabd9e66478"}
Jul 17 07:49:06.016734 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.016534    2584 scope.go:117] "RemoveContainer" containerID="885798d92eb78ebeeaeefd1443593e254d960eed83dde8e4778197985858703f"
Jul 17 07:49:06.039182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.039153    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:49:06.043190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.043165    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-92rp7"]
Jul 17 07:49:06.289840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:06.289810    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" path="/var/lib/kubelet/pods/136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb/volumes"
Jul 17 07:49:13.285764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:13.285732    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:49:14.051135 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:14.051104    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121" exitCode=1
Jul 17 07:49:14.051307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:14.051180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"}
Jul 17 07:49:14.051307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:14.051235    2584 scope.go:117] "RemoveContainer" containerID="4e75da4cf3890bce51aed411c8858960fde9864a81bac49004d6440416a968f1"
Jul 17 07:49:14.051618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:14.051600    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:14.051866 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:14.051845    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:14.240710 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:14.240638    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:49:15.057422 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:15.057393    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:15.057815 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:15.057557    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:16.824704 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:16.824671    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:49:16.889519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:16.889478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:49:19.240204 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:19.240174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:49:19.240618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:19.240533    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:19.240705 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:19.240687    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:21.547355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547315    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547631    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547643    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547652    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547658    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547668    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547675    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547681    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547688    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547698    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547716    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547725    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547730    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.547736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547739    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547745    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547751    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547757    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547800    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547806    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547812    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547818    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547824    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547830    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.548118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.547836    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="136a507e-aa2d-4ca9-ae3f-48d3b47fb4fb" containerName="revoke-keys"
Jul 17 07:49:21.550707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.550690    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:21.553235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.553214    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:49:21.557730 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.557706    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:21.567006 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.566981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb\" (UID: \"1de704c5-6bb5-45c9-8f7e-f52cb1438b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:21.668201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.668162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb\" (UID: \"1de704c5-6bb5-45c9-8f7e-f52cb1438b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:21.668815 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.668794    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb\" (UID: \"1de704c5-6bb5-45c9-8f7e-f52cb1438b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:21.861765 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.861677    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:21.993517 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:21.993494    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:21.995734 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:49:21.995684    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de704c5_6bb5_45c9_8f7e_f52cb1438b82.slice/crio-6f9b11cf139708bb8f5157e425478315c4b10324993c86ba90480351287e967a WatchSource:0}: Error finding container 6f9b11cf139708bb8f5157e425478315c4b10324993c86ba90480351287e967a: Status 404 returned error can't find the container with id 6f9b11cf139708bb8f5157e425478315c4b10324993c86ba90480351287e967a
Jul 17 07:49:22.091211 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:22.091173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" event={"ID":"1de704c5-6bb5-45c9-8f7e-f52cb1438b82","Type":"ContainerStarted","Data":"6f9b11cf139708bb8f5157e425478315c4b10324993c86ba90480351287e967a"}
Jul 17 07:49:23.097316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:23.097280    2584 generic.go:356] "Generic (PLEG): container finished" podID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerID="7b555930ccb1de9291b00b07756d6ccd361c3fcd373f4da7574a7c52ee9e3fc8" exitCode=6
Jul 17 07:49:23.097733 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:23.097367    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" event={"ID":"1de704c5-6bb5-45c9-8f7e-f52cb1438b82","Type":"ContainerDied","Data":"7b555930ccb1de9291b00b07756d6ccd361c3fcd373f4da7574a7c52ee9e3fc8"}
Jul 17 07:49:23.097733 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:23.097525    2584 scope.go:117] "RemoveContainer" containerID="7b555930ccb1de9291b00b07756d6ccd361c3fcd373f4da7574a7c52ee9e3fc8"
Jul 17 07:49:24.103296 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:24.103260    2584 generic.go:356] "Generic (PLEG): container finished" podID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerID="e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a" exitCode=6
Jul 17 07:49:24.103732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:24.103354    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" event={"ID":"1de704c5-6bb5-45c9-8f7e-f52cb1438b82","Type":"ContainerDied","Data":"e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a"}
Jul 17 07:49:24.103732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:24.103416    2584 scope.go:117] "RemoveContainer" containerID="7b555930ccb1de9291b00b07756d6ccd361c3fcd373f4da7574a7c52ee9e3fc8"
Jul 17 07:49:24.103732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:24.103642    2584 scope.go:117] "RemoveContainer" containerID="e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a"
Jul 17 07:49:24.103864 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:24.103846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb_odh-ai-gateway-infra(1de704c5-6bb5-45c9-8f7e-f52cb1438b82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82"
Jul 17 07:49:25.109288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:25.109263    2584 scope.go:117] "RemoveContainer" containerID="e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a"
Jul 17 07:49:25.109678 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:25.109456    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb_odh-ai-gateway-infra(1de704c5-6bb5-45c9-8f7e-f52cb1438b82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82"
Jul 17 07:49:28.794224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.794186    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:28.794747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.794710    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:28.794747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.794731    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:28.794872 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.794820    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="972f82cf-1f6d-4508-8339-4d742dfa568d" containerName="revoke-keys"
Jul 17 07:49:28.794872 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.794834    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b5a1ee-c35e-455f-9881-d1ef6ed0f5ca" containerName="revoke-keys"
Jul 17 07:49:28.798873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.798852    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:28.806076 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.806049    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:28.828636 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.828610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6\" (UID: \"99b9b7f1-ae73-482d-bf20-44dec095be36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:28.929841 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.929808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6\" (UID: \"99b9b7f1-ae73-482d-bf20-44dec095be36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:28.930404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:28.930384    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6\" (UID: \"99b9b7f1-ae73-482d-bf20-44dec095be36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:29.109201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:29.109120    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:29.239983 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:29.239954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:29.242164 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:49:29.242136    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b9b7f1_ae73_482d_bf20_44dec095be36.slice/crio-36ccd9d47f93eeee9bead4bbab044decbdaa39ab76a9ada3ba1e511fc7a66a18 WatchSource:0}: Error finding container 36ccd9d47f93eeee9bead4bbab044decbdaa39ab76a9ada3ba1e511fc7a66a18: Status 404 returned error can't find the container with id 36ccd9d47f93eeee9bead4bbab044decbdaa39ab76a9ada3ba1e511fc7a66a18
Jul 17 07:49:30.131224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:30.131139    2584 generic.go:356] "Generic (PLEG): container finished" podID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerID="abeba8a67cf29c81c559d23d831d8355ceef1215ab23ac473545d620334b9236" exitCode=6
Jul 17 07:49:30.131590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:30.131230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" event={"ID":"99b9b7f1-ae73-482d-bf20-44dec095be36","Type":"ContainerDied","Data":"abeba8a67cf29c81c559d23d831d8355ceef1215ab23ac473545d620334b9236"}
Jul 17 07:49:30.131590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:30.131306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" event={"ID":"99b9b7f1-ae73-482d-bf20-44dec095be36","Type":"ContainerStarted","Data":"36ccd9d47f93eeee9bead4bbab044decbdaa39ab76a9ada3ba1e511fc7a66a18"}
Jul 17 07:49:30.131590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:30.131389    2584 scope.go:117] "RemoveContainer" containerID="abeba8a67cf29c81c559d23d831d8355ceef1215ab23ac473545d620334b9236"
Jul 17 07:49:30.285300 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:30.285273    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:30.285493 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:30.285472    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:31.138162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:31.138131    2584 generic.go:356] "Generic (PLEG): container finished" podID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerID="0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b" exitCode=6
Jul 17 07:49:31.138625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:31.138191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" event={"ID":"99b9b7f1-ae73-482d-bf20-44dec095be36","Type":"ContainerDied","Data":"0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b"}
Jul 17 07:49:31.138625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:31.138234    2584 scope.go:117] "RemoveContainer" containerID="abeba8a67cf29c81c559d23d831d8355ceef1215ab23ac473545d620334b9236"
Jul 17 07:49:31.138625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:31.138455    2584 scope.go:117] "RemoveContainer" containerID="0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b"
Jul 17 07:49:31.138766 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:31.138690    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6_odh-ai-gateway-infra(99b9b7f1-ae73-482d-bf20-44dec095be36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36"
Jul 17 07:49:32.143910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:32.143877    2584 scope.go:117] "RemoveContainer" containerID="0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b"
Jul 17 07:49:32.144370 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:32.144098    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6_odh-ai-gateway-infra(99b9b7f1-ae73-482d-bf20-44dec095be36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36"
Jul 17 07:49:39.284834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:39.284804    2584 scope.go:117] "RemoveContainer" containerID="e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a"
Jul 17 07:49:40.179758 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:40.179662    2584 generic.go:356] "Generic (PLEG): container finished" podID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerID="05457010f54f9e2be320589d235535a741e5e9ef904621901960e45689f306b3" exitCode=6
Jul 17 07:49:40.179932 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:40.179752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" event={"ID":"1de704c5-6bb5-45c9-8f7e-f52cb1438b82","Type":"ContainerDied","Data":"05457010f54f9e2be320589d235535a741e5e9ef904621901960e45689f306b3"}
Jul 17 07:49:40.179932 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:40.179820    2584 scope.go:117] "RemoveContainer" containerID="e1309af88b746fb893fea94cd22a66025fb5a34ccec80132cf4f9e900633074a"
Jul 17 07:49:40.180031 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:40.180022    2584 scope.go:117] "RemoveContainer" containerID="05457010f54f9e2be320589d235535a741e5e9ef904621901960e45689f306b3"
Jul 17 07:49:40.180225 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:40.180208    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb_odh-ai-gateway-infra(1de704c5-6bb5-45c9-8f7e-f52cb1438b82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82"
Jul 17 07:49:41.210430 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:41.210398    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:41.343778 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:41.343757    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:41.430515 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:41.430472    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca\") pod \"1de704c5-6bb5-45c9-8f7e-f52cb1438b82\" (UID: \"1de704c5-6bb5-45c9-8f7e-f52cb1438b82\") "
Jul 17 07:49:41.430895 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:41.430873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1de704c5-6bb5-45c9-8f7e-f52cb1438b82" (UID: "1de704c5-6bb5-45c9-8f7e-f52cb1438b82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:49:41.531178 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:41.531092    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de704c5-6bb5-45c9-8f7e-f52cb1438b82-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:49:42.192067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.192026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb" event={"ID":"1de704c5-6bb5-45c9-8f7e-f52cb1438b82","Type":"ContainerDied","Data":"6f9b11cf139708bb8f5157e425478315c4b10324993c86ba90480351287e967a"}
Jul 17 07:49:42.192067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.192067    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"
Jul 17 07:49:42.192319 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.192098    2584 scope.go:117] "RemoveContainer" containerID="05457010f54f9e2be320589d235535a741e5e9ef904621901960e45689f306b3"
Jul 17 07:49:42.215578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.215554    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:42.224091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.224068    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-pc7nb"]
Jul 17 07:49:42.289703 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:42.289677    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" path="/var/lib/kubelet/pods/1de704c5-6bb5-45c9-8f7e-f52cb1438b82/volumes"
Jul 17 07:49:43.285704 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:43.285677    2584 scope.go:117] "RemoveContainer" containerID="0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b"
Jul 17 07:49:44.202366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:44.202273    2584 generic.go:356] "Generic (PLEG): container finished" podID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerID="4235a831f189637658b05a3664aa595902f58554bb9fae80d508611e429a205f" exitCode=6
Jul 17 07:49:44.202366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:44.202355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" event={"ID":"99b9b7f1-ae73-482d-bf20-44dec095be36","Type":"ContainerDied","Data":"4235a831f189637658b05a3664aa595902f58554bb9fae80d508611e429a205f"}
Jul 17 07:49:44.202587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:44.202435    2584 scope.go:117] "RemoveContainer" containerID="0bbacb11532a8a6e62edc999f7cd3da4a3646206ff32510402c1fac26d65780b"
Jul 17 07:49:44.202736 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:44.202718    2584 scope.go:117] "RemoveContainer" containerID="4235a831f189637658b05a3664aa595902f58554bb9fae80d508611e429a205f"
Jul 17 07:49:44.202958 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:44.202937    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6_odh-ai-gateway-infra(99b9b7f1-ae73-482d-bf20-44dec095be36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36"
Jul 17 07:49:44.286719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:44.286696    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:44.287234 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:44.286859    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:45.227918 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:45.227887    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:45.365048 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:45.365024    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:45.464397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:45.464371    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca\") pod \"99b9b7f1-ae73-482d-bf20-44dec095be36\" (UID: \"99b9b7f1-ae73-482d-bf20-44dec095be36\") "
Jul 17 07:49:45.464701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:45.464679    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99b9b7f1-ae73-482d-bf20-44dec095be36" (UID: "99b9b7f1-ae73-482d-bf20-44dec095be36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:49:45.565407 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:45.565385    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b9b7f1-ae73-482d-bf20-44dec095be36-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:49:46.213571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.213545    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"
Jul 17 07:49:46.213571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.213559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6" event={"ID":"99b9b7f1-ae73-482d-bf20-44dec095be36","Type":"ContainerDied","Data":"36ccd9d47f93eeee9bead4bbab044decbdaa39ab76a9ada3ba1e511fc7a66a18"}
Jul 17 07:49:46.213799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.213612    2584 scope.go:117] "RemoveContainer" containerID="4235a831f189637658b05a3664aa595902f58554bb9fae80d508611e429a205f"
Jul 17 07:49:46.235312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.235286    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:46.240717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.240694    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-cd2z6"]
Jul 17 07:49:46.289793 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:46.289765    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" path="/var/lib/kubelet/pods/99b9b7f1-ae73-482d-bf20-44dec095be36/volumes"
Jul 17 07:49:57.919615 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:57.919578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:49:58.285493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:58.285472    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:49:58.285653 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:49:58.285635    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:49:59.298326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:59.298290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:49:59.366856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:49:59.366816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:09.278048 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:09.278009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:09.384762 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:09.384699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:13.285025 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:13.284991    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:50:13.285429 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:13.285151    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:50:13.875595 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:13.875562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:13.977400 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:13.977361    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:21.592848 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.592813    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593129    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593139    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593152    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593158    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593164    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593169    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593177    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593182    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593191    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593196    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593244    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593251    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593256    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593262    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593269    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.593278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.593273    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b9b7f1-ae73-482d-bf20-44dec095be36" containerName="revoke-keys"
Jul 17 07:50:21.596122 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.596104    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:21.598618 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.598598    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:50:21.604037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.604014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:21.761086 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.761048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7\" (UID: \"af149b7b-48fb-4065-8866-48644d4e5006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:21.862435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.862363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7\" (UID: \"af149b7b-48fb-4065-8866-48644d4e5006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:21.862986 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.862964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7\" (UID: \"af149b7b-48fb-4065-8866-48644d4e5006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:21.907405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:21.907364    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:22.032940 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:22.032776    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:22.035477 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:50:22.035449    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf149b7b_48fb_4065_8866_48644d4e5006.slice/crio-81ac368035efa9d7fb62db56b3afa5f09193512e9f70fe75c6705e70a5a6c8a3 WatchSource:0}: Error finding container 81ac368035efa9d7fb62db56b3afa5f09193512e9f70fe75c6705e70a5a6c8a3: Status 404 returned error can't find the container with id 81ac368035efa9d7fb62db56b3afa5f09193512e9f70fe75c6705e70a5a6c8a3
Jul 17 07:50:22.370814 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:22.370763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerStarted","Data":"2c2d77aebe3f233fb387c7a3c98868bb82bf34bd9a13c6d85068a42fe290387a"}
Jul 17 07:50:22.370942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:22.370870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerStarted","Data":"81ac368035efa9d7fb62db56b3afa5f09193512e9f70fe75c6705e70a5a6c8a3"}
Jul 17 07:50:22.387823 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:22.387771    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" podStartSLOduration=1.387745032 podStartE2EDuration="1.387745032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:50:22.385931493 +0000 UTC m=+2318.639354303" watchObservedRunningTime="2026-07-17 07:50:22.387745032 +0000 UTC m=+2318.641167844"
Jul 17 07:50:23.376049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.376010    2584 generic.go:356] "Generic (PLEG): container finished" podID="af149b7b-48fb-4065-8866-48644d4e5006" containerID="2c2d77aebe3f233fb387c7a3c98868bb82bf34bd9a13c6d85068a42fe290387a" exitCode=6
Jul 17 07:50:23.376505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.376097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerDied","Data":"2c2d77aebe3f233fb387c7a3c98868bb82bf34bd9a13c6d85068a42fe290387a"}
Jul 17 07:50:23.376505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.376242    2584 scope.go:117] "RemoveContainer" containerID="2c2d77aebe3f233fb387c7a3c98868bb82bf34bd9a13c6d85068a42fe290387a"
Jul 17 07:50:23.391049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.391026    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:23.391552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.391537    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:23.391609 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.391556    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de704c5-6bb5-45c9-8f7e-f52cb1438b82" containerName="revoke-keys"
Jul 17 07:50:23.395973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.395952    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:23.401814 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.401789    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:23.474296 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.474260    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5\" (UID: \"95553ded-f1a1-4720-b554-9dcfe9da7aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:23.574862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.574831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5\" (UID: \"95553ded-f1a1-4720-b554-9dcfe9da7aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:23.575444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.575427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5\" (UID: \"95553ded-f1a1-4720-b554-9dcfe9da7aaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:23.715699 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:23.715665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:24.072563 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.072536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:24.091253 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:50:24.091212    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95553ded_f1a1_4720_b554_9dcfe9da7aaf.slice/crio-ea9b6b2408c38bb82320e1ae3b80251ef43235baea09ef9fee603e692d76ac98 WatchSource:0}: Error finding container ea9b6b2408c38bb82320e1ae3b80251ef43235baea09ef9fee603e692d76ac98: Status 404 returned error can't find the container with id ea9b6b2408c38bb82320e1ae3b80251ef43235baea09ef9fee603e692d76ac98
Jul 17 07:50:24.383050 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.383005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerStarted","Data":"e814d29308f879bcbcf1cf998fc5f63389d9bf5e698a804b0e11d1ae9406d3df"}
Jul 17 07:50:24.383050 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.383052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerStarted","Data":"ea9b6b2408c38bb82320e1ae3b80251ef43235baea09ef9fee603e692d76ac98"}
Jul 17 07:50:24.384804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.384773    2584 generic.go:356] "Generic (PLEG): container finished" podID="af149b7b-48fb-4065-8866-48644d4e5006" containerID="c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17" exitCode=6
Jul 17 07:50:24.384958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.384817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerDied","Data":"c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17"}
Jul 17 07:50:24.384958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.384866    2584 scope.go:117] "RemoveContainer" containerID="2c2d77aebe3f233fb387c7a3c98868bb82bf34bd9a13c6d85068a42fe290387a"
Jul 17 07:50:24.385082 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.384999    2584 scope.go:117] "RemoveContainer" containerID="c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17"
Jul 17 07:50:24.385180 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:24.385154    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7_odh-ai-gateway-infra(af149b7b-48fb-4065-8866-48644d4e5006)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" podUID="af149b7b-48fb-4065-8866-48644d4e5006"
Jul 17 07:50:24.399930 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:24.399886    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" podStartSLOduration=1.39987167 podStartE2EDuration="1.39987167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:50:24.397391957 +0000 UTC m=+2320.650814768" watchObservedRunningTime="2026-07-17 07:50:24.39987167 +0000 UTC m=+2320.653294480"
Jul 17 07:50:25.284798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:25.284766    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:50:25.284978 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:25.284929    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:50:25.390491 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:25.390466    2584 scope.go:117] "RemoveContainer" containerID="c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17"
Jul 17 07:50:25.390908 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:25.390687    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7_odh-ai-gateway-infra(af149b7b-48fb-4065-8866-48644d4e5006)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" podUID="af149b7b-48fb-4065-8866-48644d4e5006"
Jul 17 07:50:25.391686 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:25.391660    2584 generic.go:356] "Generic (PLEG): container finished" podID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerID="e814d29308f879bcbcf1cf998fc5f63389d9bf5e698a804b0e11d1ae9406d3df" exitCode=7
Jul 17 07:50:25.391824 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:25.391722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerDied","Data":"e814d29308f879bcbcf1cf998fc5f63389d9bf5e698a804b0e11d1ae9406d3df"}
Jul 17 07:50:25.391942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:25.391925    2584 scope.go:117] "RemoveContainer" containerID="e814d29308f879bcbcf1cf998fc5f63389d9bf5e698a804b0e11d1ae9406d3df"
Jul 17 07:50:26.398240 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:26.398202    2584 generic.go:356] "Generic (PLEG): container finished" podID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerID="3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3" exitCode=7
Jul 17 07:50:26.398711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:26.398287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerDied","Data":"3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3"}
Jul 17 07:50:26.398711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:26.398365    2584 scope.go:117] "RemoveContainer" containerID="e814d29308f879bcbcf1cf998fc5f63389d9bf5e698a804b0e11d1ae9406d3df"
Jul 17 07:50:26.398711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:26.398591    2584 scope.go:117] "RemoveContainer" containerID="3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3"
Jul 17 07:50:26.398837 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:26.398800    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5_odh-ai-gateway-infra(95553ded-f1a1-4720-b554-9dcfe9da7aaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf"
Jul 17 07:50:27.403590 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:27.403565    2584 scope.go:117] "RemoveContainer" containerID="3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3"
Jul 17 07:50:27.403958 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:27.403726    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5_odh-ai-gateway-infra(95553ded-f1a1-4720-b554-9dcfe9da7aaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf"
Jul 17 07:50:33.399409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:33.399374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:33.500283 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:33.500247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:36.339373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:36.339327    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:50:37.285392 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:37.285333    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:50:38.454813 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:38.454726    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5a013de-b564-438b-95bb-b60ca4585f22" containerID="980c8f77073b8604a43c01b413d6a4f11666d150603a64dc7ba2f313e53ca888" exitCode=1
Jul 17 07:50:38.455253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:38.454803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"980c8f77073b8604a43c01b413d6a4f11666d150603a64dc7ba2f313e53ca888"}
Jul 17 07:50:38.455253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:38.454878    2584 scope.go:117] "RemoveContainer" containerID="fb941e315a0b68b0b79ed03a9beaf5e04d69cee34b8c6104ce7f0852cdad3121"
Jul 17 07:50:38.455386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:38.455263    2584 scope.go:117] "RemoveContainer" containerID="980c8f77073b8604a43c01b413d6a4f11666d150603a64dc7ba2f313e53ca888"
Jul 17 07:50:38.455501 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:38.455480    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:50:39.240611 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.240577    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:50:39.240785 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.240638    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:50:39.285320 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.285288    2584 scope.go:117] "RemoveContainer" containerID="c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17"
Jul 17 07:50:39.461087 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.461058    2584 scope.go:117] "RemoveContainer" containerID="980c8f77073b8604a43c01b413d6a4f11666d150603a64dc7ba2f313e53ca888"
Jul 17 07:50:39.461521 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:39.461237    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8_odh-ai-gateway-infra(f5a013de-b564-438b-95bb-b60ca4585f22)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" podUID="f5a013de-b564-438b-95bb-b60ca4585f22"
Jul 17 07:50:39.534682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.534652    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:50:39.562321 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.562288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:39.670797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:39.670766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:40.467292 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.467260    2584 generic.go:356] "Generic (PLEG): container finished" podID="af149b7b-48fb-4065-8866-48644d4e5006" containerID="2fd9857d59012a2a8e9442cea2554036115b48f392f583a3fbe8d04ee6043b17" exitCode=6
Jul 17 07:50:40.467774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.467349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerDied","Data":"2fd9857d59012a2a8e9442cea2554036115b48f392f583a3fbe8d04ee6043b17"}
Jul 17 07:50:40.467774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.467418    2584 scope.go:117] "RemoveContainer" containerID="c250996a7f241ddbb5ac103eb3b3db914f70c54c5a978fdd845cbdc5a3d22e17"
Jul 17 07:50:40.467774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.467701    2584 scope.go:117] "RemoveContainer" containerID="2fd9857d59012a2a8e9442cea2554036115b48f392f583a3fbe8d04ee6043b17"
Jul 17 07:50:40.468016 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:40.467887    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7_odh-ai-gateway-infra(af149b7b-48fb-4065-8866-48644d4e5006)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" podUID="af149b7b-48fb-4065-8866-48644d4e5006"
Jul 17 07:50:40.603919 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.603893    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:50:40.702210 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.702181    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls\") pod \"f5a013de-b564-438b-95bb-b60ca4585f22\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") "
Jul 17 07:50:40.702422 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.702222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca\") pod \"f5a013de-b564-438b-95bb-b60ca4585f22\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") "
Jul 17 07:50:40.702422 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.702263    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4hkc\" (UniqueName: \"kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc\") pod \"f5a013de-b564-438b-95bb-b60ca4585f22\" (UID: \"f5a013de-b564-438b-95bb-b60ca4585f22\") "
Jul 17 07:50:40.702682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.702659    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5a013de-b564-438b-95bb-b60ca4585f22" (UID: "f5a013de-b564-438b-95bb-b60ca4585f22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:50:40.704250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.704227    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5a013de-b564-438b-95bb-b60ca4585f22" (UID: "f5a013de-b564-438b-95bb-b60ca4585f22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:50:40.704360 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.704291    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc" (OuterVolumeSpecName: "kube-api-access-c4hkc") pod "f5a013de-b564-438b-95bb-b60ca4585f22" (UID: "f5a013de-b564-438b-95bb-b60ca4585f22"). InnerVolumeSpecName "kube-api-access-c4hkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:50:40.803647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.803593    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hkc\" (UniqueName: \"kubernetes.io/projected/f5a013de-b564-438b-95bb-b60ca4585f22-kube-api-access-c4hkc\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:50:40.803647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.803640    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a013de-b564-438b-95bb-b60ca4585f22-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:50:40.803647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:40.803653    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a013de-b564-438b-95bb-b60ca4585f22-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:50:41.285170 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.285142    2584 scope.go:117] "RemoveContainer" containerID="3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3"
Jul 17 07:50:41.474998 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.474962    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8" event={"ID":"f5a013de-b564-438b-95bb-b60ca4585f22","Type":"ContainerDied","Data":"31d93ad287876099d4add22e4599074d5002220e7ed8e5eb03fa686b0f1a2de0"}
Jul 17 07:50:41.474998 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.474991    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"
Jul 17 07:50:41.475494 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.475036    2584 scope.go:117] "RemoveContainer" containerID="980c8f77073b8604a43c01b413d6a4f11666d150603a64dc7ba2f313e53ca888"
Jul 17 07:50:41.497620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.497588    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:50:41.501497 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.501474    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f44a26e0-8c6bbcfb-vgcf8"]
Jul 17 07:50:41.510313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.510290    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:41.647137 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.647112    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:41.710432 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.710401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca\") pod \"af149b7b-48fb-4065-8866-48644d4e5006\" (UID: \"af149b7b-48fb-4065-8866-48644d4e5006\") "
Jul 17 07:50:41.710773 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.710752    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af149b7b-48fb-4065-8866-48644d4e5006" (UID: "af149b7b-48fb-4065-8866-48644d4e5006"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:50:41.811694 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:41.811606    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af149b7b-48fb-4065-8866-48644d4e5006-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:50:42.290294 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.290261    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" path="/var/lib/kubelet/pods/f5a013de-b564-438b-95bb-b60ca4585f22/volumes"
Jul 17 07:50:42.480826 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.480796    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"
Jul 17 07:50:42.481278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.480798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7" event={"ID":"af149b7b-48fb-4065-8866-48644d4e5006","Type":"ContainerDied","Data":"81ac368035efa9d7fb62db56b3afa5f09193512e9f70fe75c6705e70a5a6c8a3"}
Jul 17 07:50:42.481278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.480894    2584 scope.go:117] "RemoveContainer" containerID="2fd9857d59012a2a8e9442cea2554036115b48f392f583a3fbe8d04ee6043b17"
Jul 17 07:50:42.483852 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.483828    2584 generic.go:356] "Generic (PLEG): container finished" podID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerID="6410a96913d72befc4a1b8e8039518988228206393fb75ba60d3364543656b6d" exitCode=6
Jul 17 07:50:42.483946 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.483905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerDied","Data":"6410a96913d72befc4a1b8e8039518988228206393fb75ba60d3364543656b6d"}
Jul 17 07:50:42.484249 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.484232    2584 scope.go:117] "RemoveContainer" containerID="6410a96913d72befc4a1b8e8039518988228206393fb75ba60d3364543656b6d"
Jul 17 07:50:42.484488 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:42.484468    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5_odh-ai-gateway-infra(95553ded-f1a1-4720-b554-9dcfe9da7aaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf"
Jul 17 07:50:42.489229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.489202    2584 scope.go:117] "RemoveContainer" containerID="3479069af42637b6e1437321d23115cdc78827c35996de7b63dbeb4bcab866e3"
Jul 17 07:50:42.496827 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.496806    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:42.501112 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:42.501091    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-7mmk7"]
Jul 17 07:50:43.530674 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:43.530638    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:43.662447 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:43.662421    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:43.728088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:43.728053    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca\") pod \"95553ded-f1a1-4720-b554-9dcfe9da7aaf\" (UID: \"95553ded-f1a1-4720-b554-9dcfe9da7aaf\") "
Jul 17 07:50:43.728446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:43.728423    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95553ded-f1a1-4720-b554-9dcfe9da7aaf" (UID: "95553ded-f1a1-4720-b554-9dcfe9da7aaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:50:43.829278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:43.829241    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95553ded-f1a1-4720-b554-9dcfe9da7aaf-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:50:44.291411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.291382    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af149b7b-48fb-4065-8866-48644d4e5006" path="/var/lib/kubelet/pods/af149b7b-48fb-4065-8866-48644d4e5006/volumes"
Jul 17 07:50:44.496694 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.496664    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"
Jul 17 07:50:44.496891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.496696    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5" event={"ID":"95553ded-f1a1-4720-b554-9dcfe9da7aaf","Type":"ContainerDied","Data":"ea9b6b2408c38bb82320e1ae3b80251ef43235baea09ef9fee603e692d76ac98"}
Jul 17 07:50:44.496891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.496761    2584 scope.go:117] "RemoveContainer" containerID="6410a96913d72befc4a1b8e8039518988228206393fb75ba60d3364543656b6d"
Jul 17 07:50:44.514250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.514223    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:44.516064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:44.516043    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-h9nb5"]
Jul 17 07:50:45.357303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.357269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:45.423547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423515    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:50:45.423816 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423804    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.423859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423818    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.423859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423830    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.423859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423836    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.423859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423843    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.423859 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423848    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423860    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423866    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423876    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423882    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423893    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423898    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423904    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423909    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423914    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423919    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423925    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423929    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423976    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423983    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423989    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.423996    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.424001    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.424009    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:50:45.424011 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.424015    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424565 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.424019    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:50:45.424565 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.424025    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:50:45.428428 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.428409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:50:45.430236 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.430211    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:45.430798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.430779    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:50:45.430852 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.430824    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:50:45.435970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.435940    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:50:45.542824 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.542786    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9\" (UID: \"ba2bdb60-9d38-4807-93d0-68b56139ab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:50:45.644200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.644110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9\" (UID: \"ba2bdb60-9d38-4807-93d0-68b56139ab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:50:45.644772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.644753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9\" (UID: \"ba2bdb60-9d38-4807-93d0-68b56139ab2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:50:45.740370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.740305    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:50:45.905115 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.905089    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:50:45.907954 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:50:45.907905    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2bdb60_9d38_4807_93d0_68b56139ab2c.slice/crio-6793b9d43d46c815ae790f9dfbb81845142920a3592c2d2e6d1a941657b4a4c8 WatchSource:0}: Error finding container 6793b9d43d46c815ae790f9dfbb81845142920a3592c2d2e6d1a941657b4a4c8: Status 404 returned error can't find the container with id 6793b9d43d46c815ae790f9dfbb81845142920a3592c2d2e6d1a941657b4a4c8
Jul 17 07:50:45.980481 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:45.980447    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:46.067297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:46.067260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:46.290302 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:46.290273    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" path="/var/lib/kubelet/pods/95553ded-f1a1-4720-b554-9dcfe9da7aaf/volumes"
Jul 17 07:50:46.506966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:46.506872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerStarted","Data":"023eb12eb4431052738326ea4f3c25b7f3ddb10e3a50bd9d1700b76ac056dafb"}
Jul 17 07:50:46.506966 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:46.506938    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerStarted","Data":"6793b9d43d46c815ae790f9dfbb81845142920a3592c2d2e6d1a941657b4a4c8"}
Jul 17 07:50:46.527854 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:46.527807    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" podStartSLOduration=1.5277893630000001 podStartE2EDuration="1.527789363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:50:46.525932623 +0000 UTC m=+2342.779355437" watchObservedRunningTime="2026-07-17 07:50:46.527789363 +0000 UTC m=+2342.781212174"
Jul 17 07:50:47.512223 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:47.512188    2584 generic.go:356] "Generic (PLEG): container finished" podID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerID="023eb12eb4431052738326ea4f3c25b7f3ddb10e3a50bd9d1700b76ac056dafb" exitCode=6
Jul 17 07:50:47.512673 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:47.512286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerDied","Data":"023eb12eb4431052738326ea4f3c25b7f3ddb10e3a50bd9d1700b76ac056dafb"}
Jul 17 07:50:47.512673 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:47.512463    2584 scope.go:117] "RemoveContainer" containerID="023eb12eb4431052738326ea4f3c25b7f3ddb10e3a50bd9d1700b76ac056dafb"
Jul 17 07:50:47.747528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:47.747494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:48.519394 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:48.519282    2584 generic.go:356] "Generic (PLEG): container finished" podID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerID="ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8" exitCode=6
Jul 17 07:50:48.519394 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:48.519363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerDied","Data":"ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8"}
Jul 17 07:50:48.519935 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:48.519435    2584 scope.go:117] "RemoveContainer" containerID="023eb12eb4431052738326ea4f3c25b7f3ddb10e3a50bd9d1700b76ac056dafb"
Jul 17 07:50:48.519935 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:48.519676    2584 scope.go:117] "RemoveContainer" containerID="ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8"
Jul 17 07:50:48.519935 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:48.519869    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9_odh-ai-gateway-infra(ba2bdb60-9d38-4807-93d0-68b56139ab2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c"
Jul 17 07:50:49.527322 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:49.527297    2584 scope.go:117] "RemoveContainer" containerID="ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8"
Jul 17 07:50:49.527715 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:50:49.527500    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9_odh-ai-gateway-infra(ba2bdb60-9d38-4807-93d0-68b56139ab2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c"
Jul 17 07:50:50.058051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:50.057999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:54.138140 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:54.138106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:54.242804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:54.242759    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:54.338198 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:54.338159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:54.451237 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:54.451149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:50:54.932845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:50:54.932795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:02.284961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:02.284921    2584 scope.go:117] "RemoveContainer" containerID="ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8"
Jul 17 07:51:02.583460 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:02.583376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerStarted","Data":"c14145b95563405b4912bca43b2b0e70cd31331952d5d3ac60dc01474e68082a"}
Jul 17 07:51:03.594630 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:03.594594    2584 generic.go:356] "Generic (PLEG): container finished" podID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerID="c14145b95563405b4912bca43b2b0e70cd31331952d5d3ac60dc01474e68082a" exitCode=6
Jul 17 07:51:03.595052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:03.594668    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerDied","Data":"c14145b95563405b4912bca43b2b0e70cd31331952d5d3ac60dc01474e68082a"}
Jul 17 07:51:03.595052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:03.594751    2584 scope.go:117] "RemoveContainer" containerID="ab7272d4339e65d62d74daf635b302d25d39763c53f7ec84d9c1bb27243a62b8"
Jul 17 07:51:03.595052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:03.595015    2584 scope.go:117] "RemoveContainer" containerID="c14145b95563405b4912bca43b2b0e70cd31331952d5d3ac60dc01474e68082a"
Jul 17 07:51:03.595268 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:03.595247    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9_odh-ai-gateway-infra(ba2bdb60-9d38-4807-93d0-68b56139ab2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c"
Jul 17 07:51:03.609756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:03.609727    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:51:04.734297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:04.734271    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:51:04.810954 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:04.810920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca\") pod \"ba2bdb60-9d38-4807-93d0-68b56139ab2c\" (UID: \"ba2bdb60-9d38-4807-93d0-68b56139ab2c\") "
Jul 17 07:51:04.811405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:04.811377    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba2bdb60-9d38-4807-93d0-68b56139ab2c" (UID: "ba2bdb60-9d38-4807-93d0-68b56139ab2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:51:04.912189 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:04.912093    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2bdb60-9d38-4807-93d0-68b56139ab2c-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:51:05.604749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:05.604713    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9" event={"ID":"ba2bdb60-9d38-4807-93d0-68b56139ab2c","Type":"ContainerDied","Data":"6793b9d43d46c815ae790f9dfbb81845142920a3592c2d2e6d1a941657b4a4c8"}
Jul 17 07:51:05.604932 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:05.604770    2584 scope.go:117] "RemoveContainer" containerID="c14145b95563405b4912bca43b2b0e70cd31331952d5d3ac60dc01474e68082a"
Jul 17 07:51:05.604932 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:05.604721    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"
Jul 17 07:51:05.625225 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:05.625195    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:51:05.628958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:05.628936    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-x78t9"]
Jul 17 07:51:06.290271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:06.290239    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" path="/var/lib/kubelet/pods/ba2bdb60-9d38-4807-93d0-68b56139ab2c/volumes"
Jul 17 07:51:18.887741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.887704    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.887998    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888009    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888019    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888025    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888036    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888041    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888052    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888058    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888071    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888077    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af149b7b-48fb-4065-8866-48644d4e5006" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888130    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888137    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a013de-b564-438b-95bb-b60ca4585f22" containerName="maas-api"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="95553ded-f1a1-4720-b554-9dcfe9da7aaf" containerName="revoke-keys"
Jul 17 07:51:18.888146 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.888148    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:18.892475 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.892452    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:18.895514 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.895487    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:51:18.895648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.895487    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:51:18.899745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:18.899719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:19.036850 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.036808    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg\" (UID: \"23148880-8b21-44aa-919e-c93682fb6ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:19.137390 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.137313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg\" (UID: \"23148880-8b21-44aa-919e-c93682fb6ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:19.138016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.137959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg\" (UID: \"23148880-8b21-44aa-919e-c93682fb6ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:19.204287 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.204245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:19.341003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.340970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:19.344723 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:51:19.344695    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23148880_8b21_44aa_919e_c93682fb6ba6.slice/crio-379d5c6c3900160127464d4e63c465eb0e111cf4a97a8b14ec2275027727a9c7 WatchSource:0}: Error finding container 379d5c6c3900160127464d4e63c465eb0e111cf4a97a8b14ec2275027727a9c7: Status 404 returned error can't find the container with id 379d5c6c3900160127464d4e63c465eb0e111cf4a97a8b14ec2275027727a9c7
Jul 17 07:51:19.670477 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.670443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerStarted","Data":"c500e765ce4623efad7ea9b60d1e66e4fb95e176f5d955d17cc417a6f8126995"}
Jul 17 07:51:19.670659 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.670493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerStarted","Data":"379d5c6c3900160127464d4e63c465eb0e111cf4a97a8b14ec2275027727a9c7"}
Jul 17 07:51:19.689055 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:19.689003    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" podStartSLOduration=1.6889871589999998 podStartE2EDuration="1.688987159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:51:19.687947558 +0000 UTC m=+2375.941370370" watchObservedRunningTime="2026-07-17 07:51:19.688987159 +0000 UTC m=+2375.942409970"
Jul 17 07:51:20.676243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:20.676209    2584 generic.go:356] "Generic (PLEG): container finished" podID="23148880-8b21-44aa-919e-c93682fb6ba6" containerID="c500e765ce4623efad7ea9b60d1e66e4fb95e176f5d955d17cc417a6f8126995" exitCode=6
Jul 17 07:51:20.676649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:20.676295    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerDied","Data":"c500e765ce4623efad7ea9b60d1e66e4fb95e176f5d955d17cc417a6f8126995"}
Jul 17 07:51:20.676649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:20.676488    2584 scope.go:117] "RemoveContainer" containerID="c500e765ce4623efad7ea9b60d1e66e4fb95e176f5d955d17cc417a6f8126995"
Jul 17 07:51:20.992158 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:20.992073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:21.642594 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.642560    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:21.642897 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.642884    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:21.642942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.642900    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:21.642977 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.642956    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2bdb60-9d38-4807-93d0-68b56139ab2c" containerName="revoke-keys"
Jul 17 07:51:21.645713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.645696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:21.655049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.655020    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:21.657190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.657163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4\" (UID: \"7b1891e6-199a-4d00-ad33-063d97f90799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:21.682057 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.682025    2584 generic.go:356] "Generic (PLEG): container finished" podID="23148880-8b21-44aa-919e-c93682fb6ba6" containerID="caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f" exitCode=6
Jul 17 07:51:21.682527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.682105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerDied","Data":"caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f"}
Jul 17 07:51:21.682527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.682161    2584 scope.go:117] "RemoveContainer" containerID="c500e765ce4623efad7ea9b60d1e66e4fb95e176f5d955d17cc417a6f8126995"
Jul 17 07:51:21.682527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.682356    2584 scope.go:117] "RemoveContainer" containerID="caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f"
Jul 17 07:51:21.682652 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:21.682565    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg_odh-ai-gateway-infra(23148880-8b21-44aa-919e-c93682fb6ba6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" podUID="23148880-8b21-44aa-919e-c93682fb6ba6"
Jul 17 07:51:21.758074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.758033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4\" (UID: \"7b1891e6-199a-4d00-ad33-063d97f90799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:21.758759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.758732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4\" (UID: \"7b1891e6-199a-4d00-ad33-063d97f90799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:21.956574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:21.956479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:22.094303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:22.094275    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:22.096736 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:51:22.096709    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1891e6_199a_4d00_ad33_063d97f90799.slice/crio-611aac94abcf7d8f0f393c45969bd9089ee2a6d8460b90084f39e790b5a688d2 WatchSource:0}: Error finding container 611aac94abcf7d8f0f393c45969bd9089ee2a6d8460b90084f39e790b5a688d2: Status 404 returned error can't find the container with id 611aac94abcf7d8f0f393c45969bd9089ee2a6d8460b90084f39e790b5a688d2
Jul 17 07:51:22.687862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:22.687769    2584 scope.go:117] "RemoveContainer" containerID="caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f"
Jul 17 07:51:22.688394 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:22.687975    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg_odh-ai-gateway-infra(23148880-8b21-44aa-919e-c93682fb6ba6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" podUID="23148880-8b21-44aa-919e-c93682fb6ba6"
Jul 17 07:51:22.688887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:22.688863    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerStarted","Data":"81bcef8e851eb70505e0fe32c3dbfc49d2d2de5651039cd0bb987cbc79c66152"}
Jul 17 07:51:22.688952 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:22.688910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerStarted","Data":"611aac94abcf7d8f0f393c45969bd9089ee2a6d8460b90084f39e790b5a688d2"}
Jul 17 07:51:22.718531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:22.718466    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" podStartSLOduration=1.718450467 podStartE2EDuration="1.718450467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:51:22.717183399 +0000 UTC m=+2378.970606212" watchObservedRunningTime="2026-07-17 07:51:22.718450467 +0000 UTC m=+2378.971873278"
Jul 17 07:51:23.694727 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:23.694692    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b1891e6-199a-4d00-ad33-063d97f90799" containerID="81bcef8e851eb70505e0fe32c3dbfc49d2d2de5651039cd0bb987cbc79c66152" exitCode=6
Jul 17 07:51:23.695106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:23.694775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerDied","Data":"81bcef8e851eb70505e0fe32c3dbfc49d2d2de5651039cd0bb987cbc79c66152"}
Jul 17 07:51:23.695106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:23.694956    2584 scope.go:117] "RemoveContainer" containerID="81bcef8e851eb70505e0fe32c3dbfc49d2d2de5651039cd0bb987cbc79c66152"
Jul 17 07:51:24.701027 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:24.700941    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b1891e6-199a-4d00-ad33-063d97f90799" containerID="38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8" exitCode=6
Jul 17 07:51:24.701027 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:24.700999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerDied","Data":"38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8"}
Jul 17 07:51:24.701474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:24.701049    2584 scope.go:117] "RemoveContainer" containerID="81bcef8e851eb70505e0fe32c3dbfc49d2d2de5651039cd0bb987cbc79c66152"
Jul 17 07:51:24.701474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:24.701185    2584 scope.go:117] "RemoveContainer" containerID="38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8"
Jul 17 07:51:24.701474 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:24.701431    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4_odh-ai-gateway-infra(7b1891e6-199a-4d00-ad33-063d97f90799)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" podUID="7b1891e6-199a-4d00-ad33-063d97f90799"
Jul 17 07:51:25.707257 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:25.707227    2584 scope.go:117] "RemoveContainer" containerID="38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8"
Jul 17 07:51:25.707724 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:25.707427    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4_odh-ai-gateway-infra(7b1891e6-199a-4d00-ad33-063d97f90799)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" podUID="7b1891e6-199a-4d00-ad33-063d97f90799"
Jul 17 07:51:27.301914 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:27.301880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:27.404638 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:27.404591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:27.496084 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:27.496042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:33.284815 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:33.284783    2584 scope.go:117] "RemoveContainer" containerID="caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f"
Jul 17 07:51:33.749019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:33.748929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerStarted","Data":"f747c1a43dc140d0cdbb24f14ce0440e7212cda2202f2eb127c49e41eea81119"}
Jul 17 07:51:34.754857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:34.754825    2584 generic.go:356] "Generic (PLEG): container finished" podID="23148880-8b21-44aa-919e-c93682fb6ba6" containerID="f747c1a43dc140d0cdbb24f14ce0440e7212cda2202f2eb127c49e41eea81119" exitCode=6
Jul 17 07:51:34.754857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:34.754862    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerDied","Data":"f747c1a43dc140d0cdbb24f14ce0440e7212cda2202f2eb127c49e41eea81119"}
Jul 17 07:51:34.755303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:34.754913    2584 scope.go:117] "RemoveContainer" containerID="caa4246876d566d681505889c4f21ca74ab2530ec54728d229b7f7ab10e5b36f"
Jul 17 07:51:34.755303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:34.755186    2584 scope.go:117] "RemoveContainer" containerID="f747c1a43dc140d0cdbb24f14ce0440e7212cda2202f2eb127c49e41eea81119"
Jul 17 07:51:34.755409 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:34.755390    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg_odh-ai-gateway-infra(23148880-8b21-44aa-919e-c93682fb6ba6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" podUID="23148880-8b21-44aa-919e-c93682fb6ba6"
Jul 17 07:51:34.785162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:34.785124    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:35.892557 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:35.892528    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:35.980255 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:35.980220    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca\") pod \"23148880-8b21-44aa-919e-c93682fb6ba6\" (UID: \"23148880-8b21-44aa-919e-c93682fb6ba6\") "
Jul 17 07:51:35.980638 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:35.980614    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23148880-8b21-44aa-919e-c93682fb6ba6" (UID: "23148880-8b21-44aa-919e-c93682fb6ba6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:51:36.081714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.081684    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23148880-8b21-44aa-919e-c93682fb6ba6-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:51:36.768037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.768004    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"
Jul 17 07:51:36.768221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.768057    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg" event={"ID":"23148880-8b21-44aa-919e-c93682fb6ba6","Type":"ContainerDied","Data":"379d5c6c3900160127464d4e63c465eb0e111cf4a97a8b14ec2275027727a9c7"}
Jul 17 07:51:36.768221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.768113    2584 scope.go:117] "RemoveContainer" containerID="f747c1a43dc140d0cdbb24f14ce0440e7212cda2202f2eb127c49e41eea81119"
Jul 17 07:51:36.784477 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.784451    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:36.788840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:36.788810    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-9wlqg"]
Jul 17 07:51:38.289376 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:38.289327    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" path="/var/lib/kubelet/pods/23148880-8b21-44aa-919e-c93682fb6ba6/volumes"
Jul 17 07:51:40.285535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:40.285502    2584 scope.go:117] "RemoveContainer" containerID="38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8"
Jul 17 07:51:40.786968 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:40.786929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerStarted","Data":"d3295120daf0833af053356b1a18a8363406b232e729c649f7a50c6a20091000"}
Jul 17 07:51:41.792542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:41.792508    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b1891e6-199a-4d00-ad33-063d97f90799" containerID="d3295120daf0833af053356b1a18a8363406b232e729c649f7a50c6a20091000" exitCode=6
Jul 17 07:51:41.792987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:41.792552    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerDied","Data":"d3295120daf0833af053356b1a18a8363406b232e729c649f7a50c6a20091000"}
Jul 17 07:51:41.792987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:41.792598    2584 scope.go:117] "RemoveContainer" containerID="38c428d96f060b77b627217c8285c1d85bce78cd9a16c6b7b90056f74540fba8"
Jul 17 07:51:41.792987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:41.792899    2584 scope.go:117] "RemoveContainer" containerID="d3295120daf0833af053356b1a18a8363406b232e729c649f7a50c6a20091000"
Jul 17 07:51:41.793119 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:41.793096    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4_odh-ai-gateway-infra(7b1891e6-199a-4d00-ad33-063d97f90799)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" podUID="7b1891e6-199a-4d00-ad33-063d97f90799"
Jul 17 07:51:41.818839 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:41.818812    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:42.928004 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:42.927976    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:43.035408 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.035374    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca\") pod \"7b1891e6-199a-4d00-ad33-063d97f90799\" (UID: \"7b1891e6-199a-4d00-ad33-063d97f90799\") "
Jul 17 07:51:43.035797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.035773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b1891e6-199a-4d00-ad33-063d97f90799" (UID: "7b1891e6-199a-4d00-ad33-063d97f90799"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:51:43.136912 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.136821    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1891e6-199a-4d00-ad33-063d97f90799-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:51:43.509665 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.509576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:43.803907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.803874    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"
Jul 17 07:51:43.803907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.803898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4" event={"ID":"7b1891e6-199a-4d00-ad33-063d97f90799","Type":"ContainerDied","Data":"611aac94abcf7d8f0f393c45969bd9089ee2a6d8460b90084f39e790b5a688d2"}
Jul 17 07:51:43.804162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.803967    2584 scope.go:117] "RemoveContainer" containerID="d3295120daf0833af053356b1a18a8363406b232e729c649f7a50c6a20091000"
Jul 17 07:51:43.824726 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.824684    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:43.830671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:43.830647    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-5n5g4"]
Jul 17 07:51:44.290054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:44.290023    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" path="/var/lib/kubelet/pods/7b1891e6-199a-4d00-ad33-063d97f90799/volumes"
Jul 17 07:51:44.372643 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:44.372612    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:51:44.373623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:44.373603    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:51:45.473006 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.472961    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:51:45.473647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473625    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473649    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473662    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473670    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473713    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473729    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473745    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473754    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473889    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473902    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473910    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473926    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.473964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.473934    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:51:45.479267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.479242    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:51:45.481923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.481896    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:51:45.481923 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.481912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:51:45.482102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.481961    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:51:45.554690 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.554653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-txn66\" (UID: \"b0fc520a-1678-4bfd-aac2-65dfd36e8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:51:45.656092 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.656050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-txn66\" (UID: \"b0fc520a-1678-4bfd-aac2-65dfd36e8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:51:45.656703 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.656682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-txn66\" (UID: \"b0fc520a-1678-4bfd-aac2-65dfd36e8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:51:45.790261 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.790226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:51:45.948017 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:45.947990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:51:45.950672 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:51:45.950641    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fc520a_1678_4bfd_aac2_65dfd36e8cde.slice/crio-7e5c94ba50c2c0a2e5af84085e79aed5a953f6f29df516bba109bc320d4acfb5 WatchSource:0}: Error finding container 7e5c94ba50c2c0a2e5af84085e79aed5a953f6f29df516bba109bc320d4acfb5: Status 404 returned error can't find the container with id 7e5c94ba50c2c0a2e5af84085e79aed5a953f6f29df516bba109bc320d4acfb5
Jul 17 07:51:46.818352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:46.818316    2584 generic.go:356] "Generic (PLEG): container finished" podID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerID="b500358b1ad7d3fdf69cb70f391d96b33c890c66a4dd4cdb1640088430953ce6" exitCode=6
Jul 17 07:51:46.818792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:46.818385    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" event={"ID":"b0fc520a-1678-4bfd-aac2-65dfd36e8cde","Type":"ContainerDied","Data":"b500358b1ad7d3fdf69cb70f391d96b33c890c66a4dd4cdb1640088430953ce6"}
Jul 17 07:51:46.818792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:46.818433    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" event={"ID":"b0fc520a-1678-4bfd-aac2-65dfd36e8cde","Type":"ContainerStarted","Data":"7e5c94ba50c2c0a2e5af84085e79aed5a953f6f29df516bba109bc320d4acfb5"}
Jul 17 07:51:46.818792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:46.818660    2584 scope.go:117] "RemoveContainer" containerID="b500358b1ad7d3fdf69cb70f391d96b33c890c66a4dd4cdb1640088430953ce6"
Jul 17 07:51:47.825145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:47.825112    2584 generic.go:356] "Generic (PLEG): container finished" podID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerID="f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66" exitCode=6
Jul 17 07:51:47.825586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:47.825159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" event={"ID":"b0fc520a-1678-4bfd-aac2-65dfd36e8cde","Type":"ContainerDied","Data":"f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66"}
Jul 17 07:51:47.825586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:47.825213    2584 scope.go:117] "RemoveContainer" containerID="b500358b1ad7d3fdf69cb70f391d96b33c890c66a4dd4cdb1640088430953ce6"
Jul 17 07:51:47.825586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:47.825372    2584 scope.go:117] "RemoveContainer" containerID="f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66"
Jul 17 07:51:47.825714 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:47.825588    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-txn66_odh-ai-gateway-infra(b0fc520a-1678-4bfd-aac2-65dfd36e8cde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde"
Jul 17 07:51:48.831266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:48.831241    2584 scope.go:117] "RemoveContainer" containerID="f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66"
Jul 17 07:51:48.831682 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:51:48.831466    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-txn66_odh-ai-gateway-infra(b0fc520a-1678-4bfd-aac2-65dfd36e8cde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde"
Jul 17 07:51:49.862254 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:49.862218    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:50.061294 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:50.061248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:51:50.170143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:51:50.170062    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:00.728623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:00.728585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:01.681233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:01.681198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:04.288333 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:04.288286    2584 scope.go:117] "RemoveContainer" containerID="f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66"
Jul 17 07:52:04.897185 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:04.897157    2584 generic.go:356] "Generic (PLEG): container finished" podID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerID="6f5f6cab6039c81be371b677976fb6ad8a1cf1a21252ff9d6ede348476a33cf7" exitCode=6
Jul 17 07:52:04.897354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:04.897203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" event={"ID":"b0fc520a-1678-4bfd-aac2-65dfd36e8cde","Type":"ContainerDied","Data":"6f5f6cab6039c81be371b677976fb6ad8a1cf1a21252ff9d6ede348476a33cf7"}
Jul 17 07:52:04.897354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:04.897252    2584 scope.go:117] "RemoveContainer" containerID="f22b5168fa7e072eb615a931c841bc6e3fe59cf77552980aa87e4ed3b6675f66"
Jul 17 07:52:04.897620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:04.897599    2584 scope.go:117] "RemoveContainer" containerID="6f5f6cab6039c81be371b677976fb6ad8a1cf1a21252ff9d6ede348476a33cf7"
Jul 17 07:52:04.897833 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:04.897810    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-txn66_odh-ai-gateway-infra(b0fc520a-1678-4bfd-aac2-65dfd36e8cde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde"
Jul 17 07:52:05.923790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:05.923751    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:52:06.055729 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.055703    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:52:06.231870 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.231760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca\") pod \"b0fc520a-1678-4bfd-aac2-65dfd36e8cde\" (UID: \"b0fc520a-1678-4bfd-aac2-65dfd36e8cde\") "
Jul 17 07:52:06.232139 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.232116    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0fc520a-1678-4bfd-aac2-65dfd36e8cde" (UID: "b0fc520a-1678-4bfd-aac2-65dfd36e8cde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:52:06.332970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.332930    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fc520a-1678-4bfd-aac2-65dfd36e8cde-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:06.908282 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.908202    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"
Jul 17 07:52:06.908282 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.908217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66" event={"ID":"b0fc520a-1678-4bfd-aac2-65dfd36e8cde","Type":"ContainerDied","Data":"7e5c94ba50c2c0a2e5af84085e79aed5a953f6f29df516bba109bc320d4acfb5"}
Jul 17 07:52:06.908546 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.908286    2584 scope.go:117] "RemoveContainer" containerID="6f5f6cab6039c81be371b677976fb6ad8a1cf1a21252ff9d6ede348476a33cf7"
Jul 17 07:52:06.924891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.924861    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:52:06.928907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:06.928884    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-txn66"]
Jul 17 07:52:07.953682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:07.953649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:08.035775 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.035739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:08.183619 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.183574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:08.287466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.287419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:08.290116 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.290093    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" path="/var/lib/kubelet/pods/b0fc520a-1678-4bfd-aac2-65dfd36e8cde/volumes"
Jul 17 07:52:08.937446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937414    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:08.937718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937705    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937720    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937735    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937740    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937749    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937754    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937763 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937762    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:52:08.937948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937767    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1891e6-199a-4d00-ad33-063d97f90799" containerName="revoke-keys"
Jul 17 07:52:08.937948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937812    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937819    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:08.937948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.937828    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23148880-8b21-44aa-919e-c93682fb6ba6" containerName="revoke-keys"
Jul 17 07:52:08.942010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.941994    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:08.948446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.944993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:52:08.948446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.944993    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:52:08.949232 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:08.949188    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:09.056693 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.056658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk\" (UID: \"45af657b-468b-406b-9403-d03fad974334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:09.157420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.157377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk\" (UID: \"45af657b-468b-406b-9403-d03fad974334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:09.158016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.157996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk\" (UID: \"45af657b-468b-406b-9403-d03fad974334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:09.241821 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.241712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:09.258666 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.258626    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:09.357471 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.357438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:09.391030 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.391007    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:09.393477 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:52:09.393447    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45af657b_468b_406b_9403_d03fad974334.slice/crio-f1ff45ce0bcf0199fa42229a17ba1e1ac3c8b25cb8baf8cf1bd360ded64e0cb1 WatchSource:0}: Error finding container f1ff45ce0bcf0199fa42229a17ba1e1ac3c8b25cb8baf8cf1bd360ded64e0cb1: Status 404 returned error can't find the container with id f1ff45ce0bcf0199fa42229a17ba1e1ac3c8b25cb8baf8cf1bd360ded64e0cb1
Jul 17 07:52:09.458397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.458327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:09.922471 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.922387    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerStarted","Data":"0a129d6d4b71774164dbd60854360516d84edb3eea9df824468b12c633c34469"}
Jul 17 07:52:09.922471 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.922439    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerStarted","Data":"f1ff45ce0bcf0199fa42229a17ba1e1ac3c8b25cb8baf8cf1bd360ded64e0cb1"}
Jul 17 07:52:09.938181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:09.938126    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" podStartSLOduration=1.938108351 podStartE2EDuration="1.938108351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:52:09.937012252 +0000 UTC m=+2426.190435077" watchObservedRunningTime="2026-07-17 07:52:09.938108351 +0000 UTC m=+2426.191531164"
Jul 17 07:52:10.437948 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:10.437910    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:52:10.928586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:10.928552    2584 generic.go:356] "Generic (PLEG): container finished" podID="45af657b-468b-406b-9403-d03fad974334" containerID="0a129d6d4b71774164dbd60854360516d84edb3eea9df824468b12c633c34469" exitCode=6
Jul 17 07:52:10.928790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:10.928635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerDied","Data":"0a129d6d4b71774164dbd60854360516d84edb3eea9df824468b12c633c34469"}
Jul 17 07:52:10.928790 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:10.928769    2584 scope.go:117] "RemoveContainer" containerID="0a129d6d4b71774164dbd60854360516d84edb3eea9df824468b12c633c34469"
Jul 17 07:52:11.375892 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:11.375855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:11.935460 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:11.935427    2584 generic.go:356] "Generic (PLEG): container finished" podID="45af657b-468b-406b-9403-d03fad974334" containerID="dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a" exitCode=6
Jul 17 07:52:11.935907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:11.935484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerDied","Data":"dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a"}
Jul 17 07:52:11.935907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:11.935529    2584 scope.go:117] "RemoveContainer" containerID="0a129d6d4b71774164dbd60854360516d84edb3eea9df824468b12c633c34469"
Jul 17 07:52:11.935907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:11.935707    2584 scope.go:117] "RemoveContainer" containerID="dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a"
Jul 17 07:52:11.935907 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:11.935897    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk_odh-ai-gateway-infra(45af657b-468b-406b-9403-d03fad974334)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" podUID="45af657b-468b-406b-9403-d03fad974334"
Jul 17 07:52:12.941398 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:12.941368    2584 scope.go:117] "RemoveContainer" containerID="dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a"
Jul 17 07:52:12.941781 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:12.941544    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk_odh-ai-gateway-infra(45af657b-468b-406b-9403-d03fad974334)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" podUID="45af657b-468b-406b-9403-d03fad974334"
Jul 17 07:52:17.339931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:17.339896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:18.332452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:18.332417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:52:18.638881 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:18.638789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:21.702930 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.702894    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:21.703367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.703232    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:21.703367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.703247    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:21.703367 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.703301    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fc520a-1678-4bfd-aac2-65dfd36e8cde" containerName="revoke-keys"
Jul 17 07:52:21.706182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.706165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:21.712660 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.712618    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:21.757685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.757649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-djvzl\" (UID: \"de4975c4-fa65-4dd6-b85b-2d4a87738545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:21.858634 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.858594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-djvzl\" (UID: \"de4975c4-fa65-4dd6-b85b-2d4a87738545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:21.859215 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:21.859194    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-djvzl\" (UID: \"de4975c4-fa65-4dd6-b85b-2d4a87738545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:22.017628 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.017548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:22.146862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.146822    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:22.149734 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:52:22.149707    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4975c4_fa65_4dd6_b85b_2d4a87738545.slice/crio-4a70b0d07ba5c6605de6b0531068263f5681ec4e15ff21842ef5c637e5a10095 WatchSource:0}: Error finding container 4a70b0d07ba5c6605de6b0531068263f5681ec4e15ff21842ef5c637e5a10095: Status 404 returned error can't find the container with id 4a70b0d07ba5c6605de6b0531068263f5681ec4e15ff21842ef5c637e5a10095
Jul 17 07:52:22.985365 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.985265    2584 generic.go:356] "Generic (PLEG): container finished" podID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerID="a8baa9db7223d0393f09a1cd8c28c2edbeed41e85cc16d0ef1db12f7b563ccc4" exitCode=6
Jul 17 07:52:22.985718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.985363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" event={"ID":"de4975c4-fa65-4dd6-b85b-2d4a87738545","Type":"ContainerDied","Data":"a8baa9db7223d0393f09a1cd8c28c2edbeed41e85cc16d0ef1db12f7b563ccc4"}
Jul 17 07:52:22.985718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.985413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" event={"ID":"de4975c4-fa65-4dd6-b85b-2d4a87738545","Type":"ContainerStarted","Data":"4a70b0d07ba5c6605de6b0531068263f5681ec4e15ff21842ef5c637e5a10095"}
Jul 17 07:52:22.985718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:22.985573    2584 scope.go:117] "RemoveContainer" containerID="a8baa9db7223d0393f09a1cd8c28c2edbeed41e85cc16d0ef1db12f7b563ccc4"
Jul 17 07:52:23.990958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:23.990925    2584 generic.go:356] "Generic (PLEG): container finished" podID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerID="2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887" exitCode=6
Jul 17 07:52:23.991439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:23.991015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" event={"ID":"de4975c4-fa65-4dd6-b85b-2d4a87738545","Type":"ContainerDied","Data":"2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887"}
Jul 17 07:52:23.991439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:23.991087    2584 scope.go:117] "RemoveContainer" containerID="a8baa9db7223d0393f09a1cd8c28c2edbeed41e85cc16d0ef1db12f7b563ccc4"
Jul 17 07:52:23.991439 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:23.991207    2584 scope.go:117] "RemoveContainer" containerID="2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887"
Jul 17 07:52:23.991439 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:23.991415    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-djvzl_odh-ai-gateway-infra(de4975c4-fa65-4dd6-b85b-2d4a87738545)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545"
Jul 17 07:52:24.118926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.118892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:24.183934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.183901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:24.398721 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.398686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:24.761427 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.760464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:24.861907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.861868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:24.997435 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:24.997404    2584 scope.go:117] "RemoveContainer" containerID="2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887"
Jul 17 07:52:24.997831 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:24.997579    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-djvzl_odh-ai-gateway-infra(de4975c4-fa65-4dd6-b85b-2d4a87738545)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545"
Jul 17 07:52:27.285138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.285108    2584 scope.go:117] "RemoveContainer" containerID="dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a"
Jul 17 07:52:27.425522 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.425484    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:52:27.823425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.823379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:27.919088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.919044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:27.919436 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.919402    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:27.922580 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.922551    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:27.925084 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.925061    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:52:27.925197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.925143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:52:27.925197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.925157    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f88ba1e3-serving-cert\""
Jul 17 07:52:27.931076 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.931048    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:27.982578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:27.982539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:28.011875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.011849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zwv\" (UniqueName: \"kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.012017 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.011908    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.012017 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.011968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.012768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.012749    2584 generic.go:356] "Generic (PLEG): container finished" podID="45af657b-468b-406b-9403-d03fad974334" containerID="554722df0fafa04b4461da161c18707e294cde2e19ceb0ff09845061b3f22d2a" exitCode=6
Jul 17 07:52:28.012876 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.012775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerDied","Data":"554722df0fafa04b4461da161c18707e294cde2e19ceb0ff09845061b3f22d2a"}
Jul 17 07:52:28.012876 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.012838    2584 scope.go:117] "RemoveContainer" containerID="dba6c555c0a0ada08fffe2e05a68fca063464445a7146c91d593fa0800d9622a"
Jul 17 07:52:28.013163 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.013146    2584 scope.go:117] "RemoveContainer" containerID="554722df0fafa04b4461da161c18707e294cde2e19ceb0ff09845061b3f22d2a"
Jul 17 07:52:28.013364 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:28.013321    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk_odh-ai-gateway-infra(45af657b-468b-406b-9403-d03fad974334)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" podUID="45af657b-468b-406b-9403-d03fad974334"
Jul 17 07:52:28.112950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.112876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.112950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.112952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zwv\" (UniqueName: \"kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.113169 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.112972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.113169 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:28.113067    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-serving-cert: secret "maas-api-e2e-mt-f88ba1e3-serving-cert" not found
Jul 17 07:52:28.113169 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:28.113124    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls podName:83c777af-003e-4bbd-8a84-7a2a987df495 nodeName:}" failed. No retries permitted until 2026-07-17 07:52:28.613105335 +0000 UTC m=+2444.866528128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls") pod "maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" (UID: "83c777af-003e-4bbd-8a84-7a2a987df495") : secret "maas-api-e2e-mt-f88ba1e3-serving-cert" not found
Jul 17 07:52:28.113498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.113480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.121794 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.121772    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zwv\" (UniqueName: \"kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.165175 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.165146    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:28.601799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.601768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:28.616165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.616138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.618544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.618518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") pod \"maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.702517 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.702487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:28.834514 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.834483    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:28.961209 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.961186    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:28.963670 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:52:28.963636    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c777af_003e_4bbd_8a84_7a2a987df495.slice/crio-6327dc03dde7240137b2ebc879ccf7bf0e7c8b4cc307d2cd670d2dcdfe064c56 WatchSource:0}: Error finding container 6327dc03dde7240137b2ebc879ccf7bf0e7c8b4cc307d2cd670d2dcdfe064c56: Status 404 returned error can't find the container with id 6327dc03dde7240137b2ebc879ccf7bf0e7c8b4cc307d2cd670d2dcdfe064c56
Jul 17 07:52:28.964964 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:28.964944    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:52:29.019855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.019824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" event={"ID":"83c777af-003e-4bbd-8a84-7a2a987df495","Type":"ContainerStarted","Data":"6327dc03dde7240137b2ebc879ccf7bf0e7c8b4cc307d2cd670d2dcdfe064c56"}
Jul 17 07:52:29.039040 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.039013    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:29.163987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.163965    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:29.221039 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.221008    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca\") pod \"45af657b-468b-406b-9403-d03fad974334\" (UID: \"45af657b-468b-406b-9403-d03fad974334\") "
Jul 17 07:52:29.221356 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.221315    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45af657b-468b-406b-9403-d03fad974334" (UID: "45af657b-468b-406b-9403-d03fad974334"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:52:29.322429 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:29.322395    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45af657b-468b-406b-9403-d03fad974334-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:30.025182 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.025142    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" event={"ID":"83c777af-003e-4bbd-8a84-7a2a987df495","Type":"ContainerStarted","Data":"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"}
Jul 17 07:52:30.025676 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.025252    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:30.026424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.026396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk" event={"ID":"45af657b-468b-406b-9403-d03fad974334","Type":"ContainerDied","Data":"f1ff45ce0bcf0199fa42229a17ba1e1ac3c8b25cb8baf8cf1bd360ded64e0cb1"}
Jul 17 07:52:30.026424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.026421    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"
Jul 17 07:52:30.026550 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.026462    2584 scope.go:117] "RemoveContainer" containerID="554722df0fafa04b4461da161c18707e294cde2e19ceb0ff09845061b3f22d2a"
Jul 17 07:52:30.046120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.046077    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" podStartSLOduration=2.681004943 podStartE2EDuration="3.046063344s" totalImagesPullingTime="365.058401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:52:30.043765673 +0000 UTC m=+2446.297188488" watchObservedRunningTime="2026-07-17 07:52:30.046063344 +0000 UTC m=+2446.299486156"
Jul 17 07:52:30.061886 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.061861    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:30.065670 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.065644    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lvbkk"]
Jul 17 07:52:30.289692 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:30.289657    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45af657b-468b-406b-9403-d03fad974334" path="/var/lib/kubelet/pods/45af657b-468b-406b-9403-d03fad974334/volumes"
Jul 17 07:52:32.401766 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:32.401727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:33.149811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:33.149761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:33.245727 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:33.245695    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:33.644180 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:33.644143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:36.038549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:36.038517    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:37.730129 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:37.730088    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:52:39.284796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:39.284772    2584 scope.go:117] "RemoveContainer" containerID="2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887"
Jul 17 07:52:39.807984 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:39.807908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:39.870914 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:39.870880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:40.073213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:40.073128    2584 generic.go:356] "Generic (PLEG): container finished" podID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerID="a46f5b906536699e856400a95b5b7c6aeafb5962340e25ae919be171dc8116aa" exitCode=6
Jul 17 07:52:40.073213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:40.073194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" event={"ID":"de4975c4-fa65-4dd6-b85b-2d4a87738545","Type":"ContainerDied","Data":"a46f5b906536699e856400a95b5b7c6aeafb5962340e25ae919be171dc8116aa"}
Jul 17 07:52:40.073420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:40.073258    2584 scope.go:117] "RemoveContainer" containerID="2bfadca157b5945e98a8902520b2ab6cf79f148235b62ee4dcb9061628064887"
Jul 17 07:52:40.073581 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:40.073569    2584 scope.go:117] "RemoveContainer" containerID="a46f5b906536699e856400a95b5b7c6aeafb5962340e25ae919be171dc8116aa"
Jul 17 07:52:40.073822 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:40.073802    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-djvzl_odh-ai-gateway-infra(de4975c4-fa65-4dd6-b85b-2d4a87738545)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545"
Jul 17 07:52:41.101162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:41.101125    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:41.236378 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:41.236351    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:41.319191 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:41.319151    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca\") pod \"de4975c4-fa65-4dd6-b85b-2d4a87738545\" (UID: \"de4975c4-fa65-4dd6-b85b-2d4a87738545\") "
Jul 17 07:52:41.319575 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:41.319552    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de4975c4-fa65-4dd6-b85b-2d4a87738545" (UID: "de4975c4-fa65-4dd6-b85b-2d4a87738545"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:52:41.420235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:41.420158    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4975c4-fa65-4dd6-b85b-2d4a87738545-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:42.084642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.084609    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"
Jul 17 07:52:42.084642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.084629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl" event={"ID":"de4975c4-fa65-4dd6-b85b-2d4a87738545","Type":"ContainerDied","Data":"4a70b0d07ba5c6605de6b0531068263f5681ec4e15ff21842ef5c637e5a10095"}
Jul 17 07:52:42.084855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.084727    2584 scope.go:117] "RemoveContainer" containerID="a46f5b906536699e856400a95b5b7c6aeafb5962340e25ae919be171dc8116aa"
Jul 17 07:52:42.106037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.106004    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:42.109995 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.109969    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-djvzl"]
Jul 17 07:52:42.290985 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:42.290953    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" path="/var/lib/kubelet/pods/de4975c4-fa65-4dd6-b85b-2d4a87738545/volumes"
Jul 17 07:52:45.526203 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526166    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526502    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526511    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526517    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526527    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526533    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526542    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526548    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526605    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526613    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526619    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:52:45.526645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.526627    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:52:45.530795 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.530776    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:52:45.533569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.533548    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:52:45.538468 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.538447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:52:45.656631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.656591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9\" (UID: \"f86597ab-3ee9-429b-aabe-dedc459c7e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:52:45.757613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.757577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9\" (UID: \"f86597ab-3ee9-429b-aabe-dedc459c7e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:52:45.758207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.758187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9\" (UID: \"f86597ab-3ee9-429b-aabe-dedc459c7e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:52:45.842351 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.842316    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:52:45.974017 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:45.973992    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:52:45.976036 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:52:45.976006    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86597ab_3ee9_429b_aabe_dedc459c7e4c.slice/crio-441a6865e12bc8648cbd63a48e9e0f21b62df101b547bb9dc2e1b6024004b462 WatchSource:0}: Error finding container 441a6865e12bc8648cbd63a48e9e0f21b62df101b547bb9dc2e1b6024004b462: Status 404 returned error can't find the container with id 441a6865e12bc8648cbd63a48e9e0f21b62df101b547bb9dc2e1b6024004b462
Jul 17 07:52:46.102585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:46.102493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" event={"ID":"f86597ab-3ee9-429b-aabe-dedc459c7e4c","Type":"ContainerStarted","Data":"441a6865e12bc8648cbd63a48e9e0f21b62df101b547bb9dc2e1b6024004b462"}
Jul 17 07:52:47.110066 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:47.110031    2584 generic.go:356] "Generic (PLEG): container finished" podID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerID="3a7b7336e8e3c985175a32f3f5fdba4425df8c76f7d708d21eac5556afd17e59" exitCode=6
Jul 17 07:52:47.110531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:47.110121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" event={"ID":"f86597ab-3ee9-429b-aabe-dedc459c7e4c","Type":"ContainerDied","Data":"3a7b7336e8e3c985175a32f3f5fdba4425df8c76f7d708d21eac5556afd17e59"}
Jul 17 07:52:47.110531 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:47.110309    2584 scope.go:117] "RemoveContainer" containerID="3a7b7336e8e3c985175a32f3f5fdba4425df8c76f7d708d21eac5556afd17e59"
Jul 17 07:52:48.116293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:48.116256    2584 generic.go:356] "Generic (PLEG): container finished" podID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerID="4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9" exitCode=6
Jul 17 07:52:48.116855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:48.116333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" event={"ID":"f86597ab-3ee9-429b-aabe-dedc459c7e4c","Type":"ContainerDied","Data":"4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9"}
Jul 17 07:52:48.116855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:48.116402    2584 scope.go:117] "RemoveContainer" containerID="3a7b7336e8e3c985175a32f3f5fdba4425df8c76f7d708d21eac5556afd17e59"
Jul 17 07:52:48.116855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:48.116662    2584 scope.go:117] "RemoveContainer" containerID="4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9"
Jul 17 07:52:48.117041 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:48.116886    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9_odh-ai-gateway-infra(f86597ab-3ee9-429b-aabe-dedc459c7e4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c"
Jul 17 07:52:49.121632 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:49.121605    2584 scope.go:117] "RemoveContainer" containerID="4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9"
Jul 17 07:52:49.121988 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:49.121778    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9_odh-ai-gateway-infra(f86597ab-3ee9-429b-aabe-dedc459c7e4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c"
Jul 17 07:52:50.665049 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:50.665019    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:50.665475 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:50.665199    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" podUID="83c777af-003e-4bbd-8a84-7a2a987df495" containerName="maas-api" containerID="cri-o://bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a" gracePeriod=30
Jul 17 07:52:50.688845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:50.688819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:50.905865 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:50.905837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:51.001596 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.001522    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca\") pod \"83c777af-003e-4bbd-8a84-7a2a987df495\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") "
Jul 17 07:52:51.001734 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.001628    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zwv\" (UniqueName: \"kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv\") pod \"83c777af-003e-4bbd-8a84-7a2a987df495\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") "
Jul 17 07:52:51.001734 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.001656    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") pod \"83c777af-003e-4bbd-8a84-7a2a987df495\" (UID: \"83c777af-003e-4bbd-8a84-7a2a987df495\") "
Jul 17 07:52:51.001868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.001843    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83c777af-003e-4bbd-8a84-7a2a987df495" (UID: "83c777af-003e-4bbd-8a84-7a2a987df495"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:52:51.003678 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.003649    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv" (OuterVolumeSpecName: "kube-api-access-z6zwv") pod "83c777af-003e-4bbd-8a84-7a2a987df495" (UID: "83c777af-003e-4bbd-8a84-7a2a987df495"). InnerVolumeSpecName "kube-api-access-z6zwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:52:51.003795 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.003781    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83c777af-003e-4bbd-8a84-7a2a987df495" (UID: "83c777af-003e-4bbd-8a84-7a2a987df495"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:52:51.103010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.102978    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zwv\" (UniqueName: \"kubernetes.io/projected/83c777af-003e-4bbd-8a84-7a2a987df495-kube-api-access-z6zwv\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:51.103010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.103004    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c777af-003e-4bbd-8a84-7a2a987df495-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:51.103010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.103013    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c777af-003e-4bbd-8a84-7a2a987df495-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:52:51.131564 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.131530    2584 generic.go:356] "Generic (PLEG): container finished" podID="83c777af-003e-4bbd-8a84-7a2a987df495" containerID="bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a" exitCode=0
Jul 17 07:52:51.131707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.131575    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"
Jul 17 07:52:51.131707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.131618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" event={"ID":"83c777af-003e-4bbd-8a84-7a2a987df495","Type":"ContainerDied","Data":"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"}
Jul 17 07:52:51.131707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.131672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5" event={"ID":"83c777af-003e-4bbd-8a84-7a2a987df495","Type":"ContainerDied","Data":"6327dc03dde7240137b2ebc879ccf7bf0e7c8b4cc307d2cd670d2dcdfe064c56"}
Jul 17 07:52:51.131707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.131695    2584 scope.go:117] "RemoveContainer" containerID="bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"
Jul 17 07:52:51.141022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.141005    2584 scope.go:117] "RemoveContainer" containerID="bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"
Jul 17 07:52:51.141282 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:52:51.141264    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a\": container with ID starting with bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a not found: ID does not exist" containerID="bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"
Jul 17 07:52:51.141330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.141290    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a"} err="failed to get container status \"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a\": rpc error: code = NotFound desc = could not find container \"bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a\": container with ID starting with bb276019046e1b1ff51a553b88c0d56c4e66aaac8809753c1688418794c1ec9a not found: ID does not exist"
Jul 17 07:52:51.152375 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.152323    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:51.156278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:51.156258    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f88ba1e3-7b56b4cd7b-htnd5"]
Jul 17 07:52:52.289861 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:52.289828    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c777af-003e-4bbd-8a84-7a2a987df495" path="/var/lib/kubelet/pods/83c777af-003e-4bbd-8a84-7a2a987df495/volumes"
Jul 17 07:52:53.625349 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:53.625300    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:52:55.304939 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:55.304905    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:52:59.284942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:52:59.284912    2584 scope.go:117] "RemoveContainer" containerID="4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9"
Jul 17 07:53:00.170510 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:00.170431    2584 generic.go:356] "Generic (PLEG): container finished" podID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerID="115afe03ced64b1fded59550704a0ce1101eb764cf7a3ddcf0fc90de1c10a3bb" exitCode=6
Jul 17 07:53:00.170510 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:00.170497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" event={"ID":"f86597ab-3ee9-429b-aabe-dedc459c7e4c","Type":"ContainerDied","Data":"115afe03ced64b1fded59550704a0ce1101eb764cf7a3ddcf0fc90de1c10a3bb"}
Jul 17 07:53:00.170702 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:00.170561    2584 scope.go:117] "RemoveContainer" containerID="4ee3b61fbc5c3c4a5b323096ff3d3c60147e2e9052151f193b00e276a86819a9"
Jul 17 07:53:00.170842 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:00.170827    2584 scope.go:117] "RemoveContainer" containerID="115afe03ced64b1fded59550704a0ce1101eb764cf7a3ddcf0fc90de1c10a3bb"
Jul 17 07:53:00.171055 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:00.171037    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9_odh-ai-gateway-infra(f86597ab-3ee9-429b-aabe-dedc459c7e4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c"
Jul 17 07:53:01.195889 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.195853    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:53:01.316753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.316727    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:53:01.484866 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.484771    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca\") pod \"f86597ab-3ee9-429b-aabe-dedc459c7e4c\" (UID: \"f86597ab-3ee9-429b-aabe-dedc459c7e4c\") "
Jul 17 07:53:01.485179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.485159    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f86597ab-3ee9-429b-aabe-dedc459c7e4c" (UID: "f86597ab-3ee9-429b-aabe-dedc459c7e4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:53:01.586220 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.586190    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86597ab-3ee9-429b-aabe-dedc459c7e4c-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:53:01.867503 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.867468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:01.939068 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:01.939032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:02.180987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.180897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9" event={"ID":"f86597ab-3ee9-429b-aabe-dedc459c7e4c","Type":"ContainerDied","Data":"441a6865e12bc8648cbd63a48e9e0f21b62df101b547bb9dc2e1b6024004b462"}
Jul 17 07:53:02.180987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.180929    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"
Jul 17 07:53:02.180987 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.180976    2584 scope.go:117] "RemoveContainer" containerID="115afe03ced64b1fded59550704a0ce1101eb764cf7a3ddcf0fc90de1c10a3bb"
Jul 17 07:53:02.203664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.203636    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:53:02.209279 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.209257    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-l4qn9"]
Jul 17 07:53:02.289984 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:02.289951    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" path="/var/lib/kubelet/pods/f86597ab-3ee9-429b-aabe-dedc459c7e4c/volumes"
Jul 17 07:53:03.560288 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560250    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560560    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560572    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560580    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c777af-003e-4bbd-8a84-7a2a987df495" containerName="maas-api"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560585    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c777af-003e-4bbd-8a84-7a2a987df495" containerName="maas-api"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560592    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560598    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560608    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560613    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560622    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560627    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560685    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c777af-003e-4bbd-8a84-7a2a987df495" containerName="maas-api"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560692    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45af657b-468b-406b-9403-d03fad974334" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560699    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4975c4-fa65-4dd6-b85b-2d4a87738545" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560705    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560711    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.560769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.560717    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:03.564682 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.564661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:03.568417 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.568395    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:53:03.568638 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.568621    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:53:03.581944 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.581912    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:03.704086 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.704042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt\" (UID: \"166e61ce-751d-4a37-864b-435f3ace910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:03.805598 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.805556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt\" (UID: \"166e61ce-751d-4a37-864b-435f3ace910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:03.806230 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.806208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt\" (UID: \"166e61ce-751d-4a37-864b-435f3ace910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:03.874688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.874605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:03.999002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:03.998978    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:04.001636 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:53:04.001605    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166e61ce_751d_4a37_864b_435f3ace910f.slice/crio-1c9a2e1cc565064b3ae8f22e1a48591c1d235233a088dbf52b0d06d8d92ecfcf WatchSource:0}: Error finding container 1c9a2e1cc565064b3ae8f22e1a48591c1d235233a088dbf52b0d06d8d92ecfcf: Status 404 returned error can't find the container with id 1c9a2e1cc565064b3ae8f22e1a48591c1d235233a088dbf52b0d06d8d92ecfcf
Jul 17 07:53:04.197990 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:04.197896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" event={"ID":"166e61ce-751d-4a37-864b-435f3ace910f","Type":"ContainerStarted","Data":"1c9a2e1cc565064b3ae8f22e1a48591c1d235233a088dbf52b0d06d8d92ecfcf"}
Jul 17 07:53:05.203503 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:05.203470    2584 generic.go:356] "Generic (PLEG): container finished" podID="166e61ce-751d-4a37-864b-435f3ace910f" containerID="8a79f3f02414ef9fe8ee64846506429a763669d340df94698faa1715d58ec699" exitCode=6
Jul 17 07:53:05.203881 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:05.203556    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" event={"ID":"166e61ce-751d-4a37-864b-435f3ace910f","Type":"ContainerDied","Data":"8a79f3f02414ef9fe8ee64846506429a763669d340df94698faa1715d58ec699"}
Jul 17 07:53:05.203881 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:05.203699    2584 scope.go:117] "RemoveContainer" containerID="8a79f3f02414ef9fe8ee64846506429a763669d340df94698faa1715d58ec699"
Jul 17 07:53:06.209708 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.209618    2584 generic.go:356] "Generic (PLEG): container finished" podID="166e61ce-751d-4a37-864b-435f3ace910f" containerID="c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5" exitCode=6
Jul 17 07:53:06.210126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.209702    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" event={"ID":"166e61ce-751d-4a37-864b-435f3ace910f","Type":"ContainerDied","Data":"c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5"}
Jul 17 07:53:06.210126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.209762    2584 scope.go:117] "RemoveContainer" containerID="8a79f3f02414ef9fe8ee64846506429a763669d340df94698faa1715d58ec699"
Jul 17 07:53:06.210126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.209911    2584 scope.go:117] "RemoveContainer" containerID="c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5"
Jul 17 07:53:06.210255 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:06.210150    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt_odh-ai-gateway-infra(166e61ce-751d-4a37-864b-435f3ace910f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" podUID="166e61ce-751d-4a37-864b-435f3ace910f"
Jul 17 07:53:06.279639 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.279605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:06.529255 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.529163    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:06.686139 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.686104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:06.900211 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:06.900152    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:07.216326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:07.216247    2584 scope.go:117] "RemoveContainer" containerID="c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5"
Jul 17 07:53:07.216695 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:07.216448    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt_odh-ai-gateway-infra(166e61ce-751d-4a37-864b-435f3ace910f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" podUID="166e61ce-751d-4a37-864b-435f3ace910f"
Jul 17 07:53:08.997294 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:08.997260    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:08.997685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:08.997580    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:08.997685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:08.997593    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86597ab-3ee9-429b-aabe-dedc459c7e4c" containerName="revoke-keys"
Jul 17 07:53:09.001701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.001684    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:09.011276 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.011253    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:09.151430 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.151397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr\" (UID: \"3234c5b9-c48e-4f96-9c0b-652f63174c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:09.252003 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.251928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr\" (UID: \"3234c5b9-c48e-4f96-9c0b-652f63174c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:09.252520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.252499    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr\" (UID: \"3234c5b9-c48e-4f96-9c0b-652f63174c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:09.311601 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.311549    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:09.445358 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:09.445316    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:09.447006 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:53:09.446976    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3234c5b9_c48e_4f96_9c0b_652f63174c87.slice/crio-a6c29ae71810a348dbd4c56b81458172b68292d5f5d8c34e2e4f2bfde5e07c9f WatchSource:0}: Error finding container a6c29ae71810a348dbd4c56b81458172b68292d5f5d8c34e2e4f2bfde5e07c9f: Status 404 returned error can't find the container with id a6c29ae71810a348dbd4c56b81458172b68292d5f5d8c34e2e4f2bfde5e07c9f
Jul 17 07:53:10.228865 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:10.228833    2584 generic.go:356] "Generic (PLEG): container finished" podID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerID="6fb6df1b0cc6fea589f45d2ac677e02d690404df11c5498432b300ef22279aa7" exitCode=6
Jul 17 07:53:10.229228 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:10.228898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" event={"ID":"3234c5b9-c48e-4f96-9c0b-652f63174c87","Type":"ContainerDied","Data":"6fb6df1b0cc6fea589f45d2ac677e02d690404df11c5498432b300ef22279aa7"}
Jul 17 07:53:10.229228 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:10.228954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" event={"ID":"3234c5b9-c48e-4f96-9c0b-652f63174c87","Type":"ContainerStarted","Data":"a6c29ae71810a348dbd4c56b81458172b68292d5f5d8c34e2e4f2bfde5e07c9f"}
Jul 17 07:53:10.229228 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:10.229089    2584 scope.go:117] "RemoveContainer" containerID="6fb6df1b0cc6fea589f45d2ac677e02d690404df11c5498432b300ef22279aa7"
Jul 17 07:53:11.235403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:11.235295    2584 generic.go:356] "Generic (PLEG): container finished" podID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerID="5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013" exitCode=6
Jul 17 07:53:11.235403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:11.235380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" event={"ID":"3234c5b9-c48e-4f96-9c0b-652f63174c87","Type":"ContainerDied","Data":"5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013"}
Jul 17 07:53:11.235906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:11.235443    2584 scope.go:117] "RemoveContainer" containerID="6fb6df1b0cc6fea589f45d2ac677e02d690404df11c5498432b300ef22279aa7"
Jul 17 07:53:11.235906 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:11.235736    2584 scope.go:117] "RemoveContainer" containerID="5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013"
Jul 17 07:53:11.236013 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:11.235978    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr_odh-ai-gateway-infra(3234c5b9-c48e-4f96-9c0b-652f63174c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87"
Jul 17 07:53:12.241357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:12.241322    2584 scope.go:117] "RemoveContainer" containerID="5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013"
Jul 17 07:53:12.241721 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:12.241517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr_odh-ai-gateway-infra(3234c5b9-c48e-4f96-9c0b-652f63174c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87"
Jul 17 07:53:13.371591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:13.371536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:13.470754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:13.470717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:13.636912 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:13.636834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:13.799397 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:13.799363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:19.877542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:19.877507    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:19.945415 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:19.945380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:20.284994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:20.284964    2584 scope.go:117] "RemoveContainer" containerID="c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5"
Jul 17 07:53:21.280079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:21.279998    2584 generic.go:356] "Generic (PLEG): container finished" podID="166e61ce-751d-4a37-864b-435f3ace910f" containerID="8669987f092c4d569ec2f3d77933a78ff4ee3316fa54a787f4faf3fe0139127f" exitCode=6
Jul 17 07:53:21.280505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:21.280073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" event={"ID":"166e61ce-751d-4a37-864b-435f3ace910f","Type":"ContainerDied","Data":"8669987f092c4d569ec2f3d77933a78ff4ee3316fa54a787f4faf3fe0139127f"}
Jul 17 07:53:21.280505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:21.280135    2584 scope.go:117] "RemoveContainer" containerID="c15eeb37792475c9b8c6da9c1eb779c7a636d252e4609dbc76c231ee81da2ab5"
Jul 17 07:53:21.280505 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:21.280446    2584 scope.go:117] "RemoveContainer" containerID="8669987f092c4d569ec2f3d77933a78ff4ee3316fa54a787f4faf3fe0139127f"
Jul 17 07:53:21.280651 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:21.280633    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt_odh-ai-gateway-infra(166e61ce-751d-4a37-864b-435f3ace910f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" podUID="166e61ce-751d-4a37-864b-435f3ace910f"
Jul 17 07:53:22.310458 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:22.310430    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:22.448069 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:22.448038    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:22.454241 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:22.454221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca\") pod \"166e61ce-751d-4a37-864b-435f3ace910f\" (UID: \"166e61ce-751d-4a37-864b-435f3ace910f\") "
Jul 17 07:53:22.454552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:22.454523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "166e61ce-751d-4a37-864b-435f3ace910f" (UID: "166e61ce-751d-4a37-864b-435f3ace910f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:53:22.555587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:22.555546    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/166e61ce-751d-4a37-864b-435f3ace910f-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:53:23.291804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:23.291761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt" event={"ID":"166e61ce-751d-4a37-864b-435f3ace910f","Type":"ContainerDied","Data":"1c9a2e1cc565064b3ae8f22e1a48591c1d235233a088dbf52b0d06d8d92ecfcf"}
Jul 17 07:53:23.291804 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:23.291799    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"
Jul 17 07:53:23.292078 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:23.291834    2584 scope.go:117] "RemoveContainer" containerID="8669987f092c4d569ec2f3d77933a78ff4ee3316fa54a787f4faf3fe0139127f"
Jul 17 07:53:23.316753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:23.316721    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:23.322512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:23.322487    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-kvgrt"]
Jul 17 07:53:24.290033 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:24.289995    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166e61ce-751d-4a37-864b-435f3ace910f" path="/var/lib/kubelet/pods/166e61ce-751d-4a37-864b-435f3ace910f/volumes"
Jul 17 07:53:26.284949 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:26.284913    2584 scope.go:117] "RemoveContainer" containerID="5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013"
Jul 17 07:53:26.967402 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:26.967363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:27.292817 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.292789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:27.310934 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.310903    2584 generic.go:356] "Generic (PLEG): container finished" podID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerID="42433f13062a839472b52dc90c422d80dabb3ab0cfc0ad0fd5d885dd46cab339" exitCode=6
Jul 17 07:53:27.311090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.310971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" event={"ID":"3234c5b9-c48e-4f96-9c0b-652f63174c87","Type":"ContainerDied","Data":"42433f13062a839472b52dc90c422d80dabb3ab0cfc0ad0fd5d885dd46cab339"}
Jul 17 07:53:27.311090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.311031    2584 scope.go:117] "RemoveContainer" containerID="5cff9f97d94f283a290173e818c0060d4921c0d79cb6127c039baa22dba1c013"
Jul 17 07:53:27.311383 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.311363    2584 scope.go:117] "RemoveContainer" containerID="42433f13062a839472b52dc90c422d80dabb3ab0cfc0ad0fd5d885dd46cab339"
Jul 17 07:53:27.311638 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:27.311617    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr_odh-ai-gateway-infra(3234c5b9-c48e-4f96-9c0b-652f63174c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87"
Jul 17 07:53:27.847356 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:27.847301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:28.382542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:28.382505    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:28.523774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:28.523748    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:28.597562 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:28.597528    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca\") pod \"3234c5b9-c48e-4f96-9c0b-652f63174c87\" (UID: \"3234c5b9-c48e-4f96-9c0b-652f63174c87\") "
Jul 17 07:53:28.597918 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:28.597898    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3234c5b9-c48e-4f96-9c0b-652f63174c87" (UID: "3234c5b9-c48e-4f96-9c0b-652f63174c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:53:28.698178 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:28.698087    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3234c5b9-c48e-4f96-9c0b-652f63174c87-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:53:29.322151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:29.322103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr" event={"ID":"3234c5b9-c48e-4f96-9c0b-652f63174c87","Type":"ContainerDied","Data":"a6c29ae71810a348dbd4c56b81458172b68292d5f5d8c34e2e4f2bfde5e07c9f"}
Jul 17 07:53:29.322151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:29.322126    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"
Jul 17 07:53:29.322455 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:29.322175    2584 scope.go:117] "RemoveContainer" containerID="42433f13062a839472b52dc90c422d80dabb3ab0cfc0ad0fd5d885dd46cab339"
Jul 17 07:53:29.347247 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:29.347220    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:29.352099 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:29.352075    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-xcvtr"]
Jul 17 07:53:30.289547 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.289517    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" path="/var/lib/kubelet/pods/3234c5b9-c48e-4f96-9c0b-652f63174c87/volumes"
Jul 17 07:53:30.681307 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.681234    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:30.774106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774074    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:53:30.774405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774392    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774405 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774406    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774416    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774422    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774429    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774435    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774453    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774458    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774507    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774515    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:30.774792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774522    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.774792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.774529    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:30.778724 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.778701    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.780423 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.780396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:30.781259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.781240    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:53:30.781415 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.781396    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:53:30.782250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.782233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:53:30.782362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.782234    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e8d5b32b-serving-cert\""
Jul 17 07:53:30.789740 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.789715    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:53:30.817209 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.817184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.817386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.817240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.817386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.817299    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf28\" (UniqueName: \"kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.844787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.844763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:30.917989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.917955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.918165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.918002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf28\" (UniqueName: \"kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.918165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.918075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.918165 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:30.918089    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-serving-cert: secret "maas-api-e2e-mt-e8d5b32b-serving-cert" not found
Jul 17 07:53:30.918165 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:30.918158    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls podName:af583d5b-5348-4135-9425-e7889c3021bf nodeName:}" failed. No retries permitted until 2026-07-17 07:53:31.418141854 +0000 UTC m=+2507.671564655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls") pod "maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" (UID: "af583d5b-5348-4135-9425-e7889c3021bf") : secret "maas-api-e2e-mt-e8d5b32b-serving-cert" not found
Jul 17 07:53:30.918857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.918837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:30.927293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:30.927272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf28\" (UniqueName: \"kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:31.167657 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.167591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:31.421215 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.421131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:31.423641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.423624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") pod \"maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:31.451751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.451696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:31.553563 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.553521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:31.689709 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.689618    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:31.821128 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:31.821102    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:53:31.823895 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:53:31.823867    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf583d5b_5348_4135_9425_e7889c3021bf.slice/crio-914fac06124bf01ed30abd56c587b7dd2abb72ccdb476f8b327eeb65a113c410 WatchSource:0}: Error finding container 914fac06124bf01ed30abd56c587b7dd2abb72ccdb476f8b327eeb65a113c410: Status 404 returned error can't find the container with id 914fac06124bf01ed30abd56c587b7dd2abb72ccdb476f8b327eeb65a113c410
Jul 17 07:53:32.337120 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:32.337085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" event={"ID":"af583d5b-5348-4135-9425-e7889c3021bf","Type":"ContainerStarted","Data":"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"}
Jul 17 07:53:32.337291 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:32.337146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" event={"ID":"af583d5b-5348-4135-9425-e7889c3021bf","Type":"ContainerStarted","Data":"914fac06124bf01ed30abd56c587b7dd2abb72ccdb476f8b327eeb65a113c410"}
Jul 17 07:53:32.337291 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:32.337174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:32.355595 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:32.355549    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" podStartSLOduration=1.949945476 podStartE2EDuration="2.355533463s" totalImagesPullingTime="405.587987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:53:32.354870415 +0000 UTC m=+2508.608293230" watchObservedRunningTime="2026-07-17 07:53:32.355533463 +0000 UTC m=+2508.608956275"
Jul 17 07:53:35.236890 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.236852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:35.623090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623024    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:35.623437 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623400    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:35.623437 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623420    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:35.623661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623454    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:35.623661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623476    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:35.623661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623549    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3234c5b9-c48e-4f96-9c0b-652f63174c87" containerName="revoke-keys"
Jul 17 07:53:35.623661 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.623565    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="166e61ce-751d-4a37-864b-435f3ace910f" containerName="revoke-keys"
Jul 17 07:53:35.626659 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.626638    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:35.629877 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.629860    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:53:35.654305 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.654275    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:35.655751 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.655712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt\" (UID: \"4a768bdc-773d-4892-b931-ae557b156189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:35.756893 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.756847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt\" (UID: \"4a768bdc-773d-4892-b931-ae557b156189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:35.757496 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.757475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt\" (UID: \"4a768bdc-773d-4892-b931-ae557b156189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:35.936409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:35.936293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:36.101357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:36.101320    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:36.102982 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:53:36.102948    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a768bdc_773d_4892_b931_ae557b156189.slice/crio-fe39239557acd9814c326689fb0eef2f06509bf9ab28b82449894606a5d4dc60 WatchSource:0}: Error finding container fe39239557acd9814c326689fb0eef2f06509bf9ab28b82449894606a5d4dc60: Status 404 returned error can't find the container with id fe39239557acd9814c326689fb0eef2f06509bf9ab28b82449894606a5d4dc60
Jul 17 07:53:36.357234 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:36.357194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerStarted","Data":"fe39239557acd9814c326689fb0eef2f06509bf9ab28b82449894606a5d4dc60"}
Jul 17 07:53:37.130798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:37.130764    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:53:37.364536 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:37.364497    2584 generic.go:356] "Generic (PLEG): container finished" podID="4a768bdc-773d-4892-b931-ae557b156189" containerID="2cecc4d5f4c6873d9ec013bda1436861e9cfcd5d517407eb4482e3abf1504063" exitCode=6
Jul 17 07:53:37.365022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:37.364584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerDied","Data":"2cecc4d5f4c6873d9ec013bda1436861e9cfcd5d517407eb4482e3abf1504063"}
Jul 17 07:53:37.365022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:37.364794    2584 scope.go:117] "RemoveContainer" containerID="2cecc4d5f4c6873d9ec013bda1436861e9cfcd5d517407eb4482e3abf1504063"
Jul 17 07:53:38.346892 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:38.346861    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:53:38.371148 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:38.371111    2584 generic.go:356] "Generic (PLEG): container finished" podID="4a768bdc-773d-4892-b931-ae557b156189" containerID="58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423" exitCode=6
Jul 17 07:53:38.371610 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:38.371185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerDied","Data":"58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423"}
Jul 17 07:53:38.371610 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:38.371243    2584 scope.go:117] "RemoveContainer" containerID="2cecc4d5f4c6873d9ec013bda1436861e9cfcd5d517407eb4482e3abf1504063"
Jul 17 07:53:38.371610 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:38.371519    2584 scope.go:117] "RemoveContainer" containerID="58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423"
Jul 17 07:53:38.371784 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:38.371721    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt_odh-ai-gateway-infra(4a768bdc-773d-4892-b931-ae557b156189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" podUID="4a768bdc-773d-4892-b931-ae557b156189"
Jul 17 07:53:39.377353 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:39.377323    2584 scope.go:117] "RemoveContainer" containerID="58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423"
Jul 17 07:53:39.377740 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:39.377519    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt_odh-ai-gateway-infra(4a768bdc-773d-4892-b931-ae557b156189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" podUID="4a768bdc-773d-4892-b931-ae557b156189"
Jul 17 07:53:52.934542 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.934498    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:53:52.938271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.938247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.940778 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.940758    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e8d5b32b\"/\"kube-root-ca.crt\""
Jul 17 07:53:52.941557 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.941539    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e8d5b32b\"/\"openshift-service-ca.crt\""
Jul 17 07:53:52.941988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.941968    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e8d5b32b\"/\"default-dockercfg-h7gpc\""
Jul 17 07:53:52.942042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.942025    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e8d5b32b\"/\"e2e-lifecycle-model-e8d5b32b-kserve-self-signed-certs\""
Jul 17 07:53:52.947573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.947550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:53:52.959020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.958995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:53:52.999098 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.999262 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999103    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz6w\" (UniqueName: \"kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.999262 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999138    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.999262 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.999476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:52.999476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:52.999324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.099840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.099840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz6w\" (UniqueName: \"kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100091 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.099991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100322 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.100295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100455 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.100437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.100509 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.100476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.102862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.102838    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.103106 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.103077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.108900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.108878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz6w\" (UniqueName: \"kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w\") pod \"e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") " pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.248828 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.248728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:53:53.376071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.375876    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:53:53.378687 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:53:53.378660    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfda78e8b_f365_42bd_b2c4_74dcbaab5877.slice/crio-3b9d0b86793df57928dc110e81caf1b3ede986fae45ebdc7bc5b4d0c400344dd WatchSource:0}: Error finding container 3b9d0b86793df57928dc110e81caf1b3ede986fae45ebdc7bc5b4d0c400344dd: Status 404 returned error can't find the container with id 3b9d0b86793df57928dc110e81caf1b3ede986fae45ebdc7bc5b4d0c400344dd
Jul 17 07:53:53.449403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.449371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" event={"ID":"fda78e8b-f365-42bd-b2c4-74dcbaab5877","Type":"ContainerStarted","Data":"3b9d0b86793df57928dc110e81caf1b3ede986fae45ebdc7bc5b4d0c400344dd"}
Jul 17 07:53:53.468582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:53.468528    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" podStartSLOduration=1.468512907 podStartE2EDuration="1.468512907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:53:53.467235282 +0000 UTC m=+2529.720658115" watchObservedRunningTime="2026-07-17 07:53:53.468512907 +0000 UTC m=+2529.721935762"
Jul 17 07:53:54.286800 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:54.286768    2584 scope.go:117] "RemoveContainer" containerID="58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423"
Jul 17 07:53:54.454808 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:54.454782    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerStarted","Data":"f8fe3cac72e972fe8bc33b17b986a111c4d19b9e3f4b2015d265cdfe347c7cdd"}
Jul 17 07:53:54.456248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:54.456223    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" event={"ID":"fda78e8b-f365-42bd-b2c4-74dcbaab5877","Type":"ContainerStarted","Data":"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"}
Jul 17 07:53:54.472585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:54.472546    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" podStartSLOduration=19.472533058 podStartE2EDuration="19.472533058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:53:54.470254372 +0000 UTC m=+2530.723677181" watchObservedRunningTime="2026-07-17 07:53:54.472533058 +0000 UTC m=+2530.725955869"
Jul 17 07:53:55.462074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:55.462042    2584 generic.go:356] "Generic (PLEG): container finished" podID="4a768bdc-773d-4892-b931-ae557b156189" containerID="f8fe3cac72e972fe8bc33b17b986a111c4d19b9e3f4b2015d265cdfe347c7cdd" exitCode=6
Jul 17 07:53:55.462535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:55.462111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerDied","Data":"f8fe3cac72e972fe8bc33b17b986a111c4d19b9e3f4b2015d265cdfe347c7cdd"}
Jul 17 07:53:55.462535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:55.462168    2584 scope.go:117] "RemoveContainer" containerID="58c5cc2d96f10781c310aeb5a3e70d8173cafd60cc75f1cdc6bf085d9c154423"
Jul 17 07:53:55.462535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:55.462503    2584 scope.go:117] "RemoveContainer" containerID="f8fe3cac72e972fe8bc33b17b986a111c4d19b9e3f4b2015d265cdfe347c7cdd"
Jul 17 07:53:55.462694 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:53:55.462673    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt_odh-ai-gateway-infra(4a768bdc-773d-4892-b931-ae557b156189)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" podUID="4a768bdc-773d-4892-b931-ae557b156189"
Jul 17 07:53:55.486468 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:55.486443    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:56.072631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:56.072599    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:53:56.598458 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:56.598436    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:56.730826 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:56.730724    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca\") pod \"4a768bdc-773d-4892-b931-ae557b156189\" (UID: \"4a768bdc-773d-4892-b931-ae557b156189\") "
Jul 17 07:53:56.731100 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:56.731076    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a768bdc-773d-4892-b931-ae557b156189" (UID: "4a768bdc-773d-4892-b931-ae557b156189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:53:56.831779 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:56.831745    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a768bdc-773d-4892-b931-ae557b156189-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:53:57.474020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:57.473995    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"
Jul 17 07:53:57.474020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:57.474022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt" event={"ID":"4a768bdc-773d-4892-b931-ae557b156189","Type":"ContainerDied","Data":"fe39239557acd9814c326689fb0eef2f06509bf9ab28b82449894606a5d4dc60"}
Jul 17 07:53:57.474268 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:57.474079    2584 scope.go:117] "RemoveContainer" containerID="f8fe3cac72e972fe8bc33b17b986a111c4d19b9e3f4b2015d265cdfe347c7cdd"
Jul 17 07:53:57.497765 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:57.497739    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:57.503247 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:57.503223    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-h4hdt"]
Jul 17 07:53:58.290088 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:53:58.290055    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a768bdc-773d-4892-b931-ae557b156189" path="/var/lib/kubelet/pods/4a768bdc-773d-4892-b931-ae557b156189/volumes"
Jul 17 07:54:03.249835 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:03.249790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:03.250310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:03.249857    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:03.262274 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:03.262246    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:03.511980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:03.511899    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:09.040749 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.040713    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041013    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041024    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041036    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041042    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041050    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041055    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041128    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041136    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.041155 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.041142    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a768bdc-773d-4892-b931-ae557b156189" containerName="revoke-keys"
Jul 17 07:54:09.045293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.045276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:09.048222 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.048200    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:54:09.051620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.051596    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:09.131357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.131314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-54njl\" (UID: \"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:09.232218 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.232176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-54njl\" (UID: \"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:09.232841 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.232818    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-54njl\" (UID: \"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:09.357305 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.357212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:09.488089 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.488062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:09.490746 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:54:09.490714    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e4dcf3_ab4b_48b7_bfbb_5eeff46b8cab.slice/crio-530e6ba7a4121f88ef9c9e08fb87ede82acc67e0b5fd329e96def41cf81a7116 WatchSource:0}: Error finding container 530e6ba7a4121f88ef9c9e08fb87ede82acc67e0b5fd329e96def41cf81a7116: Status 404 returned error can't find the container with id 530e6ba7a4121f88ef9c9e08fb87ede82acc67e0b5fd329e96def41cf81a7116
Jul 17 07:54:09.527840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:09.527804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerStarted","Data":"530e6ba7a4121f88ef9c9e08fb87ede82acc67e0b5fd329e96def41cf81a7116"}
Jul 17 07:54:10.532942 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:10.532857    2584 generic.go:356] "Generic (PLEG): container finished" podID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerID="f8924ab418b8188096ddc0842974e09128972fffe69b946c682d084f0d8ea4ed" exitCode=6
Jul 17 07:54:10.533324 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:10.532941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerDied","Data":"f8924ab418b8188096ddc0842974e09128972fffe69b946c682d084f0d8ea4ed"}
Jul 17 07:54:10.533324 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:10.533113    2584 scope.go:117] "RemoveContainer" containerID="f8924ab418b8188096ddc0842974e09128972fffe69b946c682d084f0d8ea4ed"
Jul 17 07:54:10.728933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:10.728904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:54:11.538395 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.538365    2584 generic.go:356] "Generic (PLEG): container finished" podID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerID="2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b" exitCode=6
Jul 17 07:54:11.538803 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.538403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerDied","Data":"2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b"}
Jul 17 07:54:11.538803 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.538448    2584 scope.go:117] "RemoveContainer" containerID="f8924ab418b8188096ddc0842974e09128972fffe69b946c682d084f0d8ea4ed"
Jul 17 07:54:11.538803 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.538637    2584 scope.go:117] "RemoveContainer" containerID="2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b"
Jul 17 07:54:11.538916 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:11.538870    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-54njl_odh-ai-gateway-infra(85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab"
Jul 17 07:54:11.807532 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.807445    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:11.812404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.812379    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:11.818752 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.818723    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:11.956836 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:11.956799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg\" (UID: \"340dd364-fac9-4810-b7c7-fa5a06d4f28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:12.058021 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.057942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg\" (UID: \"340dd364-fac9-4810-b7c7-fa5a06d4f28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:12.058585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.058566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg\" (UID: \"340dd364-fac9-4810-b7c7-fa5a06d4f28a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:12.125269 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.125237    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:12.257905 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.257863    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:12.262022 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:54:12.261995    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340dd364_fac9_4810_b7c7_fa5a06d4f28a.slice/crio-2acaface38f8ac8965c5ae214c6140abe827d8036e4ca4e4a1dfeebfb1815268 WatchSource:0}: Error finding container 2acaface38f8ac8965c5ae214c6140abe827d8036e4ca4e4a1dfeebfb1815268: Status 404 returned error can't find the container with id 2acaface38f8ac8965c5ae214c6140abe827d8036e4ca4e4a1dfeebfb1815268
Jul 17 07:54:12.544443 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.544411    2584 scope.go:117] "RemoveContainer" containerID="2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b"
Jul 17 07:54:12.544814 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:12.544637    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-54njl_odh-ai-gateway-infra(85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab"
Jul 17 07:54:12.545671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.545643    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerStarted","Data":"e26f82e8ceb49b3f22b15a4e62739577f7f8afa5998b00597e0a9af7e0f03102"}
Jul 17 07:54:12.545815 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.545701    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerStarted","Data":"2acaface38f8ac8965c5ae214c6140abe827d8036e4ca4e4a1dfeebfb1815268"}
Jul 17 07:54:12.575976 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:12.575921    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" podStartSLOduration=1.575903162 podStartE2EDuration="1.575903162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:54:12.575781627 +0000 UTC m=+2548.829204444" watchObservedRunningTime="2026-07-17 07:54:12.575903162 +0000 UTC m=+2548.829325975"
Jul 17 07:54:13.551318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:13.551281    2584 generic.go:356] "Generic (PLEG): container finished" podID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerID="e26f82e8ceb49b3f22b15a4e62739577f7f8afa5998b00597e0a9af7e0f03102" exitCode=6
Jul 17 07:54:13.551717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:13.551356    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerDied","Data":"e26f82e8ceb49b3f22b15a4e62739577f7f8afa5998b00597e0a9af7e0f03102"}
Jul 17 07:54:13.551717 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:13.551589    2584 scope.go:117] "RemoveContainer" containerID="e26f82e8ceb49b3f22b15a4e62739577f7f8afa5998b00597e0a9af7e0f03102"
Jul 17 07:54:14.557444 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:14.557407    2584 generic.go:356] "Generic (PLEG): container finished" podID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerID="4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2" exitCode=6
Jul 17 07:54:14.557856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:14.557502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerDied","Data":"4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2"}
Jul 17 07:54:14.557856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:14.557564    2584 scope.go:117] "RemoveContainer" containerID="e26f82e8ceb49b3f22b15a4e62739577f7f8afa5998b00597e0a9af7e0f03102"
Jul 17 07:54:14.557856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:14.557656    2584 scope.go:117] "RemoveContainer" containerID="4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2"
Jul 17 07:54:14.558033 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:14.557858    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg_odh-ai-gateway-infra(340dd364-fac9-4810-b7c7-fa5a06d4f28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a"
Jul 17 07:54:15.563530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:15.563504    2584 scope.go:117] "RemoveContainer" containerID="4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2"
Jul 17 07:54:15.563917 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:15.563665    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg_odh-ai-gateway-infra(340dd364-fac9-4810-b7c7-fa5a06d4f28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a"
Jul 17 07:54:23.285281 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:23.285253    2584 scope.go:117] "RemoveContainer" containerID="2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b"
Jul 17 07:54:23.601956 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:23.601874    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerStarted","Data":"99d06c28fae19ccea7ee8cb79bb99f586763ce767293e5939a8ad28edb5705bc"}
Jul 17 07:54:23.635566 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:23.635516    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" podStartSLOduration=14.635500799 podStartE2EDuration="14.635500799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:54:23.633416017 +0000 UTC m=+2559.886838823" watchObservedRunningTime="2026-07-17 07:54:23.635500799 +0000 UTC m=+2559.888923611"
Jul 17 07:54:24.608326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.608291    2584 generic.go:356] "Generic (PLEG): container finished" podID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerID="99d06c28fae19ccea7ee8cb79bb99f586763ce767293e5939a8ad28edb5705bc" exitCode=6
Jul 17 07:54:24.608720 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.608364    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerDied","Data":"99d06c28fae19ccea7ee8cb79bb99f586763ce767293e5939a8ad28edb5705bc"}
Jul 17 07:54:24.608720 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.608439    2584 scope.go:117] "RemoveContainer" containerID="2909bcf19df4cb8f68a115d68ff53a571dd3859568964d335ae8d499fc8fec6b"
Jul 17 07:54:24.608840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.608773    2584 scope.go:117] "RemoveContainer" containerID="99d06c28fae19ccea7ee8cb79bb99f586763ce767293e5939a8ad28edb5705bc"
Jul 17 07:54:24.609026 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:24.609004    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-54njl_odh-ai-gateway-infra(85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab"
Jul 17 07:54:24.649440 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.649405    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:24.734445 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.734407    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:54:24.734685 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.734659    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" podUID="af583d5b-5348-4135-9425-e7889c3021bf" containerName="maas-api" containerID="cri-o://1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6" gracePeriod=30
Jul 17 07:54:24.774613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.774580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:54:24.983738 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:24.983711    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:54:25.170891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.170787    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca\") pod \"af583d5b-5348-4135-9425-e7889c3021bf\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") "
Jul 17 07:54:25.170891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.170886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") pod \"af583d5b-5348-4135-9425-e7889c3021bf\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") "
Jul 17 07:54:25.171093 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.170940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgf28\" (UniqueName: \"kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28\") pod \"af583d5b-5348-4135-9425-e7889c3021bf\" (UID: \"af583d5b-5348-4135-9425-e7889c3021bf\") "
Jul 17 07:54:25.171186 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.171161    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af583d5b-5348-4135-9425-e7889c3021bf" (UID: "af583d5b-5348-4135-9425-e7889c3021bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:54:25.173063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.173037    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af583d5b-5348-4135-9425-e7889c3021bf" (UID: "af583d5b-5348-4135-9425-e7889c3021bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:54:25.173063 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.173053    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28" (OuterVolumeSpecName: "kube-api-access-zgf28") pod "af583d5b-5348-4135-9425-e7889c3021bf" (UID: "af583d5b-5348-4135-9425-e7889c3021bf"). InnerVolumeSpecName "kube-api-access-zgf28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:54:25.272229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.272184    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf28\" (UniqueName: \"kubernetes.io/projected/af583d5b-5348-4135-9425-e7889c3021bf-kube-api-access-zgf28\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:25.272229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.272216    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af583d5b-5348-4135-9425-e7889c3021bf-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:25.272229 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.272231    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af583d5b-5348-4135-9425-e7889c3021bf-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:25.614141 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.614108    2584 generic.go:356] "Generic (PLEG): container finished" podID="af583d5b-5348-4135-9425-e7889c3021bf" containerID="1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6" exitCode=0
Jul 17 07:54:25.614631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.614170    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"
Jul 17 07:54:25.614631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.614193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" event={"ID":"af583d5b-5348-4135-9425-e7889c3021bf","Type":"ContainerDied","Data":"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"}
Jul 17 07:54:25.614631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.614266    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh" event={"ID":"af583d5b-5348-4135-9425-e7889c3021bf","Type":"ContainerDied","Data":"914fac06124bf01ed30abd56c587b7dd2abb72ccdb476f8b327eeb65a113c410"}
Jul 17 07:54:25.614631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.614300    2584 scope.go:117] "RemoveContainer" containerID="1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"
Jul 17 07:54:25.623149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.623131    2584 scope.go:117] "RemoveContainer" containerID="1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"
Jul 17 07:54:25.623599 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:25.623466    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6\": container with ID starting with 1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6 not found: ID does not exist" containerID="1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"
Jul 17 07:54:25.623599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.623507    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6"} err="failed to get container status \"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6\": rpc error: code = NotFound desc = could not find container \"1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6\": container with ID starting with 1f9b8935c09511faab4b70fe850b1fed03dbeeab8dc50b542b5bebb6a65bd0c6 not found: ID does not exist"
Jul 17 07:54:25.658216 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.658180    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:54:25.662711 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.662681    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e8d5b32b-56474795cf-cbgdh"]
Jul 17 07:54:25.764388 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.764363    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:25.877875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.877796    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca\") pod \"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab\" (UID: \"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab\") "
Jul 17 07:54:25.878133 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.878112    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" (UID: "85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:54:25.979292 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:25.979247    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:26.284908 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.284878    2584 scope.go:117] "RemoveContainer" containerID="4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2"
Jul 17 07:54:26.290080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.290061    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af583d5b-5348-4135-9425-e7889c3021bf" path="/var/lib/kubelet/pods/af583d5b-5348-4135-9425-e7889c3021bf/volumes"
Jul 17 07:54:26.622479 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.622449    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"
Jul 17 07:54:26.622931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.622448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl" event={"ID":"85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab","Type":"ContainerDied","Data":"530e6ba7a4121f88ef9c9e08fb87ede82acc67e0b5fd329e96def41cf81a7116"}
Jul 17 07:54:26.622931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.622598    2584 scope.go:117] "RemoveContainer" containerID="99d06c28fae19ccea7ee8cb79bb99f586763ce767293e5939a8ad28edb5705bc"
Jul 17 07:54:26.624445 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.624417    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerStarted","Data":"1559ff4f1f98953bba0d623cc5fb5481b35a8083e14ecd90711e718db722466e"}
Jul 17 07:54:26.641009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.640981    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:26.643642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:26.643623    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-54njl"]
Jul 17 07:54:27.631512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:27.631482    2584 generic.go:356] "Generic (PLEG): container finished" podID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerID="1559ff4f1f98953bba0d623cc5fb5481b35a8083e14ecd90711e718db722466e" exitCode=6
Jul 17 07:54:27.631945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:27.631542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerDied","Data":"1559ff4f1f98953bba0d623cc5fb5481b35a8083e14ecd90711e718db722466e"}
Jul 17 07:54:27.631945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:27.631587    2584 scope.go:117] "RemoveContainer" containerID="4963401d71f740edb0c420eac2dec30841ab2b61811141b855e21c4483420da2"
Jul 17 07:54:27.631945 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:27.631859    2584 scope.go:117] "RemoveContainer" containerID="1559ff4f1f98953bba0d623cc5fb5481b35a8083e14ecd90711e718db722466e"
Jul 17 07:54:27.632124 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:27.632083    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg_odh-ai-gateway-infra(340dd364-fac9-4810-b7c7-fa5a06d4f28a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a"
Jul 17 07:54:27.674144 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:27.674113    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:28.289875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.289839    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" path="/var/lib/kubelet/pods/85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab/volumes"
Jul 17 07:54:28.471762 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.471723    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:54:28.770464 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.770439    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:28.802893 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.802864    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca\") pod \"340dd364-fac9-4810-b7c7-fa5a06d4f28a\" (UID: \"340dd364-fac9-4810-b7c7-fa5a06d4f28a\") "
Jul 17 07:54:28.803200 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.803180    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "340dd364-fac9-4810-b7c7-fa5a06d4f28a" (UID: "340dd364-fac9-4810-b7c7-fa5a06d4f28a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:54:28.903731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:28.903657    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340dd364-fac9-4810-b7c7-fa5a06d4f28a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:29.090245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.090211    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:54:29.642646 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.642616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg" event={"ID":"340dd364-fac9-4810-b7c7-fa5a06d4f28a","Type":"ContainerDied","Data":"2acaface38f8ac8965c5ae214c6140abe827d8036e4ca4e4a1dfeebfb1815268"}
Jul 17 07:54:29.642837 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.642654    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"
Jul 17 07:54:29.642837 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.642674    2584 scope.go:117] "RemoveContainer" containerID="1559ff4f1f98953bba0d623cc5fb5481b35a8083e14ecd90711e718db722466e"
Jul 17 07:54:29.679641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.679614    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:29.683753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:29.683733    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-nrcqg"]
Jul 17 07:54:30.289996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:30.289962    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" path="/var/lib/kubelet/pods/340dd364-fac9-4810-b7c7-fa5a06d4f28a/volumes"
Jul 17 07:54:34.059176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:34.059138    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:54:34.154176 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:34.154137    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:54:43.803306 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:43.803267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:54:49.079409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.079369    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" podUID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" containerName="main" containerID="cri-o://b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c" gracePeriod=45
Jul 17 07:54:49.329619 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.329555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:49.335965 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.335941    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.335997    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vz6w\" (UniqueName: \"kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336037    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336053    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336073    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm\") pod \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\" (UID: \"fda78e8b-f365-42bd-b2c4-74dcbaab5877\") "
Jul 17 07:54:49.336224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336183    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home" (OuterVolumeSpecName: "home") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 07:54:49.336408 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336278    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-home\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.336408 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336316    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache" (OuterVolumeSpecName: "model-cache") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 07:54:49.336519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.336418    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 07:54:49.338266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.338240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:54:49.338357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.338275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm" (OuterVolumeSpecName: "dshm") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 07:54:49.338404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.338380    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w" (OuterVolumeSpecName: "kube-api-access-4vz6w") pod "fda78e8b-f365-42bd-b2c4-74dcbaab5877" (UID: "fda78e8b-f365-42bd-b2c4-74dcbaab5877"). InnerVolumeSpecName "kube-api-access-4vz6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:54:49.436667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.436635    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-model-cache\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.436667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.436659    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tmp-dir\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.436667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.436668    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda78e8b-f365-42bd-b2c4-74dcbaab5877-tls-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.436667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.436676    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda78e8b-f365-42bd-b2c4-74dcbaab5877-dshm\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.436933 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.436685    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz6w\" (UniqueName: \"kubernetes.io/projected/fda78e8b-f365-42bd-b2c4-74dcbaab5877-kube-api-access-4vz6w\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:54:49.735267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.735169    2584 generic.go:356] "Generic (PLEG): container finished" podID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" containerID="b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c" exitCode=0
Jul 17 07:54:49.735267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.735219    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"
Jul 17 07:54:49.735534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.735261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" event={"ID":"fda78e8b-f365-42bd-b2c4-74dcbaab5877","Type":"ContainerDied","Data":"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"}
Jul 17 07:54:49.735534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.735303    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws" event={"ID":"fda78e8b-f365-42bd-b2c4-74dcbaab5877","Type":"ContainerDied","Data":"3b9d0b86793df57928dc110e81caf1b3ede986fae45ebdc7bc5b4d0c400344dd"}
Jul 17 07:54:49.735534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.735320    2584 scope.go:117] "RemoveContainer" containerID="b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"
Jul 17 07:54:49.743774 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.743757    2584 scope.go:117] "RemoveContainer" containerID="b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"
Jul 17 07:54:49.744041 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:54:49.744023    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c\": container with ID starting with b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c not found: ID does not exist" containerID="b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"
Jul 17 07:54:49.744083 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.744050    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c"} err="failed to get container status \"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c\": rpc error: code = NotFound desc = could not find container \"b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c\": container with ID starting with b1378a0fd7e81048f6cb40b5236a525e8632275ea7f99ed2de82aa8f6d4d435c not found: ID does not exist"
Jul 17 07:54:49.756576 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.756550    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:54:49.760501 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:49.760477    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e8d5b32b/e2e-lifecycle-model-e8d5b32b-kserve-77f865d57f-wsmws"]
Jul 17 07:54:50.291573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:50.291535    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" path="/var/lib/kubelet/pods/fda78e8b-f365-42bd-b2c4-74dcbaab5877/volumes"
Jul 17 07:54:55.017893 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:55.017858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:54:59.119910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.119873    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120164    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120177    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120187    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120193    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120204    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af583d5b-5348-4135-9425-e7889c3021bf" containerName="maas-api"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120209    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="af583d5b-5348-4135-9425-e7889c3021bf" containerName="maas-api"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120237    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120243    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120253    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" containerName="main"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120259    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" containerName="main"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120267    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120272    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120319    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120325    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120331    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda78e8b-f365-42bd-b2c4-74dcbaab5877" containerName="main"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120350    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120356    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120364    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="af583d5b-5348-4135-9425-e7889c3021bf" containerName="maas-api"
Jul 17 07:54:59.120772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.120371    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:54:59.123201 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.123184    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:54:59.126068 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.126043    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:54:59.126373 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.126360    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:54:59.133102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.133074    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:54:59.201937 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.201905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8\" (UID: \"1ada547a-f7d3-4198-b57a-c2c9ac76ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:54:59.303424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.303384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8\" (UID: \"1ada547a-f7d3-4198-b57a-c2c9ac76ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:54:59.304267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.304245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8\" (UID: \"1ada547a-f7d3-4198-b57a-c2c9ac76ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:54:59.434363 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.434265    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:54:59.574225 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.574189    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:54:59.576306 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:54:59.576280    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ada547a_f7d3_4198_b57a_c2c9ac76ca29.slice/crio-c2126201502f1f9b26e110890643e727d471cc8f2290475b4dbb2068f8286205 WatchSource:0}: Error finding container c2126201502f1f9b26e110890643e727d471cc8f2290475b4dbb2068f8286205: Status 404 returned error can't find the container with id c2126201502f1f9b26e110890643e727d471cc8f2290475b4dbb2068f8286205
Jul 17 07:54:59.783941 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:54:59.783906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" event={"ID":"1ada547a-f7d3-4198-b57a-c2c9ac76ca29","Type":"ContainerStarted","Data":"c2126201502f1f9b26e110890643e727d471cc8f2290475b4dbb2068f8286205"}
Jul 17 07:55:00.789769 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:00.789737    2584 generic.go:356] "Generic (PLEG): container finished" podID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerID="c2ea35e0737276749e33649f671e9b73939a3f27ee0dd42f797e096fb6dba4de" exitCode=6
Jul 17 07:55:00.790208 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:00.789792    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" event={"ID":"1ada547a-f7d3-4198-b57a-c2c9ac76ca29","Type":"ContainerDied","Data":"c2ea35e0737276749e33649f671e9b73939a3f27ee0dd42f797e096fb6dba4de"}
Jul 17 07:55:00.790208 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:00.790064    2584 scope.go:117] "RemoveContainer" containerID="c2ea35e0737276749e33649f671e9b73939a3f27ee0dd42f797e096fb6dba4de"
Jul 17 07:55:01.796779 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.796743    2584 generic.go:356] "Generic (PLEG): container finished" podID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerID="dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e" exitCode=6
Jul 17 07:55:01.797202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.796819    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" event={"ID":"1ada547a-f7d3-4198-b57a-c2c9ac76ca29","Type":"ContainerDied","Data":"dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e"}
Jul 17 07:55:01.797202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.796870    2584 scope.go:117] "RemoveContainer" containerID="c2ea35e0737276749e33649f671e9b73939a3f27ee0dd42f797e096fb6dba4de"
Jul 17 07:55:01.797202 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.797049    2584 scope.go:117] "RemoveContainer" containerID="dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e"
Jul 17 07:55:01.797359 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:01.797273    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8_odh-ai-gateway-infra(1ada547a-f7d3-4198-b57a-c2c9ac76ca29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29"
Jul 17 07:55:01.897941 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.897904    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:01.898250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.898238    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:55:01.898297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.898255    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="340dd364-fac9-4810-b7c7-fa5a06d4f28a" containerName="revoke-keys"
Jul 17 07:55:01.898297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.898271    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:55:01.898297 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.898277    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:55:01.898413 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.898323    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e4dcf3-ab4b-48b7-bfbb-5eeff46b8cab" containerName="revoke-keys"
Jul 17 07:55:01.902404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.902387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:01.922659 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.922631    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:01.923688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:01.923659    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-gscm9\" (UID: \"7f3e1a30-48b0-4008-8671-483a1f3586f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:02.024832 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.024794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-gscm9\" (UID: \"7f3e1a30-48b0-4008-8671-483a1f3586f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:02.025436 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.025418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-gscm9\" (UID: \"7f3e1a30-48b0-4008-8671-483a1f3586f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:02.211738 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.211648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:02.339777 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.339752    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:02.341781 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:55:02.341753    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3e1a30_48b0_4008_8671_483a1f3586f6.slice/crio-55f99e1f0f89a7795f1da5423dac890db0a23b56142133000a94894624cd7bd6 WatchSource:0}: Error finding container 55f99e1f0f89a7795f1da5423dac890db0a23b56142133000a94894624cd7bd6: Status 404 returned error can't find the container with id 55f99e1f0f89a7795f1da5423dac890db0a23b56142133000a94894624cd7bd6
Jul 17 07:55:02.802973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.802868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerStarted","Data":"1a059f277228cf7c0232854a54a0b2abe5e9dc61058b7cb60ba4056e7fbdc41c"}
Jul 17 07:55:02.802973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.802932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerStarted","Data":"55f99e1f0f89a7795f1da5423dac890db0a23b56142133000a94894624cd7bd6"}
Jul 17 07:55:02.804608 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.804586    2584 scope.go:117] "RemoveContainer" containerID="dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e"
Jul 17 07:55:02.804787 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:02.804770    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8_odh-ai-gateway-infra(1ada547a-f7d3-4198-b57a-c2c9ac76ca29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29"
Jul 17 07:55:02.820850 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:02.820803    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" podStartSLOduration=1.820788674 podStartE2EDuration="1.820788674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:55:02.820125097 +0000 UTC m=+2599.073547934" watchObservedRunningTime="2026-07-17 07:55:02.820788674 +0000 UTC m=+2599.074211486"
Jul 17 07:55:03.810149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:03.810117    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerID="1a059f277228cf7c0232854a54a0b2abe5e9dc61058b7cb60ba4056e7fbdc41c" exitCode=6
Jul 17 07:55:03.810586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:03.810158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerDied","Data":"1a059f277228cf7c0232854a54a0b2abe5e9dc61058b7cb60ba4056e7fbdc41c"}
Jul 17 07:55:03.810586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:03.810409    2584 scope.go:117] "RemoveContainer" containerID="1a059f277228cf7c0232854a54a0b2abe5e9dc61058b7cb60ba4056e7fbdc41c"
Jul 17 07:55:04.815991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:04.815960    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerID="2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453" exitCode=6
Jul 17 07:55:04.816499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:04.816017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerDied","Data":"2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453"}
Jul 17 07:55:04.816499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:04.816068    2584 scope.go:117] "RemoveContainer" containerID="1a059f277228cf7c0232854a54a0b2abe5e9dc61058b7cb60ba4056e7fbdc41c"
Jul 17 07:55:04.816499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:04.816264    2584 scope.go:117] "RemoveContainer" containerID="2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453"
Jul 17 07:55:04.816671 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:04.816543    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-gscm9_odh-ai-gateway-infra(7f3e1a30-48b0-4008-8671-483a1f3586f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6"
Jul 17 07:55:05.821452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:05.821423    2584 scope.go:117] "RemoveContainer" containerID="2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453"
Jul 17 07:55:05.821931 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:05.821624    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-gscm9_odh-ai-gateway-infra(7f3e1a30-48b0-4008-8671-483a1f3586f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6"
Jul 17 07:55:10.266218 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:10.266140    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:10.365862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:10.365821    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:11.117268 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:11.117236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:12.245464 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:12.245419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:13.285009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:13.284975    2584 scope.go:117] "RemoveContainer" containerID="dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e"
Jul 17 07:55:13.856950 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:13.856922    2584 generic.go:356] "Generic (PLEG): container finished" podID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerID="2379dc1d4538dce7b7a6ffc2d76b1c97d8cdb2e27d1232d76597e74843527901" exitCode=6
Jul 17 07:55:13.857081 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:13.856973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" event={"ID":"1ada547a-f7d3-4198-b57a-c2c9ac76ca29","Type":"ContainerDied","Data":"2379dc1d4538dce7b7a6ffc2d76b1c97d8cdb2e27d1232d76597e74843527901"}
Jul 17 07:55:13.857081 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:13.857044    2584 scope.go:117] "RemoveContainer" containerID="dceb6e33241632561288b3f645f40c343192361fa41c723819f37d7893c8572e"
Jul 17 07:55:13.857379 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:13.857329    2584 scope.go:117] "RemoveContainer" containerID="2379dc1d4538dce7b7a6ffc2d76b1c97d8cdb2e27d1232d76597e74843527901"
Jul 17 07:55:13.857598 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:13.857578    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8_odh-ai-gateway-infra(1ada547a-f7d3-4198-b57a-c2c9ac76ca29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29"
Jul 17 07:55:14.887210 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:14.887175    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:55:15.020645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.020613    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:55:15.214071 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.213971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca\") pod \"1ada547a-f7d3-4198-b57a-c2c9ac76ca29\" (UID: \"1ada547a-f7d3-4198-b57a-c2c9ac76ca29\") "
Jul 17 07:55:15.214375 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.214322    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ada547a-f7d3-4198-b57a-c2c9ac76ca29" (UID: "1ada547a-f7d3-4198-b57a-c2c9ac76ca29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:55:15.314965 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.314932    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ada547a-f7d3-4198-b57a-c2c9ac76ca29-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:55:15.867410 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.867362    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8" event={"ID":"1ada547a-f7d3-4198-b57a-c2c9ac76ca29","Type":"ContainerDied","Data":"c2126201502f1f9b26e110890643e727d471cc8f2290475b4dbb2068f8286205"}
Jul 17 07:55:15.867656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.867387    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"
Jul 17 07:55:15.867656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.867429    2584 scope.go:117] "RemoveContainer" containerID="2379dc1d4538dce7b7a6ffc2d76b1c97d8cdb2e27d1232d76597e74843527901"
Jul 17 07:55:15.887347 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:15.887302    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ada547a_f7d3_4198_b57a_c2c9ac76ca29.slice\": RecentStats: unable to find data in memory cache]"
Jul 17 07:55:15.890958 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.890930    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:55:15.898187 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:15.898161    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-pmvx8"]
Jul 17 07:55:16.290384 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:16.290328    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" path="/var/lib/kubelet/pods/1ada547a-f7d3-4198-b57a-c2c9ac76ca29/volumes"
Jul 17 07:55:18.880868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:18.880831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:19.285364 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:19.285324    2584 scope.go:117] "RemoveContainer" containerID="2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453"
Jul 17 07:55:19.889167 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:19.889138    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerID="177dd3d933065e8183ae73d3cfe5da5196ca69d6ea5b25745b22b068259a4996" exitCode=6
Jul 17 07:55:19.889603 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:19.889181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerDied","Data":"177dd3d933065e8183ae73d3cfe5da5196ca69d6ea5b25745b22b068259a4996"}
Jul 17 07:55:19.889603 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:19.889228    2584 scope.go:117] "RemoveContainer" containerID="2cb204dfc6a847fb0889963d8daa29c14a8fd37fff0faf4ed75db536ace64453"
Jul 17 07:55:19.889603 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:19.889517    2584 scope.go:117] "RemoveContainer" containerID="177dd3d933065e8183ae73d3cfe5da5196ca69d6ea5b25745b22b068259a4996"
Jul 17 07:55:19.889731 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:19.889698    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-gscm9_odh-ai-gateway-infra(7f3e1a30-48b0-4008-8671-483a1f3586f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6"
Jul 17 07:55:20.915968 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:20.915929    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:21.060321 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.060292    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:21.156548 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.156512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca\") pod \"7f3e1a30-48b0-4008-8671-483a1f3586f6\" (UID: \"7f3e1a30-48b0-4008-8671-483a1f3586f6\") "
Jul 17 07:55:21.156917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.156891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f3e1a30-48b0-4008-8671-483a1f3586f6" (UID: "7f3e1a30-48b0-4008-8671-483a1f3586f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:55:21.257954 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.257873    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f3e1a30-48b0-4008-8671-483a1f3586f6-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:55:21.900152 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.900119    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"
Jul 17 07:55:21.900396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.900112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9" event={"ID":"7f3e1a30-48b0-4008-8671-483a1f3586f6","Type":"ContainerDied","Data":"55f99e1f0f89a7795f1da5423dac890db0a23b56142133000a94894624cd7bd6"}
Jul 17 07:55:21.900396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.900276    2584 scope.go:117] "RemoveContainer" containerID="177dd3d933065e8183ae73d3cfe5da5196ca69d6ea5b25745b22b068259a4996"
Jul 17 07:55:21.921666 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.921636    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:21.926980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:21.926953    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-gscm9"]
Jul 17 07:55:22.290608 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:22.290578    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" path="/var/lib/kubelet/pods/7f3e1a30-48b0-4008-8671-483a1f3586f6/volumes"
Jul 17 07:55:25.719267 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.719228    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:25.720508 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720480    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.720669 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720657    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.720754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720745    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.720823 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720815    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.720907 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720898    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.720975 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.720967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.721064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721055    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.721128 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721120    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.721209 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721200    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.721310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721268    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.721558 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721542    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.721654 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721644    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.721731 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721722    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:25.721811 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.721802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:25.726629 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.726606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:25.729630 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.729608    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:55:25.729810 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.729789    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:55:25.730713 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.730691    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:25.796289 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.796257    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7\" (UID: \"e1345eaa-75dd-4424-ab03-39334bec32db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:25.897460 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.897432    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7\" (UID: \"e1345eaa-75dd-4424-ab03-39334bec32db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:25.898023 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:25.898003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7\" (UID: \"e1345eaa-75dd-4424-ab03-39334bec32db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:26.038622 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.038596    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:26.172654 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.172622    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:26.173087 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:55:26.173061    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1345eaa_75dd_4424_ab03_39334bec32db.slice/crio-b719c16eb05537184cd97a4ec42a67cd0459bbf5f8e73bba44f7175c6a4cfb5e WatchSource:0}: Error finding container b719c16eb05537184cd97a4ec42a67cd0459bbf5f8e73bba44f7175c6a4cfb5e: Status 404 returned error can't find the container with id b719c16eb05537184cd97a4ec42a67cd0459bbf5f8e73bba44f7175c6a4cfb5e
Jul 17 07:55:26.922310 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.922277    2584 generic.go:356] "Generic (PLEG): container finished" podID="e1345eaa-75dd-4424-ab03-39334bec32db" containerID="eef919c97a85d97169c65fc29546a8d65bdadbe109c172f4be075d519bf34178" exitCode=6
Jul 17 07:55:26.922742 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.922366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" event={"ID":"e1345eaa-75dd-4424-ab03-39334bec32db","Type":"ContainerDied","Data":"eef919c97a85d97169c65fc29546a8d65bdadbe109c172f4be075d519bf34178"}
Jul 17 07:55:26.922742 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.922420    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" event={"ID":"e1345eaa-75dd-4424-ab03-39334bec32db","Type":"ContainerStarted","Data":"b719c16eb05537184cd97a4ec42a67cd0459bbf5f8e73bba44f7175c6a4cfb5e"}
Jul 17 07:55:26.922742 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:26.922590    2584 scope.go:117] "RemoveContainer" containerID="eef919c97a85d97169c65fc29546a8d65bdadbe109c172f4be075d519bf34178"
Jul 17 07:55:27.928084 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:27.928052    2584 generic.go:356] "Generic (PLEG): container finished" podID="e1345eaa-75dd-4424-ab03-39334bec32db" containerID="cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a" exitCode=6
Jul 17 07:55:27.928529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:27.928112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" event={"ID":"e1345eaa-75dd-4424-ab03-39334bec32db","Type":"ContainerDied","Data":"cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a"}
Jul 17 07:55:27.928529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:27.928168    2584 scope.go:117] "RemoveContainer" containerID="eef919c97a85d97169c65fc29546a8d65bdadbe109c172f4be075d519bf34178"
Jul 17 07:55:27.928529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:27.928427    2584 scope.go:117] "RemoveContainer" containerID="cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a"
Jul 17 07:55:27.928653 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:27.928609    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7_odh-ai-gateway-infra(e1345eaa-75dd-4424-ab03-39334bec32db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" podUID="e1345eaa-75dd-4424-ab03-39334bec32db"
Jul 17 07:55:28.934041 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:28.934014    2584 scope.go:117] "RemoveContainer" containerID="cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a"
Jul 17 07:55:28.934412 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:28.934207    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7_odh-ai-gateway-infra(e1345eaa-75dd-4424-ab03-39334bec32db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" podUID="e1345eaa-75dd-4424-ab03-39334bec32db"
Jul 17 07:55:33.713093 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.713056    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:33.713474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.713386    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:33.713474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.713401    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:33.713474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.713467    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ada547a-f7d3-4198-b57a-c2c9ac76ca29" containerName="revoke-keys"
Jul 17 07:55:33.713582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.713477    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3e1a30-48b0-4008-8671-483a1f3586f6" containerName="revoke-keys"
Jul 17 07:55:33.717555 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.717537    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:33.728953 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.728930    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:33.761278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.761248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk\" (UID: \"924dc558-31bf-4a92-9cd4-714ed2d378be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:33.862181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.862146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk\" (UID: \"924dc558-31bf-4a92-9cd4-714ed2d378be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:33.862819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.862800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk\" (UID: \"924dc558-31bf-4a92-9cd4-714ed2d378be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:33.987215 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:33.987133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:34.027792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.027764    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:34.087748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.087707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:34.164072 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.164050    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:34.165933 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:55:34.165897    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924dc558_31bf_4a92_9cd4_714ed2d378be.slice/crio-6d333c5d162ce1acf1039c8d9489f9a203760c2b0c511eb4b9fad1d9661d6aea WatchSource:0}: Error finding container 6d333c5d162ce1acf1039c8d9489f9a203760c2b0c511eb4b9fad1d9661d6aea: Status 404 returned error can't find the container with id 6d333c5d162ce1acf1039c8d9489f9a203760c2b0c511eb4b9fad1d9661d6aea
Jul 17 07:55:34.961612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.961580    2584 generic.go:356] "Generic (PLEG): container finished" podID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerID="00dcd9df3c162b26d5f47ed1e1d8e4c4ce11383cb85be5e908e42d7b47e3bb77" exitCode=6
Jul 17 07:55:34.962090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.961636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" event={"ID":"924dc558-31bf-4a92-9cd4-714ed2d378be","Type":"ContainerDied","Data":"00dcd9df3c162b26d5f47ed1e1d8e4c4ce11383cb85be5e908e42d7b47e3bb77"}
Jul 17 07:55:34.962090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.961674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" event={"ID":"924dc558-31bf-4a92-9cd4-714ed2d378be","Type":"ContainerStarted","Data":"6d333c5d162ce1acf1039c8d9489f9a203760c2b0c511eb4b9fad1d9661d6aea"}
Jul 17 07:55:34.962090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:34.961871    2584 scope.go:117] "RemoveContainer" containerID="00dcd9df3c162b26d5f47ed1e1d8e4c4ce11383cb85be5e908e42d7b47e3bb77"
Jul 17 07:55:35.970961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:35.970870    2584 generic.go:356] "Generic (PLEG): container finished" podID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerID="c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218" exitCode=6
Jul 17 07:55:35.971370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:35.970957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" event={"ID":"924dc558-31bf-4a92-9cd4-714ed2d378be","Type":"ContainerDied","Data":"c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218"}
Jul 17 07:55:35.971370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:35.971051    2584 scope.go:117] "RemoveContainer" containerID="00dcd9df3c162b26d5f47ed1e1d8e4c4ce11383cb85be5e908e42d7b47e3bb77"
Jul 17 07:55:35.971370 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:35.971184    2584 scope.go:117] "RemoveContainer" containerID="c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218"
Jul 17 07:55:35.971497 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:35.971425    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk_odh-ai-gateway-infra(924dc558-31bf-4a92-9cd4-714ed2d378be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be"
Jul 17 07:55:36.976938 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:36.976906    2584 scope.go:117] "RemoveContainer" containerID="c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218"
Jul 17 07:55:36.977527 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:36.977153    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk_odh-ai-gateway-infra(924dc558-31bf-4a92-9cd4-714ed2d378be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be"
Jul 17 07:55:40.284913 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:40.284874    2584 scope.go:117] "RemoveContainer" containerID="cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a"
Jul 17 07:55:40.996570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:40.996536    2584 generic.go:356] "Generic (PLEG): container finished" podID="e1345eaa-75dd-4424-ab03-39334bec32db" containerID="5142dd05e353de3f08db3dffcfbce8428ceea57ca94ef27e08751726377f71f8" exitCode=6
Jul 17 07:55:40.996732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:40.996588    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" event={"ID":"e1345eaa-75dd-4424-ab03-39334bec32db","Type":"ContainerDied","Data":"5142dd05e353de3f08db3dffcfbce8428ceea57ca94ef27e08751726377f71f8"}
Jul 17 07:55:40.996732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:40.996658    2584 scope.go:117] "RemoveContainer" containerID="cff2040e2e50721ed209820cbfb4be5a0197158431cd78404708c4e6e6ed080a"
Jul 17 07:55:40.996982 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:40.996952    2584 scope.go:117] "RemoveContainer" containerID="5142dd05e353de3f08db3dffcfbce8428ceea57ca94ef27e08751726377f71f8"
Jul 17 07:55:40.997168 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:40.997146    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7_odh-ai-gateway-infra(e1345eaa-75dd-4424-ab03-39334bec32db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" podUID="e1345eaa-75dd-4424-ab03-39334bec32db"
Jul 17 07:55:41.658529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:41.658488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:41.904114 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:41.904074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:42.027541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:42.027455    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:42.161613 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:42.161590    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:42.237526 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:42.237497    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca\") pod \"e1345eaa-75dd-4424-ab03-39334bec32db\" (UID: \"e1345eaa-75dd-4424-ab03-39334bec32db\") "
Jul 17 07:55:42.237805 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:42.237786    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1345eaa-75dd-4424-ab03-39334bec32db" (UID: "e1345eaa-75dd-4424-ab03-39334bec32db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:55:42.338574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:42.338550    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1345eaa-75dd-4424-ab03-39334bec32db-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:55:43.008059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.008022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7" event={"ID":"e1345eaa-75dd-4424-ab03-39334bec32db","Type":"ContainerDied","Data":"b719c16eb05537184cd97a4ec42a67cd0459bbf5f8e73bba44f7175c6a4cfb5e"}
Jul 17 07:55:43.008059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.008040    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"
Jul 17 07:55:43.008586 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.008080    2584 scope.go:117] "RemoveContainer" containerID="5142dd05e353de3f08db3dffcfbce8428ceea57ca94ef27e08751726377f71f8"
Jul 17 07:55:43.041168 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.041144    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:43.055190 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.055167    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-2h7b7"]
Jul 17 07:55:43.077902 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.077870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:43.176574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.176536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:43.286707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:43.286671    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:44.067660 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:44.067629    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:44.171663 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:44.171632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:44.267143 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:44.267109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:44.289787 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:44.289754    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" path="/var/lib/kubelet/pods/e1345eaa-75dd-4424-ab03-39334bec32db/volumes"
Jul 17 07:55:45.543779 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:45.543749    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:55:45.813535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:45.813460    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:49.166573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166541    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:55:49.166927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166853    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.166927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166865    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.166927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166886    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.166927 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166894    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.167077 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166949    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.167077 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.166957    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:55:49.171260 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.171241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:55:49.181778 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.181752    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:55:49.298519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.298487    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt\" (UID: \"92f42628-6f44-4db5-8d6e-879ec394b782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:55:49.399635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.399600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt\" (UID: \"92f42628-6f44-4db5-8d6e-879ec394b782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:55:49.400252 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.400232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt\" (UID: \"92f42628-6f44-4db5-8d6e-879ec394b782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:55:49.481791 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.481710    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:55:49.612981 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:49.612948    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:55:49.616478 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:55:49.616453    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f42628_6f44_4db5_8d6e_879ec394b782.slice/crio-59b4890ccbfa9e8b4161f5583c0848f31f6fac85bb4aa75d1c2ea37f158430bb WatchSource:0}: Error finding container 59b4890ccbfa9e8b4161f5583c0848f31f6fac85bb4aa75d1c2ea37f158430bb: Status 404 returned error can't find the container with id 59b4890ccbfa9e8b4161f5583c0848f31f6fac85bb4aa75d1c2ea37f158430bb
Jul 17 07:55:50.038694 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:50.038660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerStarted","Data":"bade26b061ddc27fbf2c9c7237c46a0b6bbfbd02c2d4a9d9bb1b33ad54ec7d45"}
Jul 17 07:55:50.038861 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:50.038709    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerStarted","Data":"59b4890ccbfa9e8b4161f5583c0848f31f6fac85bb4aa75d1c2ea37f158430bb"}
Jul 17 07:55:50.059334 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:50.059292    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" podStartSLOduration=1.059276755 podStartE2EDuration="1.059276755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:55:50.057812779 +0000 UTC m=+2646.311235590" watchObservedRunningTime="2026-07-17 07:55:50.059276755 +0000 UTC m=+2646.312699566"
Jul 17 07:55:51.044654 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:51.044622    2584 generic.go:356] "Generic (PLEG): container finished" podID="92f42628-6f44-4db5-8d6e-879ec394b782" containerID="bade26b061ddc27fbf2c9c7237c46a0b6bbfbd02c2d4a9d9bb1b33ad54ec7d45" exitCode=6
Jul 17 07:55:51.045043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:51.044690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerDied","Data":"bade26b061ddc27fbf2c9c7237c46a0b6bbfbd02c2d4a9d9bb1b33ad54ec7d45"}
Jul 17 07:55:51.045043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:51.044860    2584 scope.go:117] "RemoveContainer" containerID="bade26b061ddc27fbf2c9c7237c46a0b6bbfbd02c2d4a9d9bb1b33ad54ec7d45"
Jul 17 07:55:51.285354 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:51.285312    2584 scope.go:117] "RemoveContainer" containerID="c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218"
Jul 17 07:55:52.052118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.052082    2584 generic.go:356] "Generic (PLEG): container finished" podID="92f42628-6f44-4db5-8d6e-879ec394b782" containerID="c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702" exitCode=6
Jul 17 07:55:52.052571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.052160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerDied","Data":"c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702"}
Jul 17 07:55:52.052571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.052246    2584 scope.go:117] "RemoveContainer" containerID="bade26b061ddc27fbf2c9c7237c46a0b6bbfbd02c2d4a9d9bb1b33ad54ec7d45"
Jul 17 07:55:52.052571 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.052467    2584 scope.go:117] "RemoveContainer" containerID="c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702"
Jul 17 07:55:52.052766 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:52.052702    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt_odh-ai-gateway-infra(92f42628-6f44-4db5-8d6e-879ec394b782)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" podUID="92f42628-6f44-4db5-8d6e-879ec394b782"
Jul 17 07:55:52.054116 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.054097    2584 generic.go:356] "Generic (PLEG): container finished" podID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerID="46848c5a1623f2248cb7e357e6975a86c0147647d2633c121a002694e430214d" exitCode=6
Jul 17 07:55:52.054194 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.054139    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" event={"ID":"924dc558-31bf-4a92-9cd4-714ed2d378be","Type":"ContainerDied","Data":"46848c5a1623f2248cb7e357e6975a86c0147647d2633c121a002694e430214d"}
Jul 17 07:55:52.054422 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.054409    2584 scope.go:117] "RemoveContainer" containerID="46848c5a1623f2248cb7e357e6975a86c0147647d2633c121a002694e430214d"
Jul 17 07:55:52.054594 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:52.054575    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk_odh-ai-gateway-infra(924dc558-31bf-4a92-9cd4-714ed2d378be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be"
Jul 17 07:55:52.062973 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.062955    2584 scope.go:117] "RemoveContainer" containerID="c5157b78b2c603830a195df1c6a3c6b93d89eb173944dca0cc4c407b5ebb0218"
Jul 17 07:55:52.342655 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.342611    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:52.415266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:52.415232    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:53.062456 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.062427    2584 scope.go:117] "RemoveContainer" containerID="c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702"
Jul 17 07:55:53.062820 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:55:53.062610    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt_odh-ai-gateway-infra(92f42628-6f44-4db5-8d6e-879ec394b782)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" podUID="92f42628-6f44-4db5-8d6e-879ec394b782"
Jul 17 07:55:53.071823 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.071801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:53.106076 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.106049    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:53.258741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.258712    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:53.337601 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.337516    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca\") pod \"924dc558-31bf-4a92-9cd4-714ed2d378be\" (UID: \"924dc558-31bf-4a92-9cd4-714ed2d378be\") "
Jul 17 07:55:53.337875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.337853    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "924dc558-31bf-4a92-9cd4-714ed2d378be" (UID: "924dc558-31bf-4a92-9cd4-714ed2d378be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:55:53.438996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:53.438948    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924dc558-31bf-4a92-9cd4-714ed2d378be-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:55:54.067036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.067006    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"
Jul 17 07:55:54.067488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.067047    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk" event={"ID":"924dc558-31bf-4a92-9cd4-714ed2d378be","Type":"ContainerDied","Data":"6d333c5d162ce1acf1039c8d9489f9a203760c2b0c511eb4b9fad1d9661d6aea"}
Jul 17 07:55:54.067488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.067105    2584 scope.go:117] "RemoveContainer" containerID="46848c5a1623f2248cb7e357e6975a86c0147647d2633c121a002694e430214d"
Jul 17 07:55:54.089072 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.089048    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:54.092647 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.092627    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-zf9zk"]
Jul 17 07:55:54.289929 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:54.289896    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" path="/var/lib/kubelet/pods/924dc558-31bf-4a92-9cd4-714ed2d378be/volumes"
Jul 17 07:55:56.346863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:56.346830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:55:59.390657 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:59.390621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:59.505741 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:59.505702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:55:59.714043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:55:59.713963    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:01.942946 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.942916    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943213    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943224    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943244    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943249    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943260    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943266    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943311    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943319    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1345eaa-75dd-4424-ab03-39334bec32db" containerName="revoke-keys"
Jul 17 07:56:01.943328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.943327    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:01.947636 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.947617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:01.955406 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:01.955383    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:02.113253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:02.113222    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q\" (UID: \"364696bb-7dcb-4548-a89e-9c4c7164021a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:02.214587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:02.214496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q\" (UID: \"364696bb-7dcb-4548-a89e-9c4c7164021a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:02.215149 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:02.215123    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q\" (UID: \"364696bb-7dcb-4548-a89e-9c4c7164021a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:02.258698 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:02.258667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:02.392383 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:02.392355    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:02.394410 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:02.394379    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364696bb_7dcb_4548_a89e_9c4c7164021a.slice/crio-b57a9fafe8eb4e67cde4fa175692ee32dd881a9e0d74712b0f641a7d1eb0c2ef WatchSource:0}: Error finding container b57a9fafe8eb4e67cde4fa175692ee32dd881a9e0d74712b0f641a7d1eb0c2ef: Status 404 returned error can't find the container with id b57a9fafe8eb4e67cde4fa175692ee32dd881a9e0d74712b0f641a7d1eb0c2ef
Jul 17 07:56:03.115614 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:03.115572    2584 generic.go:356] "Generic (PLEG): container finished" podID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerID="2883a1ca0f83caae3f8a607e228a8ce23c5a2cf4d59402b7d40c72eeea4abc2f" exitCode=6
Jul 17 07:56:03.116080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:03.115655    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" event={"ID":"364696bb-7dcb-4548-a89e-9c4c7164021a","Type":"ContainerDied","Data":"2883a1ca0f83caae3f8a607e228a8ce23c5a2cf4d59402b7d40c72eeea4abc2f"}
Jul 17 07:56:03.116080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:03.115710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" event={"ID":"364696bb-7dcb-4548-a89e-9c4c7164021a","Type":"ContainerStarted","Data":"b57a9fafe8eb4e67cde4fa175692ee32dd881a9e0d74712b0f641a7d1eb0c2ef"}
Jul 17 07:56:03.116080 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:03.115848    2584 scope.go:117] "RemoveContainer" containerID="2883a1ca0f83caae3f8a607e228a8ce23c5a2cf4d59402b7d40c72eeea4abc2f"
Jul 17 07:56:04.122165 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:04.122132    2584 generic.go:356] "Generic (PLEG): container finished" podID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerID="68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98" exitCode=6
Jul 17 07:56:04.122633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:04.122170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" event={"ID":"364696bb-7dcb-4548-a89e-9c4c7164021a","Type":"ContainerDied","Data":"68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98"}
Jul 17 07:56:04.122633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:04.122214    2584 scope.go:117] "RemoveContainer" containerID="2883a1ca0f83caae3f8a607e228a8ce23c5a2cf4d59402b7d40c72eeea4abc2f"
Jul 17 07:56:04.122633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:04.122437    2584 scope.go:117] "RemoveContainer" containerID="68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98"
Jul 17 07:56:04.122758 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:04.122657    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q_odh-ai-gateway-infra(364696bb-7dcb-4548-a89e-9c4c7164021a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a"
Jul 17 07:56:05.128275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:05.128247    2584 scope.go:117] "RemoveContainer" containerID="68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98"
Jul 17 07:56:05.128645 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:05.128451    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q_odh-ai-gateway-infra(364696bb-7dcb-4548-a89e-9c4c7164021a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a"
Jul 17 07:56:05.285398 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:05.285374    2584 scope.go:117] "RemoveContainer" containerID="c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702"
Jul 17 07:56:06.133645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:06.133564    2584 generic.go:356] "Generic (PLEG): container finished" podID="92f42628-6f44-4db5-8d6e-879ec394b782" containerID="356f6a460d738d9e92ceb2ed096cbdf640d8f915e4af53013d0b7ab4c8404077" exitCode=6
Jul 17 07:56:06.133645 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:06.133616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerDied","Data":"356f6a460d738d9e92ceb2ed096cbdf640d8f915e4af53013d0b7ab4c8404077"}
Jul 17 07:56:06.134111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:06.133664    2584 scope.go:117] "RemoveContainer" containerID="c750c968b2625deab414e8f949bae6efc1aef4c38fd283f9b48dd9064d24e702"
Jul 17 07:56:06.134111 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:06.133959    2584 scope.go:117] "RemoveContainer" containerID="356f6a460d738d9e92ceb2ed096cbdf640d8f915e4af53013d0b7ab4c8404077"
Jul 17 07:56:06.134202 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:06.134175    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt_odh-ai-gateway-infra(92f42628-6f44-4db5-8d6e-879ec394b782)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" podUID="92f42628-6f44-4db5-8d6e-879ec394b782"
Jul 17 07:56:07.162469 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:07.162436    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:56:07.291582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:07.291561    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:56:07.348578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:07.348552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca\") pod \"92f42628-6f44-4db5-8d6e-879ec394b782\" (UID: \"92f42628-6f44-4db5-8d6e-879ec394b782\") "
Jul 17 07:56:07.348879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:07.348859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92f42628-6f44-4db5-8d6e-879ec394b782" (UID: "92f42628-6f44-4db5-8d6e-879ec394b782"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:56:07.449925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:07.449855    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f42628-6f44-4db5-8d6e-879ec394b782-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:56:08.144709 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.144662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt" event={"ID":"92f42628-6f44-4db5-8d6e-879ec394b782","Type":"ContainerDied","Data":"59b4890ccbfa9e8b4161f5583c0848f31f6fac85bb4aa75d1c2ea37f158430bb"}
Jul 17 07:56:08.144709 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.144691    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"
Jul 17 07:56:08.145079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.144721    2584 scope.go:117] "RemoveContainer" containerID="356f6a460d738d9e92ceb2ed096cbdf640d8f915e4af53013d0b7ab4c8404077"
Jul 17 07:56:08.166820 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.166796    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:56:08.172737 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.172717    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lcfpt"]
Jul 17 07:56:08.289565 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:08.289540    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" path="/var/lib/kubelet/pods/92f42628-6f44-4db5-8d6e-879ec394b782/volumes"
Jul 17 07:56:10.222117 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:10.222080    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:10.629554 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:10.629513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:15.769031 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.768998    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769363    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769377    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769389    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769395    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769404    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769409    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:15.769467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769468    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="924dc558-31bf-4a92-9cd4-714ed2d378be" containerName="revoke-keys"
Jul 17 07:56:15.769697 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769475    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769697 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769483    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.769697 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.769491    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:15.773810 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.773790    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:15.784304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.784228    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:15.815750 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.815721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-244n4\" (UID: \"333a0b9e-f3ed-40a2-921a-405d56bf28d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:15.916546 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.916508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-244n4\" (UID: \"333a0b9e-f3ed-40a2-921a-405d56bf28d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:15.917075 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:15.917058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-244n4\" (UID: \"333a0b9e-f3ed-40a2-921a-405d56bf28d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:16.083990 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:16.083969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:16.214549 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:16.214516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:16.216261 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:16.216229    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333a0b9e_f3ed_40a2_921a_405d56bf28d7.slice/crio-0fdc9676913cdc1d41c8466d6044a138d4146a3b614b5e01fdef80c2c693049e WatchSource:0}: Error finding container 0fdc9676913cdc1d41c8466d6044a138d4146a3b614b5e01fdef80c2c693049e: Status 404 returned error can't find the container with id 0fdc9676913cdc1d41c8466d6044a138d4146a3b614b5e01fdef80c2c693049e
Jul 17 07:56:17.190494 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:17.190407    2584 generic.go:356] "Generic (PLEG): container finished" podID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerID="9fd9ec68361b1548c764bc3ac79f7a63c5ae93c1420772b9ba3573db7778e322" exitCode=6
Jul 17 07:56:17.190494 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:17.190457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" event={"ID":"333a0b9e-f3ed-40a2-921a-405d56bf28d7","Type":"ContainerDied","Data":"9fd9ec68361b1548c764bc3ac79f7a63c5ae93c1420772b9ba3573db7778e322"}
Jul 17 07:56:17.190896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:17.190502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" event={"ID":"333a0b9e-f3ed-40a2-921a-405d56bf28d7","Type":"ContainerStarted","Data":"0fdc9676913cdc1d41c8466d6044a138d4146a3b614b5e01fdef80c2c693049e"}
Jul 17 07:56:17.190896 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:17.190711    2584 scope.go:117] "RemoveContainer" containerID="9fd9ec68361b1548c764bc3ac79f7a63c5ae93c1420772b9ba3573db7778e322"
Jul 17 07:56:17.284602 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:17.284577    2584 scope.go:117] "RemoveContainer" containerID="68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98"
Jul 17 07:56:18.196788 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.196706    2584 generic.go:356] "Generic (PLEG): container finished" podID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerID="b8996d222daa594ddabc50451a44de055899550b668f8269f8ad3b4e252b0945" exitCode=6
Jul 17 07:56:18.196788 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.196769    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" event={"ID":"364696bb-7dcb-4548-a89e-9c4c7164021a","Type":"ContainerDied","Data":"b8996d222daa594ddabc50451a44de055899550b668f8269f8ad3b4e252b0945"}
Jul 17 07:56:18.197265 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.196824    2584 scope.go:117] "RemoveContainer" containerID="68c80413c596e86f2d2e07adfea65810121ae2f0459434d8be3ed4c3671b5e98"
Jul 17 07:56:18.197265 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.197137    2584 scope.go:117] "RemoveContainer" containerID="b8996d222daa594ddabc50451a44de055899550b668f8269f8ad3b4e252b0945"
Jul 17 07:56:18.197408 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:18.197388    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q_odh-ai-gateway-infra(364696bb-7dcb-4548-a89e-9c4c7164021a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a"
Jul 17 07:56:18.198712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.198694    2584 generic.go:356] "Generic (PLEG): container finished" podID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerID="2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e" exitCode=6
Jul 17 07:56:18.198798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.198724    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" event={"ID":"333a0b9e-f3ed-40a2-921a-405d56bf28d7","Type":"ContainerDied","Data":"2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e"}
Jul 17 07:56:18.198940 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.198926    2584 scope.go:117] "RemoveContainer" containerID="2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e"
Jul 17 07:56:18.199082 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:18.199065    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-244n4_odh-ai-gateway-infra(333a0b9e-f3ed-40a2-921a-405d56bf28d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7"
Jul 17 07:56:18.207813 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:18.207794    2584 scope.go:117] "RemoveContainer" containerID="9fd9ec68361b1548c764bc3ac79f7a63c5ae93c1420772b9ba3573db7778e322"
Jul 17 07:56:19.206628 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.206597    2584 scope.go:117] "RemoveContainer" containerID="2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e"
Jul 17 07:56:19.207017 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:19.206777    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-244n4_odh-ai-gateway-infra(333a0b9e-f3ed-40a2-921a-405d56bf28d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7"
Jul 17 07:56:19.232145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.232111    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:19.364486 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.364459    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:19.446147 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.446114    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca\") pod \"364696bb-7dcb-4548-a89e-9c4c7164021a\" (UID: \"364696bb-7dcb-4548-a89e-9c4c7164021a\") "
Jul 17 07:56:19.446510 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.446488    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "364696bb-7dcb-4548-a89e-9c4c7164021a" (UID: "364696bb-7dcb-4548-a89e-9c4c7164021a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:56:19.547024 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:19.546997    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364696bb-7dcb-4548-a89e-9c4c7164021a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:56:20.217387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.217351    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"
Jul 17 07:56:20.217387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.217377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q" event={"ID":"364696bb-7dcb-4548-a89e-9c4c7164021a","Type":"ContainerDied","Data":"b57a9fafe8eb4e67cde4fa175692ee32dd881a9e0d74712b0f641a7d1eb0c2ef"}
Jul 17 07:56:20.217856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.217468    2584 scope.go:117] "RemoveContainer" containerID="b8996d222daa594ddabc50451a44de055899550b668f8269f8ad3b4e252b0945"
Jul 17 07:56:20.240952 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.240924    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:20.247192 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.247169    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-hrq6q"]
Jul 17 07:56:20.290181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:20.290146    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" path="/var/lib/kubelet/pods/364696bb-7dcb-4548-a89e-9c4c7164021a/volumes"
Jul 17 07:56:31.654515 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:31.654482    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:33.785493 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785460    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785781    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785793    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785801    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785809    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f42628-6f44-4db5-8d6e-879ec394b782" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785820    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785825    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785835    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.785864 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785840    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.786142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785904    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.786142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785911    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.786142 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.785917    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="364696bb-7dcb-4548-a89e-9c4c7164021a" containerName="revoke-keys"
Jul 17 07:56:33.788926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.788905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:33.797159 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.797131    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:33.872029 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.871993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb\" (UID: \"2d1b510e-6a2c-4378-bf62-37b4d462f78d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:33.973293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.973248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb\" (UID: \"2d1b510e-6a2c-4378-bf62-37b4d462f78d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:33.973931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:33.973910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb\" (UID: \"2d1b510e-6a2c-4378-bf62-37b4d462f78d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:34.100946 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:34.100867    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:34.234560 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:34.234529    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:34.236919 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:34.236880    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1b510e_6a2c_4378_bf62_37b4d462f78d.slice/crio-1eb663a681ab21b815389fd5522cea225b610bb45401b445e53d0c3cd5041894 WatchSource:0}: Error finding container 1eb663a681ab21b815389fd5522cea225b610bb45401b445e53d0c3cd5041894: Status 404 returned error can't find the container with id 1eb663a681ab21b815389fd5522cea225b610bb45401b445e53d0c3cd5041894
Jul 17 07:56:34.280587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:34.280555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" event={"ID":"2d1b510e-6a2c-4378-bf62-37b4d462f78d","Type":"ContainerStarted","Data":"1eb663a681ab21b815389fd5522cea225b610bb45401b445e53d0c3cd5041894"}
Jul 17 07:56:34.286719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:34.286702    2584 scope.go:117] "RemoveContainer" containerID="2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e"
Jul 17 07:56:35.286329 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.286296    2584 generic.go:356] "Generic (PLEG): container finished" podID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerID="f3412aa8858605da5d75c4dd3bbd7d6e8037d2f597bc205dfbb80cafcafd9adf" exitCode=6
Jul 17 07:56:35.286780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.286370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" event={"ID":"333a0b9e-f3ed-40a2-921a-405d56bf28d7","Type":"ContainerDied","Data":"f3412aa8858605da5d75c4dd3bbd7d6e8037d2f597bc205dfbb80cafcafd9adf"}
Jul 17 07:56:35.286780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.286431    2584 scope.go:117] "RemoveContainer" containerID="2e971db59c4e8391924d23913444b0dca8860a8ec521de8f349cd20e5c59bb5e"
Jul 17 07:56:35.286780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.286734    2584 scope.go:117] "RemoveContainer" containerID="f3412aa8858605da5d75c4dd3bbd7d6e8037d2f597bc205dfbb80cafcafd9adf"
Jul 17 07:56:35.286949 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:35.286929    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-244n4_odh-ai-gateway-infra(333a0b9e-f3ed-40a2-921a-405d56bf28d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7"
Jul 17 07:56:35.288103 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.288080    2584 generic.go:356] "Generic (PLEG): container finished" podID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerID="ac15b443c7e90af150c3d3a33ad5e4b6b65f3aeed9dc6a8f967ef29f07414f9c" exitCode=6
Jul 17 07:56:35.288183 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.288109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" event={"ID":"2d1b510e-6a2c-4378-bf62-37b4d462f78d","Type":"ContainerDied","Data":"ac15b443c7e90af150c3d3a33ad5e4b6b65f3aeed9dc6a8f967ef29f07414f9c"}
Jul 17 07:56:35.288301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:35.288289    2584 scope.go:117] "RemoveContainer" containerID="ac15b443c7e90af150c3d3a33ad5e4b6b65f3aeed9dc6a8f967ef29f07414f9c"
Jul 17 07:56:36.294599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.294565    2584 generic.go:356] "Generic (PLEG): container finished" podID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerID="5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd" exitCode=6
Jul 17 07:56:36.295020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.294646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" event={"ID":"2d1b510e-6a2c-4378-bf62-37b4d462f78d","Type":"ContainerDied","Data":"5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd"}
Jul 17 07:56:36.295020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.294703    2584 scope.go:117] "RemoveContainer" containerID="ac15b443c7e90af150c3d3a33ad5e4b6b65f3aeed9dc6a8f967ef29f07414f9c"
Jul 17 07:56:36.295020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.294885    2584 scope.go:117] "RemoveContainer" containerID="5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd"
Jul 17 07:56:36.295159 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:36.295056    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb_odh-ai-gateway-infra(2d1b510e-6a2c-4378-bf62-37b4d462f78d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d"
Jul 17 07:56:36.314707 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.314682    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:36.448312 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.448291    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:36.595992 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.595888    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca\") pod \"333a0b9e-f3ed-40a2-921a-405d56bf28d7\" (UID: \"333a0b9e-f3ed-40a2-921a-405d56bf28d7\") "
Jul 17 07:56:36.596316 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.596295    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "333a0b9e-f3ed-40a2-921a-405d56bf28d7" (UID: "333a0b9e-f3ed-40a2-921a-405d56bf28d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:56:36.696855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:36.696789    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333a0b9e-f3ed-40a2-921a-405d56bf28d7-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:56:37.300144 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.300114    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"
Jul 17 07:56:37.300144 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.300122    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4" event={"ID":"333a0b9e-f3ed-40a2-921a-405d56bf28d7","Type":"ContainerDied","Data":"0fdc9676913cdc1d41c8466d6044a138d4146a3b614b5e01fdef80c2c693049e"}
Jul 17 07:56:37.300672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.300191    2584 scope.go:117] "RemoveContainer" containerID="f3412aa8858605da5d75c4dd3bbd7d6e8037d2f597bc205dfbb80cafcafd9adf"
Jul 17 07:56:37.302303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.302278    2584 scope.go:117] "RemoveContainer" containerID="5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd"
Jul 17 07:56:37.302499 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:37.302480    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb_odh-ai-gateway-infra(2d1b510e-6a2c-4378-bf62-37b4d462f78d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d"
Jul 17 07:56:37.340569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.340539    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:37.345836 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.345814    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-244n4"]
Jul 17 07:56:37.915529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:37.915493    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:38.059090 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:38.059053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:38.289627 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:38.289596    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" path="/var/lib/kubelet/pods/333a0b9e-f3ed-40a2-921a-405d56bf28d7/volumes"
Jul 17 07:56:38.398654 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:38.398620    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:38.706059 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:38.705978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:38.812594 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:38.812555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:39.215512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215481    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:56:39.215792 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215780    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.215878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215793    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.215878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215804    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.215878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215809    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.215878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215868    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.215878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.215878    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:39.219857 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.219840    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:39.229949 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.229925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:56:39.319698 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.319660    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7\" (UID: \"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:39.420653 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.420617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7\" (UID: \"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:39.421293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.421273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7\" (UID: \"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:39.530718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.530613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:39.669429 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:39.669352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:56:39.672101 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:39.672075    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c2dae9_f7e2_4ba2_abe1_542fb6aa61e9.slice/crio-d22af50e3c0ed8159a52ed32733908da93ca9e35c328cfc6b409f47cf06df700 WatchSource:0}: Error finding container d22af50e3c0ed8159a52ed32733908da93ca9e35c328cfc6b409f47cf06df700: Status 404 returned error can't find the container with id d22af50e3c0ed8159a52ed32733908da93ca9e35c328cfc6b409f47cf06df700
Jul 17 07:56:40.316416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:40.316369    2584 generic.go:356] "Generic (PLEG): container finished" podID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerID="0a5bed831f624642003c429bb28f31144063373ef4281f6f791e2a76dd338db4" exitCode=6
Jul 17 07:56:40.316610 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:40.316434    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" event={"ID":"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9","Type":"ContainerDied","Data":"0a5bed831f624642003c429bb28f31144063373ef4281f6f791e2a76dd338db4"}
Jul 17 07:56:40.316610 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:40.316487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" event={"ID":"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9","Type":"ContainerStarted","Data":"d22af50e3c0ed8159a52ed32733908da93ca9e35c328cfc6b409f47cf06df700"}
Jul 17 07:56:40.316732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:40.316711    2584 scope.go:117] "RemoveContainer" containerID="0a5bed831f624642003c429bb28f31144063373ef4281f6f791e2a76dd338db4"
Jul 17 07:56:41.323145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.323113    2584 generic.go:356] "Generic (PLEG): container finished" podID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerID="e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1" exitCode=6
Jul 17 07:56:41.323625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.323167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" event={"ID":"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9","Type":"ContainerDied","Data":"e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1"}
Jul 17 07:56:41.323625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.323220    2584 scope.go:117] "RemoveContainer" containerID="0a5bed831f624642003c429bb28f31144063373ef4281f6f791e2a76dd338db4"
Jul 17 07:56:41.323625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.323407    2584 scope.go:117] "RemoveContainer" containerID="e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1"
Jul 17 07:56:41.323783 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:41.323636    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7_odh-ai-gateway-infra(93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9"
Jul 17 07:56:41.768496 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.768397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:41.915097 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.915059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:41.945036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.944997    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:56:41.945401 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.945384    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:41.945401 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.945402    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:41.945560 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.945523    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="333a0b9e-f3ed-40a2-921a-405d56bf28d7" containerName="revoke-keys"
Jul 17 07:56:41.949603 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.949582    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:41.953513 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.953493    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f268b-serving-cert\""
Jul 17 07:56:41.955246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.955228    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:56:41.956359 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.956324    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:56:41.972849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:41.972819    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:56:42.044503 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.044472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.044672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.044519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6q5\" (UniqueName: \"kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.044672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.044567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.104841 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.104807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:42.145416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.145382    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6q5\" (UniqueName: \"kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.145600 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.145427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.145600 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.145492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.145721 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:42.145612    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-serving-cert: secret "maas-api-e2e-inf-a-1f268b-serving-cert" not found
Jul 17 07:56:42.145721 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:42.145688    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls podName:84d021d0-1e61-4d84-82da-d4766d051b01 nodeName:}" failed. No retries permitted until 2026-07-17 07:56:42.645665479 +0000 UTC m=+2698.899088286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls") pod "maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" (UID: "84d021d0-1e61-4d84-82da-d4766d051b01") : secret "maas-api-e2e-inf-a-1f268b-serving-cert" not found
Jul 17 07:56:42.146094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.146075    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.165462 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.165441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6q5\" (UniqueName: \"kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.330275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.330199    2584 scope.go:117] "RemoveContainer" containerID="e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1"
Jul 17 07:56:42.330660 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:42.330404    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7_odh-ai-gateway-infra(93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9"
Jul 17 07:56:42.501954 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.501920    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:42.649601 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.649507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.651967 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.651936    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.845235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.845204    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:42.859327 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.859293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:42.947208 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:42.947172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:43.000194 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:43.000165    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:56:43.001195 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:43.001161    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d021d0_1e61_4d84_82da_d4766d051b01.slice/crio-22cf48e0ddf9e6f0fb1366db52c8bc6c339159ca48826bc366b06d8c1e886c31 WatchSource:0}: Error finding container 22cf48e0ddf9e6f0fb1366db52c8bc6c339159ca48826bc366b06d8c1e886c31: Status 404 returned error can't find the container with id 22cf48e0ddf9e6f0fb1366db52c8bc6c339159ca48826bc366b06d8c1e886c31
Jul 17 07:56:43.335007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:43.334975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" event={"ID":"84d021d0-1e61-4d84-82da-d4766d051b01","Type":"ContainerStarted","Data":"22cf48e0ddf9e6f0fb1366db52c8bc6c339159ca48826bc366b06d8c1e886c31"}
Jul 17 07:56:44.341582 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:44.341544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" event={"ID":"84d021d0-1e61-4d84-82da-d4766d051b01","Type":"ContainerStarted","Data":"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"}
Jul 17 07:56:44.342048 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:44.341607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:44.389456 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:44.389406    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" podStartSLOduration=2.942325448 podStartE2EDuration="3.389387752s" totalImagesPullingTime="447.062304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:56:44.386470617 +0000 UTC m=+2700.639893455" watchObservedRunningTime="2026-07-17 07:56:44.389387752 +0000 UTC m=+2700.642810566"
Jul 17 07:56:44.396714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:44.396691    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:56:44.399425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:44.399404    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 07:56:46.317781 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:46.317750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:46.685783 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:46.685675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:47.059273 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:47.059236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:47.300067 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:47.300030    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:49.284793 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.284764    2584 scope.go:117] "RemoveContainer" containerID="5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd"
Jul 17 07:56:49.450863 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.450826    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:56:49.907587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.907495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:49.981903 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.981859    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:56:49.985366 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.985314    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:49.988135 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:49.988107    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cab91c-serving-cert\""
Jul 17 07:56:50.010196 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.010169    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:56:50.019659 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.019214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:50.106970 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.106937    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:50.116134 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.116085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.116134 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.116142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.116332 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.116267    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746km\" (UniqueName: \"kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.217326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.217222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746km\" (UniqueName: \"kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.217326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.217290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.217326 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.217314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.218051 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.218031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.219798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.219780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.233743 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.233715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-746km\" (UniqueName: \"kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km\") pod \"maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.235352 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.235316    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:50.296028 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.296000    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:50.353622 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.353599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:56:50.368010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.367982    2584 generic.go:356] "Generic (PLEG): container finished" podID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerID="94a50c2b17c3de53ece5af88aa8b90be8d1c5cde934a10c5a6e5482ffc479da1" exitCode=6
Jul 17 07:56:50.368315 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.368054    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" event={"ID":"2d1b510e-6a2c-4378-bf62-37b4d462f78d","Type":"ContainerDied","Data":"94a50c2b17c3de53ece5af88aa8b90be8d1c5cde934a10c5a6e5482ffc479da1"}
Jul 17 07:56:50.368315 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.368122    2584 scope.go:117] "RemoveContainer" containerID="5bb53930964c25d66344c1baa4f27f81fbf297a406e6d1f1e2e20100358d56dd"
Jul 17 07:56:50.368599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.368584    2584 scope.go:117] "RemoveContainer" containerID="94a50c2b17c3de53ece5af88aa8b90be8d1c5cde934a10c5a6e5482ffc479da1"
Jul 17 07:56:50.368854 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:50.368813    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb_odh-ai-gateway-infra(2d1b510e-6a2c-4378-bf62-37b4d462f78d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d"
Jul 17 07:56:50.432732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.432691    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:56:50.436256 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:56:50.436225    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83080ca4_bf8f_472c_ab01_e9e382274db2.slice/crio-44ee743a8af3d7ad866098eda863fcc345ee537da7d3521c50cf0f085ebb3500 WatchSource:0}: Error finding container 44ee743a8af3d7ad866098eda863fcc345ee537da7d3521c50cf0f085ebb3500: Status 404 returned error can't find the container with id 44ee743a8af3d7ad866098eda863fcc345ee537da7d3521c50cf0f085ebb3500
Jul 17 07:56:50.685694 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.685661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:50.790887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:50.790848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:51.375414 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.375375    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" event={"ID":"83080ca4-bf8f-472c-ab01-e9e382274db2","Type":"ContainerStarted","Data":"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"}
Jul 17 07:56:51.375887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.375431    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" event={"ID":"83080ca4-bf8f-472c-ab01-e9e382274db2","Type":"ContainerStarted","Data":"44ee743a8af3d7ad866098eda863fcc345ee537da7d3521c50cf0f085ebb3500"}
Jul 17 07:56:51.375887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.375465    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:51.409759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.409710    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" podStartSLOduration=2.019155598 podStartE2EDuration="2.409694837s" totalImagesPullingTime="390.539239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:56:51.409463054 +0000 UTC m=+2707.662885865" watchObservedRunningTime="2026-07-17 07:56:51.409694837 +0000 UTC m=+2707.663117649"
Jul 17 07:56:51.410802 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.410784    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:51.544089 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.544064    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:51.731220 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.731133    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca\") pod \"2d1b510e-6a2c-4378-bf62-37b4d462f78d\" (UID: \"2d1b510e-6a2c-4378-bf62-37b4d462f78d\") "
Jul 17 07:56:51.731552 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.731533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d1b510e-6a2c-4378-bf62-37b4d462f78d" (UID: "2d1b510e-6a2c-4378-bf62-37b4d462f78d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:56:51.832463 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:51.832431    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b510e-6a2c-4378-bf62-37b4d462f78d-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:56:52.380466 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:52.380428    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb" event={"ID":"2d1b510e-6a2c-4378-bf62-37b4d462f78d","Type":"ContainerDied","Data":"1eb663a681ab21b815389fd5522cea225b610bb45401b445e53d0c3cd5041894"}
Jul 17 07:56:52.380869 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:52.380494    2584 scope.go:117] "RemoveContainer" containerID="94a50c2b17c3de53ece5af88aa8b90be8d1c5cde934a10c5a6e5482ffc479da1"
Jul 17 07:56:52.380869 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:52.380438    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"
Jul 17 07:56:52.397330 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:52.397308    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:52.409023 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:52.405195    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-sk8fb"]
Jul 17 07:56:54.291043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:54.291005    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" path="/var/lib/kubelet/pods/2d1b510e-6a2c-4378-bf62-37b4d462f78d/volumes"
Jul 17 07:56:54.382640 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:54.382604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:56.284994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:56.284961    2584 scope.go:117] "RemoveContainer" containerID="e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1"
Jul 17 07:56:57.358114 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.358079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:56:57.386499 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.386476    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:56:57.403664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.403631    2584 generic.go:356] "Generic (PLEG): container finished" podID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerID="0b01f6c421e5b668700b44180432d1e7e2f6ddf2ddde7b15061a2b6f6bf9ffd3" exitCode=6
Jul 17 07:56:57.403799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.403734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" event={"ID":"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9","Type":"ContainerDied","Data":"0b01f6c421e5b668700b44180432d1e7e2f6ddf2ddde7b15061a2b6f6bf9ffd3"}
Jul 17 07:56:57.403799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.403793    2584 scope.go:117] "RemoveContainer" containerID="e21d28cc498a1afb30e9627a0c04b0aa8d1486ad26a90e70f4eb19a7e3219ec1"
Jul 17 07:56:57.404138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:57.404079    2584 scope.go:117] "RemoveContainer" containerID="0b01f6c421e5b668700b44180432d1e7e2f6ddf2ddde7b15061a2b6f6bf9ffd3"
Jul 17 07:56:57.404364 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:56:57.404300    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7_odh-ai-gateway-infra(93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9"
Jul 17 07:56:58.430233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:58.430199    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:56:58.566296 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:58.566273    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:58.582250 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:58.582224    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca\") pod \"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9\" (UID: \"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9\") "
Jul 17 07:56:58.582634 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:58.582613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" (UID: "93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:56:58.683797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:58.683718    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:56:59.414910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:59.414885    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"
Jul 17 07:56:59.415094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:59.414881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7" event={"ID":"93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9","Type":"ContainerDied","Data":"d22af50e3c0ed8159a52ed32733908da93ca9e35c328cfc6b409f47cf06df700"}
Jul 17 07:56:59.415094 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:59.415019    2584 scope.go:117] "RemoveContainer" containerID="0b01f6c421e5b668700b44180432d1e7e2f6ddf2ddde7b15061a2b6f6bf9ffd3"
Jul 17 07:56:59.438199 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:59.438173    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:56:59.443360 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:56:59.443319    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-np7f7"]
Jul 17 07:57:00.290193 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:00.290158    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" path="/var/lib/kubelet/pods/93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9/volumes"
Jul 17 07:57:01.052522 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:01.052489    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:03.747403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:03.747368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:03.858409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:03.858374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:05.990758 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:05.990639    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:08.041445 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:08.041410    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:11.503962 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:11.503926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:11.604867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:11.604831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:15.841429 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841395    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841719    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841731    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841762    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841771    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841781    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.841799 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841788    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841807    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841812    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841867    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841875    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841883    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841889    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:15.842007 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.841896    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:15.845015 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.844982    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:15.848039 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.848019    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:57:15.854604 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.854579    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:15.923060 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:15.923025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p\" (UID: \"e7c3e561-8864-49e9-91d2-92e62f8205f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:16.023483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:16.023449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p\" (UID: \"e7c3e561-8864-49e9-91d2-92e62f8205f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:16.024074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:16.024054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p\" (UID: \"e7c3e561-8864-49e9-91d2-92e62f8205f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:16.155574 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:16.155492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:16.289513 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:57:16.289484    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c3e561_8864_49e9_91d2_92e62f8205f5.slice/crio-5b33933ae19eb60476c253074f18d714aaa2716f19db23ff458de9c0e9090afa WatchSource:0}: Error finding container 5b33933ae19eb60476c253074f18d714aaa2716f19db23ff458de9c0e9090afa: Status 404 returned error can't find the container with id 5b33933ae19eb60476c253074f18d714aaa2716f19db23ff458de9c0e9090afa
Jul 17 07:57:16.290730 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:16.290705    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:16.491967 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:16.491882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerStarted","Data":"5b33933ae19eb60476c253074f18d714aaa2716f19db23ff458de9c0e9090afa"}
Jul 17 07:57:17.498642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:17.498608    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerID="7911396b4ced06c3c31729bac4eca3585e9b0a5a009212f36deefa0378f55451" exitCode=6
Jul 17 07:57:17.499009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:17.498691    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerDied","Data":"7911396b4ced06c3c31729bac4eca3585e9b0a5a009212f36deefa0378f55451"}
Jul 17 07:57:17.499009 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:17.498888    2584 scope.go:117] "RemoveContainer" containerID="7911396b4ced06c3c31729bac4eca3585e9b0a5a009212f36deefa0378f55451"
Jul 17 07:57:18.504577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:18.504491    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerID="7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645" exitCode=6
Jul 17 07:57:18.504577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:18.504529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerDied","Data":"7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645"}
Jul 17 07:57:18.505026 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:18.504594    2584 scope.go:117] "RemoveContainer" containerID="7911396b4ced06c3c31729bac4eca3585e9b0a5a009212f36deefa0378f55451"
Jul 17 07:57:18.505026 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:18.504759    2584 scope.go:117] "RemoveContainer" containerID="7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645"
Jul 17 07:57:18.505026 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:18.504951    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p_odh-ai-gateway-infra(e7c3e561-8864-49e9-91d2-92e62f8205f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5"
Jul 17 07:57:19.510954 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:19.510929    2584 scope.go:117] "RemoveContainer" containerID="7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645"
Jul 17 07:57:19.511303 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:19.511104    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p_odh-ai-gateway-infra(e7c3e561-8864-49e9-91d2-92e62f8205f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5"
Jul 17 07:57:22.242328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:22.242296    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:23.845925 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.845894    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:23.846309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.846210    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:23.846309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.846222    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:23.846309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.846240    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:23.846309 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.846246    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b510e-6a2c-4378-bf62-37b4d462f78d" containerName="revoke-keys"
Jul 17 07:57:23.846495 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.846315    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2dae9-f7e2-4ba2-abe1-542fb6aa61e9" containerName="revoke-keys"
Jul 17 07:57:23.849158 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.849143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:23.860683 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.860656    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:23.894888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.894860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc\" (UID: \"38a5de6f-04f1-4dff-81fb-839b60f617eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:23.995847 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.995811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc\" (UID: \"38a5de6f-04f1-4dff-81fb-839b60f617eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:23.996520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:23.996495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc\" (UID: \"38a5de6f-04f1-4dff-81fb-839b60f617eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:24.160043 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:24.159947    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:24.302243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:24.302217    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:24.304328 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:57:24.304297    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a5de6f_04f1_4dff_81fb_839b60f617eb.slice/crio-bd33547351f60761a75f94614842db5ad81e8b5d09e7c11aea4439681f1b643f WatchSource:0}: Error finding container bd33547351f60761a75f94614842db5ad81e8b5d09e7c11aea4439681f1b643f: Status 404 returned error can't find the container with id bd33547351f60761a75f94614842db5ad81e8b5d09e7c11aea4439681f1b643f
Jul 17 07:57:24.539671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:24.539626    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerStarted","Data":"bd33547351f60761a75f94614842db5ad81e8b5d09e7c11aea4439681f1b643f"}
Jul 17 07:57:25.545798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:25.545765    2584 generic.go:356] "Generic (PLEG): container finished" podID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerID="7d440aa38dff8866c91f93b6cb2bc244820aa5c1ab3a1846684fe902bbe29248" exitCode=6
Jul 17 07:57:25.546272 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:25.545852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerDied","Data":"7d440aa38dff8866c91f93b6cb2bc244820aa5c1ab3a1846684fe902bbe29248"}
Jul 17 07:57:25.546272 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:25.546054    2584 scope.go:117] "RemoveContainer" containerID="7d440aa38dff8866c91f93b6cb2bc244820aa5c1ab3a1846684fe902bbe29248"
Jul 17 07:57:26.551873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:26.551845    2584 generic.go:356] "Generic (PLEG): container finished" podID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerID="e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334" exitCode=6
Jul 17 07:57:26.552275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:26.551887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerDied","Data":"e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334"}
Jul 17 07:57:26.552275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:26.551929    2584 scope.go:117] "RemoveContainer" containerID="7d440aa38dff8866c91f93b6cb2bc244820aa5c1ab3a1846684fe902bbe29248"
Jul 17 07:57:26.552275 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:26.552120    2584 scope.go:117] "RemoveContainer" containerID="e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334"
Jul 17 07:57:26.552489 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:26.552310    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc_odh-ai-gateway-infra(38a5de6f-04f1-4dff-81fb-839b60f617eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb"
Jul 17 07:57:27.557840 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:27.557815    2584 scope.go:117] "RemoveContainer" containerID="e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334"
Jul 17 07:57:27.558253 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:27.557978    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc_odh-ai-gateway-infra(38a5de6f-04f1-4dff-81fb-839b60f617eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb"
Jul 17 07:57:28.445357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:28.445307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:31.285628 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:31.285590    2584 scope.go:117] "RemoveContainer" containerID="7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645"
Jul 17 07:57:31.576745 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:31.576651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerStarted","Data":"ca1e2d9595e446aa8e095da1ad361bf2697f9e88d1da7495f2967a56c913c15d"}
Jul 17 07:57:31.596835 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:31.596786    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" podStartSLOduration=16.596771174 podStartE2EDuration="16.596771174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:57:31.595542441 +0000 UTC m=+2747.848965253" watchObservedRunningTime="2026-07-17 07:57:31.596771174 +0000 UTC m=+2747.850194023"
Jul 17 07:57:32.582585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:32.582554    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerID="ca1e2d9595e446aa8e095da1ad361bf2697f9e88d1da7495f2967a56c913c15d" exitCode=6
Jul 17 07:57:32.583016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:32.582606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerDied","Data":"ca1e2d9595e446aa8e095da1ad361bf2697f9e88d1da7495f2967a56c913c15d"}
Jul 17 07:57:32.583016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:32.582655    2584 scope.go:117] "RemoveContainer" containerID="7a52dd39e680d98121ee4cba673d1e6bf990fe4fad45fbb2d77facc4dabc2645"
Jul 17 07:57:32.583016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:32.582922    2584 scope.go:117] "RemoveContainer" containerID="ca1e2d9595e446aa8e095da1ad361bf2697f9e88d1da7495f2967a56c913c15d"
Jul 17 07:57:32.583174 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:32.583107    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p_odh-ai-gateway-infra(e7c3e561-8864-49e9-91d2-92e62f8205f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5"
Jul 17 07:57:32.606611 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:32.606584    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:33.635798 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:33.635766    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:33.718952 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:33.718930    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:33.774211 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:33.774183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca\") pod \"e7c3e561-8864-49e9-91d2-92e62f8205f5\" (UID: \"e7c3e561-8864-49e9-91d2-92e62f8205f5\") "
Jul 17 07:57:33.774579 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:33.774557    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7c3e561-8864-49e9-91d2-92e62f8205f5" (UID: "e7c3e561-8864-49e9-91d2-92e62f8205f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:57:33.875195 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:33.875112    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3e561-8864-49e9-91d2-92e62f8205f5-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:57:34.593878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:34.593794    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p" event={"ID":"e7c3e561-8864-49e9-91d2-92e62f8205f5","Type":"ContainerDied","Data":"5b33933ae19eb60476c253074f18d714aaa2716f19db23ff458de9c0e9090afa"}
Jul 17 07:57:34.593878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:34.593822    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"
Jul 17 07:57:34.593878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:34.593857    2584 scope.go:117] "RemoveContainer" containerID="ca1e2d9595e446aa8e095da1ad361bf2697f9e88d1da7495f2967a56c913c15d"
Jul 17 07:57:34.615528 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:34.615502    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:34.621822 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:34.621799    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-22d5p"]
Jul 17 07:57:36.290323 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:36.290283    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" path="/var/lib/kubelet/pods/e7c3e561-8864-49e9-91d2-92e62f8205f5/volumes"
Jul 17 07:57:38.284901 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:38.284868    2584 scope.go:117] "RemoveContainer" containerID="e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334"
Jul 17 07:57:38.614095 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:38.613998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerStarted","Data":"97f2cc7964b959e93e94184a53a1f12b378443e3742cbe07936116a7ad790cec"}
Jul 17 07:57:38.637378 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:38.637306    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" podStartSLOduration=15.637290743 podStartE2EDuration="15.637290743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:57:38.637110257 +0000 UTC m=+2754.890533069" watchObservedRunningTime="2026-07-17 07:57:38.637290743 +0000 UTC m=+2754.890713556"
Jul 17 07:57:39.265387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265310    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:57:39.265710 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265694    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.265776 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265712    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.265776 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265729    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.265776 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265735    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.265875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265804    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.265875 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.265814    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:39.268696 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.268680    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:57:39.277527 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.277501    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:57:39.426293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.426249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j\" (UID: \"ac778275-6761-4fdc-a18e-3e080eed272a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:57:39.527400 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.527279    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j\" (UID: \"ac778275-6761-4fdc-a18e-3e080eed272a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:57:39.527962 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.527941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j\" (UID: \"ac778275-6761-4fdc-a18e-3e080eed272a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:57:39.579042 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.579004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:57:39.622688 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.622659    2584 generic.go:356] "Generic (PLEG): container finished" podID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerID="97f2cc7964b959e93e94184a53a1f12b378443e3742cbe07936116a7ad790cec" exitCode=6
Jul 17 07:57:39.622858 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.622734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerDied","Data":"97f2cc7964b959e93e94184a53a1f12b378443e3742cbe07936116a7ad790cec"}
Jul 17 07:57:39.622858 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.622809    2584 scope.go:117] "RemoveContainer" containerID="e5ea08a7d72a213c9057c53fa3d060981b286ba36064e9d5c8e764f12b9e3334"
Jul 17 07:57:39.623131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.623115    2584 scope.go:117] "RemoveContainer" containerID="97f2cc7964b959e93e94184a53a1f12b378443e3742cbe07936116a7ad790cec"
Jul 17 07:57:39.623362 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:39.623327    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc_odh-ai-gateway-infra(38a5de6f-04f1-4dff-81fb-839b60f617eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb"
Jul 17 07:57:39.649588 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.649559    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:39.715990 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:39.715966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:57:39.717855 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:57:39.717826    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac778275_6761_4fdc_a18e_3e080eed272a.slice/crio-3a2e1109c1fa6cb3f45cfb8cec2d7194004820c6a085476a7365b9f9efcf75c8 WatchSource:0}: Error finding container 3a2e1109c1fa6cb3f45cfb8cec2d7194004820c6a085476a7365b9f9efcf75c8: Status 404 returned error can't find the container with id 3a2e1109c1fa6cb3f45cfb8cec2d7194004820c6a085476a7365b9f9efcf75c8
Jul 17 07:57:40.628554 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.628467    2584 generic.go:356] "Generic (PLEG): container finished" podID="ac778275-6761-4fdc-a18e-3e080eed272a" containerID="777898fedc10c9492ab947c47f05338fadde4bca0d9a93c06606b3c85de5bf27" exitCode=6
Jul 17 07:57:40.628972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.628551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerDied","Data":"777898fedc10c9492ab947c47f05338fadde4bca0d9a93c06606b3c85de5bf27"}
Jul 17 07:57:40.628972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.628606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerStarted","Data":"3a2e1109c1fa6cb3f45cfb8cec2d7194004820c6a085476a7365b9f9efcf75c8"}
Jul 17 07:57:40.628972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.628743    2584 scope.go:117] "RemoveContainer" containerID="777898fedc10c9492ab947c47f05338fadde4bca0d9a93c06606b3c85de5bf27"
Jul 17 07:57:40.768022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.767998    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:40.838789 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.838762    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca\") pod \"38a5de6f-04f1-4dff-81fb-839b60f617eb\" (UID: \"38a5de6f-04f1-4dff-81fb-839b60f617eb\") "
Jul 17 07:57:40.839085 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.839064    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38a5de6f-04f1-4dff-81fb-839b60f617eb" (UID: "38a5de6f-04f1-4dff-81fb-839b60f617eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:57:40.939529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:40.939503    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a5de6f-04f1-4dff-81fb-839b60f617eb-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:57:41.637097 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.637013    2584 generic.go:356] "Generic (PLEG): container finished" podID="ac778275-6761-4fdc-a18e-3e080eed272a" containerID="3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3" exitCode=6
Jul 17 07:57:41.637568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.637097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerDied","Data":"3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3"}
Jul 17 07:57:41.637568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.637170    2584 scope.go:117] "RemoveContainer" containerID="777898fedc10c9492ab947c47f05338fadde4bca0d9a93c06606b3c85de5bf27"
Jul 17 07:57:41.637568 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.637264    2584 scope.go:117] "RemoveContainer" containerID="3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3"
Jul 17 07:57:41.637568 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:41.637512    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j_odh-ai-gateway-infra(ac778275-6761-4fdc-a18e-3e080eed272a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" podUID="ac778275-6761-4fdc-a18e-3e080eed272a"
Jul 17 07:57:41.638323 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.638309    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc" event={"ID":"38a5de6f-04f1-4dff-81fb-839b60f617eb","Type":"ContainerDied","Data":"bd33547351f60761a75f94614842db5ad81e8b5d09e7c11aea4439681f1b643f"}
Jul 17 07:57:41.638387 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.638325    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"
Jul 17 07:57:41.647714 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.647698    2584 scope.go:117] "RemoveContainer" containerID="97f2cc7964b959e93e94184a53a1f12b378443e3742cbe07936116a7ad790cec"
Jul 17 07:57:41.684138 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.684107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:41.690314 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:41.690289    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-vp5mc"]
Jul 17 07:57:42.290856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:42.290824    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" path="/var/lib/kubelet/pods/38a5de6f-04f1-4dff-81fb-839b60f617eb/volumes"
Jul 17 07:57:42.644675 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:42.644582    2584 scope.go:117] "RemoveContainer" containerID="3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3"
Jul 17 07:57:42.645140 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:42.644810    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j_odh-ai-gateway-infra(ac778275-6761-4fdc-a18e-3e080eed272a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" podUID="ac778275-6761-4fdc-a18e-3e080eed272a"
Jul 17 07:57:46.030193 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:46.030155    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:52.044530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044442    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044788    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044799    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044815    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044820    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044837    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044842    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044890    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.044898 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c3e561-8864-49e9-91d2-92e62f8205f5" containerName="revoke-keys"
Jul 17 07:57:52.045181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.044904    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:52.049277 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.049256    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:57:52.058087 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.058054    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:57:52.139797 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.139757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t\" (UID: \"e69875cf-81be-46ed-badc-74008eb799bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:57:52.241019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.240982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t\" (UID: \"e69875cf-81be-46ed-badc-74008eb799bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:57:52.241598 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.241579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t\" (UID: \"e69875cf-81be-46ed-badc-74008eb799bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:57:52.360415 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.360313    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:57:52.493411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.493384    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:57:52.495628 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:57:52.495594    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69875cf_81be_46ed_badc_74008eb799bb.slice/crio-93f5b6b8039e26c8aa44c41db5006c3b947191bca4ae4f2dbea165235783c012 WatchSource:0}: Error finding container 93f5b6b8039e26c8aa44c41db5006c3b947191bca4ae4f2dbea165235783c012: Status 404 returned error can't find the container with id 93f5b6b8039e26c8aa44c41db5006c3b947191bca4ae4f2dbea165235783c012
Jul 17 07:57:52.688900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:52.688816    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerStarted","Data":"93f5b6b8039e26c8aa44c41db5006c3b947191bca4ae4f2dbea165235783c012"}
Jul 17 07:57:53.509651 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:53.509566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:53.695280 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:53.695245    2584 generic.go:356] "Generic (PLEG): container finished" podID="e69875cf-81be-46ed-badc-74008eb799bb" containerID="5fc4b167a15e744d6131b9754b0246c4e28ef8670e698d85319e5e8af1314bf8" exitCode=6
Jul 17 07:57:53.695483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:53.695332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerDied","Data":"5fc4b167a15e744d6131b9754b0246c4e28ef8670e698d85319e5e8af1314bf8"}
Jul 17 07:57:53.695543 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:53.695516    2584 scope.go:117] "RemoveContainer" containerID="5fc4b167a15e744d6131b9754b0246c4e28ef8670e698d85319e5e8af1314bf8"
Jul 17 07:57:54.476037 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.476005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:54.577753 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.577666    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"]
Jul 17 07:57:54.578192 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.578138    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:54.578192 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.578155    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:54.578314 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.578265    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a5de6f-04f1-4dff-81fb-839b60f617eb" containerName="revoke-keys"
Jul 17 07:57:54.581130 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.581108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:54.588129 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.588105    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"]
Jul 17 07:57:54.659050 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.659013    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw\" (UID: \"bdc40031-d917-49a8-9ee8-f4c83090ce0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:54.701225 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.701192    2584 generic.go:356] "Generic (PLEG): container finished" podID="e69875cf-81be-46ed-badc-74008eb799bb" containerID="0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199" exitCode=6
Jul 17 07:57:54.701425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.701286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerDied","Data":"0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199"}
Jul 17 07:57:54.701425 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.701355    2584 scope.go:117] "RemoveContainer" containerID="5fc4b167a15e744d6131b9754b0246c4e28ef8670e698d85319e5e8af1314bf8"
Jul 17 07:57:54.701556 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.701527    2584 scope.go:117] "RemoveContainer" containerID="0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199"
Jul 17 07:57:54.701764 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:54.701742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t_odh-ai-gateway-infra(e69875cf-81be-46ed-badc-74008eb799bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" podUID="e69875cf-81be-46ed-badc-74008eb799bb"
Jul 17 07:57:54.760446 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.760410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw\" (UID: \"bdc40031-d917-49a8-9ee8-f4c83090ce0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:54.761064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.761046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw\" (UID: \"bdc40031-d917-49a8-9ee8-f4c83090ce0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:54.892016 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:54.891938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:55.022951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.022921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"]
Jul 17 07:57:55.025189 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:57:55.025162    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc40031_d917_49a8_9ee8_f4c83090ce0b.slice/crio-af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d WatchSource:0}: Error finding container af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d: Status 404 returned error can't find the container with id af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d
Jul 17 07:57:55.104362 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.104312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:55.207890 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.207792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:57:55.707175 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.707132    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw" event={"ID":"bdc40031-d917-49a8-9ee8-f4c83090ce0b","Type":"ContainerStarted","Data":"6c457f60eb6387094cb0b84e748883b9ed6f878db71f07588ee46d9fed4f47cf"}
Jul 17 07:57:55.707667 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.707192    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw" event={"ID":"bdc40031-d917-49a8-9ee8-f4c83090ce0b","Type":"ContainerStarted","Data":"af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d"}
Jul 17 07:57:55.708776 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.708758    2584 scope.go:117] "RemoveContainer" containerID="0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199"
Jul 17 07:57:55.708922 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:55.708904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t_odh-ai-gateway-infra(e69875cf-81be-46ed-badc-74008eb799bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" podUID="e69875cf-81be-46ed-badc-74008eb799bb"
Jul 17 07:57:55.728951 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:55.728902    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw" podStartSLOduration=1.7288861 podStartE2EDuration="1.7288861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:57:55.728159629 +0000 UTC m=+2771.981582441" watchObservedRunningTime="2026-07-17 07:57:55.7288861 +0000 UTC m=+2771.982308926"
Jul 17 07:57:56.714266 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:56.714235    2584 generic.go:356] "Generic (PLEG): container finished" podID="bdc40031-d917-49a8-9ee8-f4c83090ce0b" containerID="6c457f60eb6387094cb0b84e748883b9ed6f878db71f07588ee46d9fed4f47cf" exitCode=0
Jul 17 07:57:56.714648 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:56.714327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw" event={"ID":"bdc40031-d917-49a8-9ee8-f4c83090ce0b","Type":"ContainerDied","Data":"6c457f60eb6387094cb0b84e748883b9ed6f878db71f07588ee46d9fed4f47cf"}
Jul 17 07:57:57.848689 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:57.848666    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:57.884990 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:57.884961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca\") pod \"bdc40031-d917-49a8-9ee8-f4c83090ce0b\" (UID: \"bdc40031-d917-49a8-9ee8-f4c83090ce0b\") "
Jul 17 07:57:57.885268 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:57.885249    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdc40031-d917-49a8-9ee8-f4c83090ce0b" (UID: "bdc40031-d917-49a8-9ee8-f4c83090ce0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:57:57.986406 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:57.986317    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc40031-d917-49a8-9ee8-f4c83090ce0b-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:57:58.284953 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.284927    2584 scope.go:117] "RemoveContainer" containerID="3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3"
Jul 17 07:57:58.725108 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.725029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw"
Jul 17 07:57:58.725269 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.725028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw" event={"ID":"bdc40031-d917-49a8-9ee8-f4c83090ce0b","Type":"ContainerDied","Data":"af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d"}
Jul 17 07:57:58.725269 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.725145    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9a191d333e16fcb578e70650cfeee6369c8a9a9cc0bd60e145a742199e297d"
Jul 17 07:57:58.726894 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.726864    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerStarted","Data":"9ce112eecf6cc8788126945dcf373ce7d98608939e525d83626628dae41dee4f"}
Jul 17 07:57:58.746036 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:58.745990    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" podStartSLOduration=19.745978343 podStartE2EDuration="19.745978343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:57:58.745134115 +0000 UTC m=+2774.998556926" watchObservedRunningTime="2026-07-17 07:57:58.745978343 +0000 UTC m=+2774.999401155"
Jul 17 07:57:59.326394 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.326359    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:57:59.733005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.732914    2584 generic.go:356] "Generic (PLEG): container finished" podID="ac778275-6761-4fdc-a18e-3e080eed272a" containerID="9ce112eecf6cc8788126945dcf373ce7d98608939e525d83626628dae41dee4f" exitCode=6
Jul 17 07:57:59.733005 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.732981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerDied","Data":"9ce112eecf6cc8788126945dcf373ce7d98608939e525d83626628dae41dee4f"}
Jul 17 07:57:59.733245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.733041    2584 scope.go:117] "RemoveContainer" containerID="3353bc4c4985c6c8e74f7f34c48c2c37c2f660b8e3d28a91b9247c90abf314f3"
Jul 17 07:57:59.733382 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.733368    2584 scope.go:117] "RemoveContainer" containerID="9ce112eecf6cc8788126945dcf373ce7d98608939e525d83626628dae41dee4f"
Jul 17 07:57:59.733603 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:57:59.733573    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j_odh-ai-gateway-infra(ac778275-6761-4fdc-a18e-3e080eed272a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" podUID="ac778275-6761-4fdc-a18e-3e080eed272a"
Jul 17 07:57:59.784480 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:57:59.784438    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:58:00.874269 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:00.874244    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:58:00.911593 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:00.911565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca\") pod \"ac778275-6761-4fdc-a18e-3e080eed272a\" (UID: \"ac778275-6761-4fdc-a18e-3e080eed272a\") "
Jul 17 07:58:00.911915 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:00.911895    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac778275-6761-4fdc-a18e-3e080eed272a" (UID: "ac778275-6761-4fdc-a18e-3e080eed272a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:01.013126 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.013055    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac778275-6761-4fdc-a18e-3e080eed272a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:01.745377 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.745326    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"
Jul 17 07:58:01.745559 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.745325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j" event={"ID":"ac778275-6761-4fdc-a18e-3e080eed272a","Type":"ContainerDied","Data":"3a2e1109c1fa6cb3f45cfb8cec2d7194004820c6a085476a7365b9f9efcf75c8"}
Jul 17 07:58:01.745559 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.745491    2584 scope.go:117] "RemoveContainer" containerID="9ce112eecf6cc8788126945dcf373ce7d98608939e525d83626628dae41dee4f"
Jul 17 07:58:01.768471 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.768444    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:58:01.774092 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:01.774071    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gxh4j"]
Jul 17 07:58:02.289900 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:02.289867    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" path="/var/lib/kubelet/pods/ac778275-6761-4fdc-a18e-3e080eed272a/volumes"
Jul 17 07:58:05.222389 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.222329    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:58:05.222876 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.222601    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" containerName="maas-api" containerID="cri-o://a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97" gracePeriod=30
Jul 17 07:58:05.260512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.260461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:05.347062 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.347026    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": dial tcp 10.133.0.84:8443: connect: connection refused"
Jul 17 07:58:05.479377 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.479295    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:58:05.553207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.553173    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn6q5\" (UniqueName: \"kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5\") pod \"84d021d0-1e61-4d84-82da-d4766d051b01\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") "
Jul 17 07:58:05.553372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.553241    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca\") pod \"84d021d0-1e61-4d84-82da-d4766d051b01\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") "
Jul 17 07:58:05.553372 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.553274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") pod \"84d021d0-1e61-4d84-82da-d4766d051b01\" (UID: \"84d021d0-1e61-4d84-82da-d4766d051b01\") "
Jul 17 07:58:05.553651 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.553616    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84d021d0-1e61-4d84-82da-d4766d051b01" (UID: "84d021d0-1e61-4d84-82da-d4766d051b01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:05.555270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.555240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84d021d0-1e61-4d84-82da-d4766d051b01" (UID: "84d021d0-1e61-4d84-82da-d4766d051b01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:58:05.555270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.555246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5" (OuterVolumeSpecName: "kube-api-access-rn6q5") pod "84d021d0-1e61-4d84-82da-d4766d051b01" (UID: "84d021d0-1e61-4d84-82da-d4766d051b01"). InnerVolumeSpecName "kube-api-access-rn6q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:58:05.654079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.654054    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84d021d0-1e61-4d84-82da-d4766d051b01-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:05.654079 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.654077    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84d021d0-1e61-4d84-82da-d4766d051b01-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:05.654237 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.654087    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6q5\" (UniqueName: \"kubernetes.io/projected/84d021d0-1e61-4d84-82da-d4766d051b01-kube-api-access-rn6q5\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:05.765917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.765836    2584 generic.go:356] "Generic (PLEG): container finished" podID="84d021d0-1e61-4d84-82da-d4766d051b01" containerID="a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97" exitCode=0
Jul 17 07:58:05.765917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.765878    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"
Jul 17 07:58:05.765917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.765892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" event={"ID":"84d021d0-1e61-4d84-82da-d4766d051b01","Type":"ContainerDied","Data":"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"}
Jul 17 07:58:05.766151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.765933    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr" event={"ID":"84d021d0-1e61-4d84-82da-d4766d051b01","Type":"ContainerDied","Data":"22cf48e0ddf9e6f0fb1366db52c8bc6c339159ca48826bc366b06d8c1e886c31"}
Jul 17 07:58:05.766151 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.765954    2584 scope.go:117] "RemoveContainer" containerID="a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"
Jul 17 07:58:05.776563 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.776541    2584 scope.go:117] "RemoveContainer" containerID="a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"
Jul 17 07:58:05.776831 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:05.776807    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97\": container with ID starting with a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97 not found: ID does not exist" containerID="a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"
Jul 17 07:58:05.776931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.776837    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97"} err="failed to get container status \"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97\": rpc error: code = NotFound desc = could not find container \"a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97\": container with ID starting with a33f65afee82eedf97337d0b43290d146660a7186832c27a65edb6133681ed97 not found: ID does not exist"
Jul 17 07:58:05.794598 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.794573    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:58:05.807740 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.807717    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f268b-6464f56bc9-wc2kr"]
Jul 17 07:58:05.892400 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892369    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:05.892702 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892691    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" containerName="maas-api"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892704    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" containerName="maas-api"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892719    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc40031-d917-49a8-9ee8-f4c83090ce0b" containerName="revoke-keys"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892725    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc40031-d917-49a8-9ee8-f4c83090ce0b" containerName="revoke-keys"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892735    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892741    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892746    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892752    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892808    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" containerName="maas-api"
Jul 17 07:58:05.892926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892815    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892822    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:05.892926 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.892828    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc40031-d917-49a8-9ee8-f4c83090ce0b" containerName="revoke-keys"
Jul 17 07:58:05.897066 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.897050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:05.905248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.905221    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:05.955991 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:05.955966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89\" (UID: \"0a627f97-cf7c-437b-82c8-3417d24c7a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:06.056878 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.056855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89\" (UID: \"0a627f97-cf7c-437b-82c8-3417d24c7a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:06.057451 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.057433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89\" (UID: \"0a627f97-cf7c-437b-82c8-3417d24c7a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:06.208756 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.208718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:06.291280 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.291246    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d021d0-1e61-4d84-82da-d4766d051b01" path="/var/lib/kubelet/pods/84d021d0-1e61-4d84-82da-d4766d051b01/volumes"
Jul 17 07:58:06.344374 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.344334    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:06.346773 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:58:06.346745    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a627f97_cf7c_437b_82c8_3417d24c7a1a.slice/crio-999145fbf954f6da9fb366d8a5bf4bd7101320ce7ebeb0f5055dc6c2055d46e3 WatchSource:0}: Error finding container 999145fbf954f6da9fb366d8a5bf4bd7101320ce7ebeb0f5055dc6c2055d46e3: Status 404 returned error can't find the container with id 999145fbf954f6da9fb366d8a5bf4bd7101320ce7ebeb0f5055dc6c2055d46e3
Jul 17 07:58:06.772534 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.772459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerStarted","Data":"d86e6992fdcc1db1d032568e986b8a7ce1348ff5071889622e12a465ef9a2ce0"}
Jul 17 07:58:06.772651 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.772558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerStarted","Data":"999145fbf954f6da9fb366d8a5bf4bd7101320ce7ebeb0f5055dc6c2055d46e3"}
Jul 17 07:58:06.789302 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:06.789255    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" podStartSLOduration=1.7892427629999998 podStartE2EDuration="1.789242763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:58:06.788757416 +0000 UTC m=+2783.042180228" watchObservedRunningTime="2026-07-17 07:58:06.789242763 +0000 UTC m=+2783.042665595"
Jul 17 07:58:07.777917 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:07.777880    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerID="d86e6992fdcc1db1d032568e986b8a7ce1348ff5071889622e12a465ef9a2ce0" exitCode=6
Jul 17 07:58:07.778322 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:07.777968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerDied","Data":"d86e6992fdcc1db1d032568e986b8a7ce1348ff5071889622e12a465ef9a2ce0"}
Jul 17 07:58:07.778322 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:07.778129    2584 scope.go:117] "RemoveContainer" containerID="d86e6992fdcc1db1d032568e986b8a7ce1348ff5071889622e12a465ef9a2ce0"
Jul 17 07:58:08.784976 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:08.784944    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerID="6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b" exitCode=6
Jul 17 07:58:08.785472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:08.785036    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerDied","Data":"6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b"}
Jul 17 07:58:08.785472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:08.785085    2584 scope.go:117] "RemoveContainer" containerID="d86e6992fdcc1db1d032568e986b8a7ce1348ff5071889622e12a465ef9a2ce0"
Jul 17 07:58:08.785472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:08.785328    2584 scope.go:117] "RemoveContainer" containerID="6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b"
Jul 17 07:58:08.785603 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:08.785533    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89_odh-ai-gateway-infra(0a627f97-cf7c-437b-82c8-3417d24c7a1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a"
Jul 17 07:58:09.689718 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:09.689688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:09.790594 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:09.790567    2584 scope.go:117] "RemoveContainer" containerID="6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b"
Jul 17 07:58:09.790927 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:09.790737    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89_odh-ai-gateway-infra(0a627f97-cf7c-437b-82c8-3417d24c7a1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a"
Jul 17 07:58:10.135474 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:10.135447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:58:10.285052 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:10.285020    2584 scope.go:117] "RemoveContainer" containerID="0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199"
Jul 17 07:58:10.797841 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:10.797802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerStarted","Data":"e92aadeb94139a81975d2a06e7fccffafd673ffdfcf0c0625bb02f3473b8308c"}
Jul 17 07:58:10.822608 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:10.822563    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" podStartSLOduration=18.822547422 podStartE2EDuration="18.822547422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:58:10.820416375 +0000 UTC m=+2787.073839187" watchObservedRunningTime="2026-07-17 07:58:10.822547422 +0000 UTC m=+2787.075970235"
Jul 17 07:58:11.804128 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:11.804093    2584 generic.go:356] "Generic (PLEG): container finished" podID="e69875cf-81be-46ed-badc-74008eb799bb" containerID="e92aadeb94139a81975d2a06e7fccffafd673ffdfcf0c0625bb02f3473b8308c" exitCode=6
Jul 17 07:58:11.804635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:11.804165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerDied","Data":"e92aadeb94139a81975d2a06e7fccffafd673ffdfcf0c0625bb02f3473b8308c"}
Jul 17 07:58:11.804635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:11.804224    2584 scope.go:117] "RemoveContainer" containerID="0148d5931058c2e63d8014b12b84ac384448254599c7691fa70a779072978199"
Jul 17 07:58:11.804635 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:11.804532    2584 scope.go:117] "RemoveContainer" containerID="e92aadeb94139a81975d2a06e7fccffafd673ffdfcf0c0625bb02f3473b8308c"
Jul 17 07:58:11.804794 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:11.804765    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t_odh-ai-gateway-infra(e69875cf-81be-46ed-badc-74008eb799bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" podUID="e69875cf-81be-46ed-badc-74008eb799bb"
Jul 17 07:58:11.836265 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:11.836227    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:58:12.963993 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:12.963966    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:58:13.015056 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.015021    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca\") pod \"e69875cf-81be-46ed-badc-74008eb799bb\" (UID: \"e69875cf-81be-46ed-badc-74008eb799bb\") "
Jul 17 07:58:13.015447 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.015424    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e69875cf-81be-46ed-badc-74008eb799bb" (UID: "e69875cf-81be-46ed-badc-74008eb799bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:13.116070 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.115981    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69875cf-81be-46ed-badc-74008eb799bb-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:13.816271 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.816231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t" event={"ID":"e69875cf-81be-46ed-badc-74008eb799bb","Type":"ContainerDied","Data":"93f5b6b8039e26c8aa44c41db5006c3b947191bca4ae4f2dbea165235783c012"}
Jul 17 07:58:13.816573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.816292    2584 scope.go:117] "RemoveContainer" containerID="e92aadeb94139a81975d2a06e7fccffafd673ffdfcf0c0625bb02f3473b8308c"
Jul 17 07:58:13.816573 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.816248    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"
Jul 17 07:58:13.841371 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.841330    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:58:13.848921 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:13.848896    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6wh4t"]
Jul 17 07:58:14.289778 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:14.289744    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69875cf-81be-46ed-badc-74008eb799bb" path="/var/lib/kubelet/pods/e69875cf-81be-46ed-badc-74008eb799bb/volumes"
Jul 17 07:58:16.363581 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:16.363547    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:16.523842 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:16.523806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:20.464454 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.464422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:20.704759 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.704727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:20.826265 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826232    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"]
Jul 17 07:58:20.826578 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826564    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826580    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826596    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826602    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826609    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826615    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826626    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826637 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826631    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826682    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826689    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac778275-6761-4fdc-a18e-3e080eed272a" containerName="revoke-keys"
Jul 17 07:58:20.826849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826696    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.826849 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.826704    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69875cf-81be-46ed-badc-74008eb799bb" containerName="revoke-keys"
Jul 17 07:58:20.830996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.830977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:20.837523 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.837497    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"]
Jul 17 07:58:20.880672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.880645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64\" (UID: \"c9e64d70-3a41-495d-a36c-43d56784df99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:20.981389 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.981364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64\" (UID: \"c9e64d70-3a41-495d-a36c-43d56784df99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:20.981894 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:20.981876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64\" (UID: \"c9e64d70-3a41-495d-a36c-43d56784df99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:21.142544 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.142463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:21.272980 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.272956    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"]
Jul 17 07:58:21.275146 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:58:21.275112    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e64d70_3a41_495d_a36c_43d56784df99.slice/crio-152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01 WatchSource:0}: Error finding container 152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01: Status 404 returned error can't find the container with id 152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01
Jul 17 07:58:21.436509 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.436483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:21.533064 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.533033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:21.853264 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.853228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64" event={"ID":"c9e64d70-3a41-495d-a36c-43d56784df99","Type":"ContainerStarted","Data":"7aa023406eb935c0d8f74de88cfbd27b9d680ee6e20eaf4b02dcb09668dd5693"}
Jul 17 07:58:21.853464 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.853294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64" event={"ID":"c9e64d70-3a41-495d-a36c-43d56784df99","Type":"ContainerStarted","Data":"152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01"}
Jul 17 07:58:21.875381 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:21.875300    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64" podStartSLOduration=1.875281954 podStartE2EDuration="1.875281954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:58:21.874658823 +0000 UTC m=+2798.128081636" watchObservedRunningTime="2026-07-17 07:58:21.875281954 +0000 UTC m=+2798.128704768"
Jul 17 07:58:22.859181 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:22.859146    2584 generic.go:356] "Generic (PLEG): container finished" podID="c9e64d70-3a41-495d-a36c-43d56784df99" containerID="7aa023406eb935c0d8f74de88cfbd27b9d680ee6e20eaf4b02dcb09668dd5693" exitCode=0
Jul 17 07:58:22.859653 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:22.859231    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64" event={"ID":"c9e64d70-3a41-495d-a36c-43d56784df99","Type":"ContainerDied","Data":"7aa023406eb935c0d8f74de88cfbd27b9d680ee6e20eaf4b02dcb09668dd5693"}
Jul 17 07:58:23.992102 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:23.992074    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:24.109060 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.109023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca\") pod \"c9e64d70-3a41-495d-a36c-43d56784df99\" (UID: \"c9e64d70-3a41-495d-a36c-43d56784df99\") "
Jul 17 07:58:24.109429 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.109408    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9e64d70-3a41-495d-a36c-43d56784df99" (UID: "c9e64d70-3a41-495d-a36c-43d56784df99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:24.210388 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.210284    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e64d70-3a41-495d-a36c-43d56784df99-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:24.290243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.290208    2584 scope.go:117] "RemoveContainer" containerID="6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b"
Jul 17 07:58:24.869732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.869689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerStarted","Data":"2a930fefd3a3280173fa8bce3a4eb708f749ed5c70aeda0337bc9303cb5769bc"}
Jul 17 07:58:24.871014 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.870986    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64"
Jul 17 07:58:24.871154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.871016    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64" event={"ID":"c9e64d70-3a41-495d-a36c-43d56784df99","Type":"ContainerDied","Data":"152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01"}
Jul 17 07:58:24.871154 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:24.871050    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="152e530428035b3425da825b81d9a8cebe0ecc43e991016ab407695fb5ef4d01"
Jul 17 07:58:25.877233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:25.877200    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerID="2a930fefd3a3280173fa8bce3a4eb708f749ed5c70aeda0337bc9303cb5769bc" exitCode=6
Jul 17 07:58:25.877672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:25.877275    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerDied","Data":"2a930fefd3a3280173fa8bce3a4eb708f749ed5c70aeda0337bc9303cb5769bc"}
Jul 17 07:58:25.877672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:25.877334    2584 scope.go:117] "RemoveContainer" containerID="6b35f90254674a90ea930a30298000cf9be77b9dda01b8c0be72c0736b41444b"
Jul 17 07:58:25.877672 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:25.877629    2584 scope.go:117] "RemoveContainer" containerID="2a930fefd3a3280173fa8bce3a4eb708f749ed5c70aeda0337bc9303cb5769bc"
Jul 17 07:58:25.877893 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:25.877874    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89_odh-ai-gateway-infra(0a627f97-cf7c-437b-82c8-3417d24c7a1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a"
Jul 17 07:58:25.904928 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:25.904882    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:27.021218 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.021192    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:27.134856 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.134765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca\") pod \"0a627f97-cf7c-437b-82c8-3417d24c7a1a\" (UID: \"0a627f97-cf7c-437b-82c8-3417d24c7a1a\") "
Jul 17 07:58:27.135162 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.135142    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a627f97-cf7c-437b-82c8-3417d24c7a1a" (UID: "0a627f97-cf7c-437b-82c8-3417d24c7a1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:27.236114 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.236079    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a627f97-cf7c-437b-82c8-3417d24c7a1a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:27.888762 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.888732    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"
Jul 17 07:58:27.888762 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.888758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89" event={"ID":"0a627f97-cf7c-437b-82c8-3417d24c7a1a","Type":"ContainerDied","Data":"999145fbf954f6da9fb366d8a5bf4bd7101320ce7ebeb0f5055dc6c2055d46e3"}
Jul 17 07:58:27.888994 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.888809    2584 scope.go:117] "RemoveContainer" containerID="2a930fefd3a3280173fa8bce3a4eb708f749ed5c70aeda0337bc9303cb5769bc"
Jul 17 07:58:27.911453 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.911399    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:27.913074 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:27.913052    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-gtq89"]
Jul 17 07:58:28.290219 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:28.290189    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" path="/var/lib/kubelet/pods/0a627f97-cf7c-437b-82c8-3417d24c7a1a/volumes"
Jul 17 07:58:31.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.315731    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:58:31.316131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.315922    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" podUID="83080ca4-bf8f-472c-ab01-e9e382274db2" containerName="maas-api" containerID="cri-o://4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279" gracePeriod=30
Jul 17 07:58:31.343748 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.343725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:31.564967 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.564944    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:58:31.672867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.672774    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls\") pod \"83080ca4-bf8f-472c-ab01-e9e382274db2\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") "
Jul 17 07:58:31.672867 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.672833    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-746km\" (UniqueName: \"kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km\") pod \"83080ca4-bf8f-472c-ab01-e9e382274db2\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") "
Jul 17 07:58:31.673065 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.672887    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca\") pod \"83080ca4-bf8f-472c-ab01-e9e382274db2\" (UID: \"83080ca4-bf8f-472c-ab01-e9e382274db2\") "
Jul 17 07:58:31.673270 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.673240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83080ca4-bf8f-472c-ab01-e9e382274db2" (UID: "83080ca4-bf8f-472c-ab01-e9e382274db2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:31.674928 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.674910    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83080ca4-bf8f-472c-ab01-e9e382274db2" (UID: "83080ca4-bf8f-472c-ab01-e9e382274db2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:58:31.675019 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.674971    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km" (OuterVolumeSpecName: "kube-api-access-746km") pod "83080ca4-bf8f-472c-ab01-e9e382274db2" (UID: "83080ca4-bf8f-472c-ab01-e9e382274db2"). InnerVolumeSpecName "kube-api-access-746km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:58:31.774131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.774093    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746km\" (UniqueName: \"kubernetes.io/projected/83080ca4-bf8f-472c-ab01-e9e382274db2-kube-api-access-746km\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:31.774131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.774123    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83080ca4-bf8f-472c-ab01-e9e382274db2-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:31.774131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.774134    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83080ca4-bf8f-472c-ab01-e9e382274db2-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:31.913458 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.913423    2584 generic.go:356] "Generic (PLEG): container finished" podID="83080ca4-bf8f-472c-ab01-e9e382274db2" containerID="4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279" exitCode=0
Jul 17 07:58:31.913625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.913481    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"
Jul 17 07:58:31.913625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.913511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" event={"ID":"83080ca4-bf8f-472c-ab01-e9e382274db2","Type":"ContainerDied","Data":"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"}
Jul 17 07:58:31.913625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.913562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs" event={"ID":"83080ca4-bf8f-472c-ab01-e9e382274db2","Type":"ContainerDied","Data":"44ee743a8af3d7ad866098eda863fcc345ee537da7d3521c50cf0f085ebb3500"}
Jul 17 07:58:31.913625 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.913582    2584 scope.go:117] "RemoveContainer" containerID="4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"
Jul 17 07:58:31.923243 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.923224    2584 scope.go:117] "RemoveContainer" containerID="4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"
Jul 17 07:58:31.923565 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:31.923543    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279\": container with ID starting with 4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279 not found: ID does not exist" containerID="4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"
Jul 17 07:58:31.923631 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.923575    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279"} err="failed to get container status \"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279\": rpc error: code = NotFound desc = could not find container \"4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279\": container with ID starting with 4401718bc4242f47276a8e5260cb387ead02440b9daf28e6447661702372b279 not found: ID does not exist"
Jul 17 07:58:31.936494 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.936462    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:58:31.940633 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:31.940612    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cab91c-5bb96d96fb-q4hcs"]
Jul 17 07:58:32.290893 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:32.290862    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83080ca4-bf8f-472c-ab01-e9e382274db2" path="/var/lib/kubelet/pods/83080ca4-bf8f-472c-ab01-e9e382274db2/volumes"
Jul 17 07:58:34.631420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:34.631383    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:58:35.949611 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:35.949576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:39.315374 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315323    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315627    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315637    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315644    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e64d70-3a41-495d-a36c-43d56784df99" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315650    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e64d70-3a41-495d-a36c-43d56784df99" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315656    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315662    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315682    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83080ca4-bf8f-472c-ab01-e9e382274db2" containerName="maas-api"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315687    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="83080ca4-bf8f-472c-ab01-e9e382274db2" containerName="maas-api"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315735    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315743    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="83080ca4-bf8f-472c-ab01-e9e382274db2" containerName="maas-api"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315754    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e64d70-3a41-495d-a36c-43d56784df99" containerName="revoke-keys"
Jul 17 07:58:39.315764 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.315760    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:39.319816 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.319798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:39.322516 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.322478    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:58:39.322641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.322565    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:58:39.327664 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.327640    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:39.437233 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.437198    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg\" (UID: \"fe5be1db-2caa-48d8-99d1-715b19e578e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:39.537641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.537603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg\" (UID: \"fe5be1db-2caa-48d8-99d1-715b19e578e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:39.538207 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.538187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg\" (UID: \"fe5be1db-2caa-48d8-99d1-715b19e578e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:39.630831 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.630743    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:39.766646 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.766617    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:39.768728 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:58:39.768700    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5be1db_2caa_48d8_99d1_715b19e578e3.slice/crio-429e0ff3ebae317e1c34a92ccaf94b4c9881859bf79d69a5eafbb982ce525e54 WatchSource:0}: Error finding container 429e0ff3ebae317e1c34a92ccaf94b4c9881859bf79d69a5eafbb982ce525e54: Status 404 returned error can't find the container with id 429e0ff3ebae317e1c34a92ccaf94b4c9881859bf79d69a5eafbb982ce525e54
Jul 17 07:58:39.952177 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.952146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerStarted","Data":"7df55a782c06edfba2dd66c0d029e5073436507f8e983ed0ea9968a33d7a6320"}
Jul 17 07:58:39.952386 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.952204    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerStarted","Data":"429e0ff3ebae317e1c34a92ccaf94b4c9881859bf79d69a5eafbb982ce525e54"}
Jul 17 07:58:39.970420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:39.970374    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" podStartSLOduration=0.97035842 podStartE2EDuration="970.35842ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:58:39.968789477 +0000 UTC m=+2816.222212293" watchObservedRunningTime="2026-07-17 07:58:39.97035842 +0000 UTC m=+2816.223781230"
Jul 17 07:58:40.957812 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:40.957781    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerID="7df55a782c06edfba2dd66c0d029e5073436507f8e983ed0ea9968a33d7a6320" exitCode=6
Jul 17 07:58:40.957812 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:40.957816    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerDied","Data":"7df55a782c06edfba2dd66c0d029e5073436507f8e983ed0ea9968a33d7a6320"}
Jul 17 07:58:40.958244 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:40.958019    2584 scope.go:117] "RemoveContainer" containerID="7df55a782c06edfba2dd66c0d029e5073436507f8e983ed0ea9968a33d7a6320"
Jul 17 07:58:41.964032 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:41.963949    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerID="6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6" exitCode=6
Jul 17 07:58:41.964032 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:41.964002    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerDied","Data":"6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6"}
Jul 17 07:58:41.964530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:41.964062    2584 scope.go:117] "RemoveContainer" containerID="7df55a782c06edfba2dd66c0d029e5073436507f8e983ed0ea9968a33d7a6320"
Jul 17 07:58:41.964530 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:41.964320    2584 scope.go:117] "RemoveContainer" containerID="6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6"
Jul 17 07:58:41.964595 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:41.964555    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg_odh-ai-gateway-infra(fe5be1db-2caa-48d8-99d1-715b19e578e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3"
Jul 17 07:58:42.194272 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:42.194237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:42.293306 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:42.293270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:42.969773 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:42.969738    2584 scope.go:117] "RemoveContainer" containerID="6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6"
Jul 17 07:58:42.970163 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:42.969905    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg_odh-ai-gateway-infra(fe5be1db-2caa-48d8-99d1-715b19e578e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3"
Jul 17 07:58:46.703985 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:46.703931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:46.948175 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:46.948138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:47.311828 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:47.311791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:47.665363 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:47.665248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:47.758385 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:47.758323    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:50.527114 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.527080    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:50.783572 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.783486    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:58:50.783888 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.783869    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:50.783972 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.783890    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:50.784035 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.783973    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a627f97-cf7c-437b-82c8-3417d24c7a1a" containerName="revoke-keys"
Jul 17 07:58:50.788299 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.788278    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.791773 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.791754    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 07:58:50.797029 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.797011    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5cf58d-serving-cert\""
Jul 17 07:58:50.797144 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.797050    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 07:58:50.808961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.808942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:58:50.827483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.827460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.827577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.827493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.827577 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.827519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffjd\" (UniqueName: \"kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.862318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.862256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:50.928649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.928614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.928802 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.928654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.928802 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.928704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffjd\" (UniqueName: \"kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.928885 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:50.928795    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-serving-cert: secret "maas-api-e2e-api-a-5cf58d-serving-cert" not found
Jul 17 07:58:50.928885 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:50.928872    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls podName:5f30091f-8e20-478e-a4dc-71a7c53d1405 nodeName:}" failed. No retries permitted until 2026-07-17 07:58:51.428850205 +0000 UTC m=+2827.682273000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls") pod "maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" (UID: "5f30091f-8e20-478e-a4dc-71a7c53d1405") : secret "maas-api-e2e-api-a-5cf58d-serving-cert" not found
Jul 17 07:58:50.929406 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.929387    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.937623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.937601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffjd\" (UniqueName: \"kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:50.953895 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:50.953874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:51.358164 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.358118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:51.434214 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.434179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:51.436498 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.436477    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") pod \"maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:51.460543 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.460513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:51.559179 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.559146    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:51.698092 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.697991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:51.838981 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.838955    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:58:51.841508 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:58:51.841482    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f30091f_8e20_478e_a4dc_71a7c53d1405.slice/crio-cbc459f992e01842700443b1d3ec54cd31b879fd10f0f2ac31e0b98aeca810db WatchSource:0}: Error finding container cbc459f992e01842700443b1d3ec54cd31b879fd10f0f2ac31e0b98aeca810db: Status 404 returned error can't find the container with id cbc459f992e01842700443b1d3ec54cd31b879fd10f0f2ac31e0b98aeca810db
Jul 17 07:58:51.842996 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:51.842973    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 07:58:52.014371 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:52.014256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" event={"ID":"5f30091f-8e20-478e-a4dc-71a7c53d1405","Type":"ContainerStarted","Data":"cbc459f992e01842700443b1d3ec54cd31b879fd10f0f2ac31e0b98aeca810db"}
Jul 17 07:58:53.020096 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:53.020060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" event={"ID":"5f30091f-8e20-478e-a4dc-71a7c53d1405","Type":"ContainerStarted","Data":"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"}
Jul 17 07:58:53.020512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:53.020189    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:54.287131 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:54.287099    2584 scope.go:117] "RemoveContainer" containerID="6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6"
Jul 17 07:58:55.032459 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.032426    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerID="d539ca21263c7c0067fb03b5cf21d21d7ba7bfb10e5ae2d7dd2cd5a33bbd8fab" exitCode=6
Jul 17 07:58:55.032630 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.032495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerDied","Data":"d539ca21263c7c0067fb03b5cf21d21d7ba7bfb10e5ae2d7dd2cd5a33bbd8fab"}
Jul 17 07:58:55.032630 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.032554    2584 scope.go:117] "RemoveContainer" containerID="6c172c412b360ca28a2b81cb4538146ced5292afb412423e63acdcacf01520b6"
Jul 17 07:58:55.032941 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.032922    2584 scope.go:117] "RemoveContainer" containerID="d539ca21263c7c0067fb03b5cf21d21d7ba7bfb10e5ae2d7dd2cd5a33bbd8fab"
Jul 17 07:58:55.033185 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:55.033163    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg_odh-ai-gateway-infra(fe5be1db-2caa-48d8-99d1-715b19e578e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3"
Jul 17 07:58:55.051671 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.051621    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" podStartSLOduration=4.66520452 podStartE2EDuration="5.051607185s" totalImagesPullingTime="386.402665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:58:53.049394395 +0000 UTC m=+2829.302817206" watchObservedRunningTime="2026-07-17 07:58:55.051607185 +0000 UTC m=+2831.305029998"
Jul 17 07:58:55.061416 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.061389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:55.134754 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.134716    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:58:55.556809 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.556774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:55.929947 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:55.929865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:56.062735 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.062700    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:56.193786 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.193726    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:56.275240 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.275212    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca\") pod \"fe5be1db-2caa-48d8-99d1-715b19e578e3\" (UID: \"fe5be1db-2caa-48d8-99d1-715b19e578e3\") "
Jul 17 07:58:56.275657 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.275633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe5be1db-2caa-48d8-99d1-715b19e578e3" (UID: "fe5be1db-2caa-48d8-99d1-715b19e578e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:58:56.375988 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.375961    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5be1db-2caa-48d8-99d1-715b19e578e3-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:58:56.431235 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:56.431209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:57.044459 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:57.044427    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg" event={"ID":"fe5be1db-2caa-48d8-99d1-715b19e578e3","Type":"ContainerDied","Data":"429e0ff3ebae317e1c34a92ccaf94b4c9881859bf79d69a5eafbb982ce525e54"}
Jul 17 07:58:57.044901 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:57.044473    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"
Jul 17 07:58:57.044901 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:57.044490    2584 scope.go:117] "RemoveContainer" containerID="d539ca21263c7c0067fb03b5cf21d21d7ba7bfb10e5ae2d7dd2cd5a33bbd8fab"
Jul 17 07:58:57.061461 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:57.061437    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:57.067357 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:57.067311    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4dwlg"]
Jul 17 07:58:58.290210 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.290178    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" path="/var/lib/kubelet/pods/fe5be1db-2caa-48d8-99d1-715b19e578e3/volumes"
Jul 17 07:58:58.549452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.549374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:58.655761 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.655726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:58.661655 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.661632    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:58:58.661961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.661948    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.662008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.661965    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.662008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.661979    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.662008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.661984    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.662104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.662032    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.662104 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.662047    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:58:58.666221 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.666203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.677529 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.677511    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-38e02d-serving-cert\""
Jul 17 07:58:58.696328 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.696304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkrm\" (UniqueName: \"kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.696462 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.696374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.696462 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.696435    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.697255 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.697199    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:58:58.755292 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.755263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:58.797393 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.797360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.797569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.797457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.797569 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.797536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkrm\" (UniqueName: \"kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.797569 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:58.797535    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-serving-cert: secret "maas-api-e2e-api-b-38e02d-serving-cert" not found
Jul 17 07:58:58.797714 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:58:58.797613    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls podName:5dcc3d12-2a8c-4829-9010-32cf463f6bb9 nodeName:}" failed. No retries permitted until 2026-07-17 07:58:59.297589067 +0000 UTC m=+2835.551011861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls") pod "maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" (UID: "5dcc3d12-2a8c-4829-9010-32cf463f6bb9") : secret "maas-api-e2e-api-b-38e02d-serving-cert" not found
Jul 17 07:58:58.798084 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.798067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.807282 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.807217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkrm\" (UniqueName: \"kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:58.957868 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:58.957828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:59.031897 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.031866    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:58:59.271452 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.270027    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:59.301989 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.301948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:59.304303 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.304277    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") pod \"maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:59.370213 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.370175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:58:59.576313 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.576277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:58:59.709145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:58:59.709034    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:58:59.711761 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:58:59.711733    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcc3d12_2a8c_4829_9010_32cf463f6bb9.slice/crio-0d1b80760bd0979b463656748e79d90e40b85ef17e45136c9e176313bd6cfd5b WatchSource:0}: Error finding container 0d1b80760bd0979b463656748e79d90e40b85ef17e45136c9e176313bd6cfd5b: Status 404 returned error can't find the container with id 0d1b80760bd0979b463656748e79d90e40b85ef17e45136c9e176313bd6cfd5b
Jul 17 07:59:00.062583 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:00.062550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" event={"ID":"5dcc3d12-2a8c-4829-9010-32cf463f6bb9","Type":"ContainerStarted","Data":"0d1b80760bd0979b463656748e79d90e40b85ef17e45136c9e176313bd6cfd5b"}
Jul 17 07:59:01.067963 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:01.067927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" event={"ID":"5dcc3d12-2a8c-4829-9010-32cf463f6bb9","Type":"ContainerStarted","Data":"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"}
Jul 17 07:59:01.068418 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:01.067987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:59:01.087653 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:01.087605    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" podStartSLOduration=2.747859746 podStartE2EDuration="3.087590835s" totalImagesPullingTime="339.731089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:59:01.086846607 +0000 UTC m=+2837.340269462" watchObservedRunningTime="2026-07-17 07:59:01.087590835 +0000 UTC m=+2837.341013667"
Jul 17 07:59:01.169355 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:01.169306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:03.097482 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:03.097445    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:05.964723 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.964688    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:05.965125 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.965035    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:59:05.965125 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.965046    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:59:05.965196 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.965129    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5be1db-2caa-48d8-99d1-715b19e578e3" containerName="revoke-keys"
Jul 17 07:59:05.968121 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.968104    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:05.970873 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.970848    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:59:05.976768 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:05.976747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:06.058224 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.058195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf\" (UID: \"8148249e-9e3d-4081-a174-f93265483674\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:06.158886 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.158845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf\" (UID: \"8148249e-9e3d-4081-a174-f93265483674\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:06.159483 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.159463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf\" (UID: \"8148249e-9e3d-4081-a174-f93265483674\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:06.279253 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.279180    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:06.407806 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.407777    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:06.409597 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:59:06.409572    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8148249e_9e3d_4081_a174_f93265483674.slice/crio-bd87f7050d58d14f596487f1db4571ca7ba35b52f8074c6a0c23327d3dc8e2f7 WatchSource:0}: Error finding container bd87f7050d58d14f596487f1db4571ca7ba35b52f8074c6a0c23327d3dc8e2f7: Status 404 returned error can't find the container with id bd87f7050d58d14f596487f1db4571ca7ba35b52f8074c6a0c23327d3dc8e2f7
Jul 17 07:59:06.944145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:06.944061    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:59:07.077720 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:07.077694    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:59:07.095747 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:07.095714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerStarted","Data":"7860001a1441adca8185547d1f17cbec03df05f8675cbf99b058b2227a7027ba"}
Jul 17 07:59:07.095887 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:07.095817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerStarted","Data":"bd87f7050d58d14f596487f1db4571ca7ba35b52f8074c6a0c23327d3dc8e2f7"}
Jul 17 07:59:07.116248 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:07.116204    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" podStartSLOduration=2.116190021 podStartE2EDuration="2.116190021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:59:07.114634854 +0000 UTC m=+2843.368057666" watchObservedRunningTime="2026-07-17 07:59:07.116190021 +0000 UTC m=+2843.369612833"
Jul 17 07:59:08.101475 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:08.101442    2584 generic.go:356] "Generic (PLEG): container finished" podID="8148249e-9e3d-4081-a174-f93265483674" containerID="7860001a1441adca8185547d1f17cbec03df05f8675cbf99b058b2227a7027ba" exitCode=6
Jul 17 07:59:08.101844 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:08.101504    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerDied","Data":"7860001a1441adca8185547d1f17cbec03df05f8675cbf99b058b2227a7027ba"}
Jul 17 07:59:08.101844 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:08.101764    2584 scope.go:117] "RemoveContainer" containerID="7860001a1441adca8185547d1f17cbec03df05f8675cbf99b058b2227a7027ba"
Jul 17 07:59:09.107630 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:09.107595    2584 generic.go:356] "Generic (PLEG): container finished" podID="8148249e-9e3d-4081-a174-f93265483674" containerID="6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526" exitCode=6
Jul 17 07:59:09.108073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:09.107684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerDied","Data":"6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526"}
Jul 17 07:59:09.108073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:09.107742    2584 scope.go:117] "RemoveContainer" containerID="7860001a1441adca8185547d1f17cbec03df05f8675cbf99b058b2227a7027ba"
Jul 17 07:59:09.108073 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:09.107874    2584 scope.go:117] "RemoveContainer" containerID="6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526"
Jul 17 07:59:09.108205 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:09.108093    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf_odh-ai-gateway-infra(8148249e-9e3d-4081-a174-f93265483674)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" podUID="8148249e-9e3d-4081-a174-f93265483674"
Jul 17 07:59:10.113183 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:10.113158    2584 scope.go:117] "RemoveContainer" containerID="6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526"
Jul 17 07:59:10.113589 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:10.113318    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf_odh-ai-gateway-infra(8148249e-9e3d-4081-a174-f93265483674)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" podUID="8148249e-9e3d-4081-a174-f93265483674"
Jul 17 07:59:11.475658 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.475612    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"]
Jul 17 07:59:11.479978 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.479960    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:11.494918 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.494889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"]
Jul 17 07:59:11.605656 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.605617    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8\" (UID: \"7240de31-df71-4ec4-9dc2-5906b6b1bfc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:11.706834 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.706798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8\" (UID: \"7240de31-df71-4ec4-9dc2-5906b6b1bfc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:11.707424 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.707407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8\" (UID: \"7240de31-df71-4ec4-9dc2-5906b6b1bfc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:11.788880 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.788857    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:11.919695 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:11.919669    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"]
Jul 17 07:59:11.921176 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:59:11.921144    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240de31_df71_4ec4_9dc2_5906b6b1bfc8.slice/crio-da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51 WatchSource:0}: Error finding container da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51: Status 404 returned error can't find the container with id da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51
Jul 17 07:59:12.122913 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:12.122835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8" event={"ID":"7240de31-df71-4ec4-9dc2-5906b6b1bfc8","Type":"ContainerStarted","Data":"da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51"}
Jul 17 07:59:12.126944 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:12.126923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:12.225587 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:12.225557    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:13.128721 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:13.128691    2584 generic.go:356] "Generic (PLEG): container finished" podID="7240de31-df71-4ec4-9dc2-5906b6b1bfc8" containerID="d8048da52cfed74e0fc414509fc69ebddb122946c4a6d2a303a4df43515430e7" exitCode=0
Jul 17 07:59:13.129095 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:13.128759    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8" event={"ID":"7240de31-df71-4ec4-9dc2-5906b6b1bfc8","Type":"ContainerDied","Data":"d8048da52cfed74e0fc414509fc69ebddb122946c4a6d2a303a4df43515430e7"}
Jul 17 07:59:14.264054 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:14.264029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:14.330298 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:14.330273    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca\") pod \"7240de31-df71-4ec4-9dc2-5906b6b1bfc8\" (UID: \"7240de31-df71-4ec4-9dc2-5906b6b1bfc8\") "
Jul 17 07:59:14.330697 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:14.330667    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7240de31-df71-4ec4-9dc2-5906b6b1bfc8" (UID: "7240de31-df71-4ec4-9dc2-5906b6b1bfc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:14.431318 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:14.431237    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240de31-df71-4ec4-9dc2-5906b6b1bfc8-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:15.140173 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:15.140141    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8"
Jul 17 07:59:15.140173 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:15.140162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8" event={"ID":"7240de31-df71-4ec4-9dc2-5906b6b1bfc8","Type":"ContainerDied","Data":"da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51"}
Jul 17 07:59:15.140396 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:15.140210    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da509b9f610c0d9058882388bed5a4b656cf908100dbe30f18b7c25c4875bd51"
Jul 17 07:59:20.172762 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:20.172731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:20.273099 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:20.273067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:22.148788 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.148757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:22.242360 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.242314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:22.434034 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.433932    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:59:22.434507 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.434444    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" podUID="5f30091f-8e20-478e-a4dc-71a7c53d1405" containerName="maas-api" containerID="cri-o://2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b" gracePeriod=30
Jul 17 07:59:22.459962 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.459936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:22.687039 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.686985    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:59:22.803254 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.803219    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca\") pod \"5f30091f-8e20-478e-a4dc-71a7c53d1405\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") "
Jul 17 07:59:22.803450 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.803293    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xffjd\" (UniqueName: \"kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd\") pod \"5f30091f-8e20-478e-a4dc-71a7c53d1405\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") "
Jul 17 07:59:22.803450 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.803364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") pod \"5f30091f-8e20-478e-a4dc-71a7c53d1405\" (UID: \"5f30091f-8e20-478e-a4dc-71a7c53d1405\") "
Jul 17 07:59:22.803609 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.803588    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f30091f-8e20-478e-a4dc-71a7c53d1405" (UID: "5f30091f-8e20-478e-a4dc-71a7c53d1405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:22.805401 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.805374    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd" (OuterVolumeSpecName: "kube-api-access-xffjd") pod "5f30091f-8e20-478e-a4dc-71a7c53d1405" (UID: "5f30091f-8e20-478e-a4dc-71a7c53d1405"). InnerVolumeSpecName "kube-api-access-xffjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:59:22.805401 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.805385    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f30091f-8e20-478e-a4dc-71a7c53d1405" (UID: "5f30091f-8e20-478e-a4dc-71a7c53d1405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:59:22.904845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.904808    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xffjd\" (UniqueName: \"kubernetes.io/projected/5f30091f-8e20-478e-a4dc-71a7c53d1405-kube-api-access-xffjd\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:22.904845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.904838    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f30091f-8e20-478e-a4dc-71a7c53d1405-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:22.904845 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:22.904848    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f30091f-8e20-478e-a4dc-71a7c53d1405-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:23.186909 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.186873    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f30091f-8e20-478e-a4dc-71a7c53d1405" containerID="2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b" exitCode=0
Jul 17 07:59:23.187301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.186932    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"
Jul 17 07:59:23.187301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.186957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" event={"ID":"5f30091f-8e20-478e-a4dc-71a7c53d1405","Type":"ContainerDied","Data":"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"}
Jul 17 07:59:23.187301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.187038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7" event={"ID":"5f30091f-8e20-478e-a4dc-71a7c53d1405","Type":"ContainerDied","Data":"cbc459f992e01842700443b1d3ec54cd31b879fd10f0f2ac31e0b98aeca810db"}
Jul 17 07:59:23.187301 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.187068    2584 scope.go:117] "RemoveContainer" containerID="2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"
Jul 17 07:59:23.196293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.196276    2584 scope.go:117] "RemoveContainer" containerID="2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"
Jul 17 07:59:23.196531 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:23.196512    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b\": container with ID starting with 2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b not found: ID does not exist" containerID="2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"
Jul 17 07:59:23.196599 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.196542    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b"} err="failed to get container status \"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b\": rpc error: code = NotFound desc = could not find container \"2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b\": container with ID starting with 2f10d4babfcd5e67175097d0fd68005f8fa8a98e782519107116de0ac7a9ea5b not found: ID does not exist"
Jul 17 07:59:23.225626 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.225605    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:59:23.228621 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:23.228598    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5cf58d-6df4c7ff78-qhmh7"]
Jul 17 07:59:24.286541 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:24.286512    2584 scope.go:117] "RemoveContainer" containerID="6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526"
Jul 17 07:59:24.290124 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:24.290102    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f30091f-8e20-478e-a4dc-71a7c53d1405" path="/var/lib/kubelet/pods/5f30091f-8e20-478e-a4dc-71a7c53d1405/volumes"
Jul 17 07:59:25.198314 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:25.198231    2584 generic.go:356] "Generic (PLEG): container finished" podID="8148249e-9e3d-4081-a174-f93265483674" containerID="416a341628d1b4a024058085b31a37df2ac85851b559a32304ffdd1d6614194c" exitCode=6
Jul 17 07:59:25.198519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:25.198306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerDied","Data":"416a341628d1b4a024058085b31a37df2ac85851b559a32304ffdd1d6614194c"}
Jul 17 07:59:25.198519 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:25.198378    2584 scope.go:117] "RemoveContainer" containerID="6158083b7e401233c924ea333f9f9dbb2c843be9b22b1d0aec6ec2edebe60526"
Jul 17 07:59:25.198695 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:25.198678    2584 scope.go:117] "RemoveContainer" containerID="416a341628d1b4a024058085b31a37df2ac85851b559a32304ffdd1d6614194c"
Jul 17 07:59:25.198920 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:25.198896    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf_odh-ai-gateway-infra(8148249e-9e3d-4081-a174-f93265483674)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" podUID="8148249e-9e3d-4081-a174-f93265483674"
Jul 17 07:59:26.230127 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.230092    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:26.243450 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.243419    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:59:26.367375 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.367311    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:26.436585 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.436548    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca\") pod \"8148249e-9e3d-4081-a174-f93265483674\" (UID: \"8148249e-9e3d-4081-a174-f93265483674\") "
Jul 17 07:59:26.436947 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.436925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8148249e-9e3d-4081-a174-f93265483674" (UID: "8148249e-9e3d-4081-a174-f93265483674"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:26.538118 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.538074    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8148249e-9e3d-4081-a174-f93265483674-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:26.646112 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:26.646072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:27.210454 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:27.210418    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf" event={"ID":"8148249e-9e3d-4081-a174-f93265483674","Type":"ContainerDied","Data":"bd87f7050d58d14f596487f1db4571ca7ba35b52f8074c6a0c23327d3dc8e2f7"}
Jul 17 07:59:27.210646 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:27.210475    2584 scope.go:117] "RemoveContainer" containerID="416a341628d1b4a024058085b31a37df2ac85851b559a32304ffdd1d6614194c"
Jul 17 07:59:27.210646 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:27.210434    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"
Jul 17 07:59:27.240403 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:27.240368    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:27.245489 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:27.245464    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hmlnf"]
Jul 17 07:59:28.289796 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:28.289763    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8148249e-9e3d-4081-a174-f93265483674" path="/var/lib/kubelet/pods/8148249e-9e3d-4081-a174-f93265483674/volumes"
Jul 17 07:59:29.364891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.364851    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:29.365404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365360    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365380    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365395    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365404 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365404    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365416    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7240de31-df71-4ec4-9dc2-5906b6b1bfc8" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365425    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7240de31-df71-4ec4-9dc2-5906b6b1bfc8" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365438    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365446    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365464    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f30091f-8e20-478e-a4dc-71a7c53d1405" containerName="maas-api"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365473    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f30091f-8e20-478e-a4dc-71a7c53d1405" containerName="maas-api"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365579    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7240de31-df71-4ec4-9dc2-5906b6b1bfc8" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365590    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365599    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f30091f-8e20-478e-a4dc-71a7c53d1405" containerName="maas-api"
Jul 17 07:59:29.365623 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.365609    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:29.370110 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.370088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:29.372872 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.372855    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:59:29.379089 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.379065    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:29.461850 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.461811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6\" (UID: \"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:29.562862 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.562827    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6\" (UID: \"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:29.563478 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.563460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6\" (UID: \"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:29.681331 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.681241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:29.812512 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:29.812482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:29.814516 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:59:29.814486    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5f603b_b2b1_4c6a_bcc0_6eb9a5c7d45e.slice/crio-5b207048e0a15e4c44475cab8deee9b6fa1a736e12aac485d9446678a6df6037 WatchSource:0}: Error finding container 5b207048e0a15e4c44475cab8deee9b6fa1a736e12aac485d9446678a6df6037: Status 404 returned error can't find the container with id 5b207048e0a15e4c44475cab8deee9b6fa1a736e12aac485d9446678a6df6037
Jul 17 07:59:30.227448 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:30.227406    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerStarted","Data":"13f25c95f5d41190e1ae1bc86df88af692e0f0a534c1df39a345caa7f4bf6249"}
Jul 17 07:59:30.227612 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:30.227465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerStarted","Data":"5b207048e0a15e4c44475cab8deee9b6fa1a736e12aac485d9446678a6df6037"}
Jul 17 07:59:30.248910 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:30.248854    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" podStartSLOduration=1.248834501 podStartE2EDuration="1.248834501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 07:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 07:59:30.247663554 +0000 UTC m=+2866.501086367" watchObservedRunningTime="2026-07-17 07:59:30.248834501 +0000 UTC m=+2866.502257314"
Jul 17 07:59:31.233209 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:31.233173    2584 generic.go:356] "Generic (PLEG): container finished" podID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerID="13f25c95f5d41190e1ae1bc86df88af692e0f0a534c1df39a345caa7f4bf6249" exitCode=6
Jul 17 07:59:31.233719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:31.233262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerDied","Data":"13f25c95f5d41190e1ae1bc86df88af692e0f0a534c1df39a345caa7f4bf6249"}
Jul 17 07:59:31.233719 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:31.233502    2584 scope.go:117] "RemoveContainer" containerID="13f25c95f5d41190e1ae1bc86df88af692e0f0a534c1df39a345caa7f4bf6249"
Jul 17 07:59:32.240246 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:32.240167    2584 generic.go:356] "Generic (PLEG): container finished" podID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerID="aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d" exitCode=6
Jul 17 07:59:32.240649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:32.240246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerDied","Data":"aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d"}
Jul 17 07:59:32.240649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:32.240303    2584 scope.go:117] "RemoveContainer" containerID="13f25c95f5d41190e1ae1bc86df88af692e0f0a534c1df39a345caa7f4bf6249"
Jul 17 07:59:32.240649 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:32.240558    2584 scope.go:117] "RemoveContainer" containerID="aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d"
Jul 17 07:59:32.240762 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:32.240745    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6_odh-ai-gateway-infra(fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e"
Jul 17 07:59:33.246295 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:33.246263    2584 scope.go:117] "RemoveContainer" containerID="aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d"
Jul 17 07:59:33.246719 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:33.246492    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6_odh-ai-gateway-infra(fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e"
Jul 17 07:59:33.273081 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:33.273049    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:33.342134 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:33.342102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:37.355245 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.355142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:37.620780 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.620657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:37.748057 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.748017    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"]
Jul 17 07:59:37.748521 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.748503    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8148249e-9e3d-4081-a174-f93265483674" containerName="revoke-keys"
Jul 17 07:59:37.752806 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.752785    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:37.760931 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.760911    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"]
Jul 17 07:59:37.932579 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:37.932490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx\" (UID: \"cbfcecbb-fd49-411a-9e03-36afb3aac785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:38.033420 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.033367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx\" (UID: \"cbfcecbb-fd49-411a-9e03-36afb3aac785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:38.034010 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.033988    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx\" (UID: \"cbfcecbb-fd49-411a-9e03-36afb3aac785\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:38.064086 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.064058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:38.196475 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.196444    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"]
Jul 17 07:59:38.198454 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:59:38.198418    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfcecbb_fd49_411a_9e03_36afb3aac785.slice/crio-fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f WatchSource:0}: Error finding container fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f: Status 404 returned error can't find the container with id fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f
Jul 17 07:59:38.269145 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.269111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx" event={"ID":"cbfcecbb-fd49-411a-9e03-36afb3aac785","Type":"ContainerStarted","Data":"fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f"}
Jul 17 07:59:38.296199 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.296175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:38.451918 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:38.451887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:39.275490 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:39.275405    2584 generic.go:356] "Generic (PLEG): container finished" podID="cbfcecbb-fd49-411a-9e03-36afb3aac785" containerID="c436dd662ec14b767ee0228d13d86f493af6d6a00099b7c7680e992e2658510b" exitCode=0
Jul 17 07:59:39.275490 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:39.275461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx" event={"ID":"cbfcecbb-fd49-411a-9e03-36afb3aac785","Type":"ContainerDied","Data":"c436dd662ec14b767ee0228d13d86f493af6d6a00099b7c7680e992e2658510b"}
Jul 17 07:59:39.495472 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:39.495440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:40.413286 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:40.413261    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:40.555101 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:40.555072    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca\") pod \"cbfcecbb-fd49-411a-9e03-36afb3aac785\" (UID: \"cbfcecbb-fd49-411a-9e03-36afb3aac785\") "
Jul 17 07:59:40.555520 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:40.555454    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbfcecbb-fd49-411a-9e03-36afb3aac785" (UID: "cbfcecbb-fd49-411a-9e03-36afb3aac785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:40.656535 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:40.656495    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfcecbb-fd49-411a-9e03-36afb3aac785-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:41.285381 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:41.285333    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx"
Jul 17 07:59:41.285381 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:41.285363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx" event={"ID":"cbfcecbb-fd49-411a-9e03-36afb3aac785","Type":"ContainerDied","Data":"fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f"}
Jul 17 07:59:41.285591 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:41.285407    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc35d21dc62bcbdf274e4f3a85a6da02aa2dd6676f6e72bf5ab3edb1aaf7638f"
Jul 17 07:59:47.284860 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:47.284826    2584 scope.go:117] "RemoveContainer" containerID="aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d"
Jul 17 07:59:48.317732 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.317704    2584 generic.go:356] "Generic (PLEG): container finished" podID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerID="7a42f2639405229676f960822b78c2926bcace90de1350dea8cd8a64bc08964f" exitCode=6
Jul 17 07:59:48.318197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.317784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerDied","Data":"7a42f2639405229676f960822b78c2926bcace90de1350dea8cd8a64bc08964f"}
Jul 17 07:59:48.318197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.317846    2584 scope.go:117] "RemoveContainer" containerID="aa54b3fb3a7201e75cfb75f8e1d63768b1129939e907316ad9260edaba67040d"
Jul 17 07:59:48.318197 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.318089    2584 scope.go:117] "RemoveContainer" containerID="7a42f2639405229676f960822b78c2926bcace90de1350dea8cd8a64bc08964f"
Jul 17 07:59:48.318326 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:48.318271    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6_odh-ai-gateway-infra(fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e"
Jul 17 07:59:48.457642 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.457605    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:59:48.457819 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.457788    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" podUID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" containerName="maas-api" containerID="cri-o://9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe" gracePeriod=30
Jul 17 07:59:48.486772 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.486737    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:48.705846 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.705821    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:59:48.717331 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.717304    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca\") pod \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") "
Jul 17 07:59:48.717476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.717408    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") pod \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") "
Jul 17 07:59:48.717476 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.717436    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nkrm\" (UniqueName: \"kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm\") pod \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\" (UID: \"5dcc3d12-2a8c-4829-9010-32cf463f6bb9\") "
Jul 17 07:59:48.717641 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.717620    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dcc3d12-2a8c-4829-9010-32cf463f6bb9" (UID: "5dcc3d12-2a8c-4829-9010-32cf463f6bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:48.719488 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.719464    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dcc3d12-2a8c-4829-9010-32cf463f6bb9" (UID: "5dcc3d12-2a8c-4829-9010-32cf463f6bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 07:59:48.719570 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.719544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm" (OuterVolumeSpecName: "kube-api-access-9nkrm") pod "5dcc3d12-2a8c-4829-9010-32cf463f6bb9" (UID: "5dcc3d12-2a8c-4829-9010-32cf463f6bb9"). InnerVolumeSpecName "kube-api-access-9nkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 07:59:48.817961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.817924    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:48.817961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.817955    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:48.817961 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:48.817965    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkrm\" (UniqueName: \"kubernetes.io/projected/5dcc3d12-2a8c-4829-9010-32cf463f6bb9-kube-api-access-9nkrm\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:49.324232 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.324198    2584 generic.go:356] "Generic (PLEG): container finished" podID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" containerID="9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe" exitCode=0
Jul 17 07:59:49.324712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.324246    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"
Jul 17 07:59:49.324712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.324277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" event={"ID":"5dcc3d12-2a8c-4829-9010-32cf463f6bb9","Type":"ContainerDied","Data":"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"}
Jul 17 07:59:49.324712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.324328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p" event={"ID":"5dcc3d12-2a8c-4829-9010-32cf463f6bb9","Type":"ContainerDied","Data":"0d1b80760bd0979b463656748e79d90e40b85ef17e45136c9e176313bd6cfd5b"}
Jul 17 07:59:49.324712 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.324375    2584 scope.go:117] "RemoveContainer" containerID="9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"
Jul 17 07:59:49.333002 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.332984    2584 scope.go:117] "RemoveContainer" containerID="9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"
Jul 17 07:59:49.333269 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:49.333250    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe\": container with ID starting with 9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe not found: ID does not exist" containerID="9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"
Jul 17 07:59:49.333333 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.333287    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe"} err="failed to get container status \"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe\": rpc error: code = NotFound desc = could not find container \"9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe\": container with ID starting with 9967f20e5bccf4180534daa350f75de59bdb92acba6e51e6c3d04909ae3635fe not found: ID does not exist"
Jul 17 07:59:49.345809 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.345782    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:59:49.348442 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.348419    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-38e02d-6874d56b8-ksp4p"]
Jul 17 07:59:49.354304 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.354283    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:49.481112 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.481090    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:49.523259 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.523227    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca\") pod \"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e\" (UID: \"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e\") "
Jul 17 07:59:49.523588 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.523566    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" (UID: "fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 07:59:49.624278 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:49.624174    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 07:59:50.290726 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.290690    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" path="/var/lib/kubelet/pods/5dcc3d12-2a8c-4829-9010-32cf463f6bb9/volumes"
Jul 17 07:59:50.331969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.331933    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"
Jul 17 07:59:50.331969 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.331961    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6" event={"ID":"fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e","Type":"ContainerDied","Data":"5b207048e0a15e4c44475cab8deee9b6fa1a736e12aac485d9446678a6df6037"}
Jul 17 07:59:50.332515 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.332030    2584 scope.go:117] "RemoveContainer" containerID="7a42f2639405229676f960822b78c2926bcace90de1350dea8cd8a64bc08964f"
Jul 17 07:59:50.350960 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.350933    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:50.355409 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:50.355375    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-lpxf6"]
Jul 17 07:59:51.737879 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:51.737842    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 07:59:52.195670 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.195638    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 07:59:52.195978 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.195965    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" containerName="maas-api"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.195980    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" containerName="maas-api"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.195995    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfcecbb-fd49-411a-9e03-36afb3aac785" containerName="revoke-keys"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196003    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfcecbb-fd49-411a-9e03-36afb3aac785" containerName="revoke-keys"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196012    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196018    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196022 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196024    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196029    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196038    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196045    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196094    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcc3d12-2a8c-4829-9010-32cf463f6bb9" containerName="maas-api"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196102    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196108    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfcecbb-fd49-411a-9e03-36afb3aac785" containerName="revoke-keys"
Jul 17 07:59:52.196212 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.196118    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" containerName="revoke-keys"
Jul 17 07:59:52.200467 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.200451    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 07:59:52.203203 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.203182    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 07:59:52.203319 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.203206    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 07:59:52.206891 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.206867    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 07:59:52.243411 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.243372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7\" (UID: \"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 07:59:52.291502 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.291466    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e" path="/var/lib/kubelet/pods/fa5f603b-b2b1-4c6a-bcc0-6eb9a5c7d45e/volumes"
Jul 17 07:59:52.344833 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.344803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7\" (UID: \"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 07:59:52.345433 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.345408    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7\" (UID: \"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 07:59:52.511855 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.511757    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 07:59:52.637701 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.637676    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 07:59:52.640062 ip-10-0-138-131 kubenswrapper[2584]: W0717 07:59:52.640032    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480021a1_ecc4_4c1e_b3c7_deef8b87d2c9.slice/crio-a4b753d3b3503eb40f8d2c5d86c59c57f2640bcd948d1bcc147c2dba64c094bd WatchSource:0}: Error finding container a4b753d3b3503eb40f8d2c5d86c59c57f2640bcd948d1bcc147c2dba64c094bd: Status 404 returned error can't find the container with id a4b753d3b3503eb40f8d2c5d86c59c57f2640bcd948d1bcc147c2dba64c094bd
Jul 17 07:59:52.844020 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:52.843991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:53.348716 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:53.348681    2584 generic.go:356] "Generic (PLEG): container finished" podID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerID="56c6bc7ce1e356d7224c2b538bf3bff39cbe986fe750c40753c5d3564b4ecde0" exitCode=6
Jul 17 07:59:53.348885 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:53.348772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" event={"ID":"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9","Type":"ContainerDied","Data":"56c6bc7ce1e356d7224c2b538bf3bff39cbe986fe750c40753c5d3564b4ecde0"}
Jul 17 07:59:53.348885 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:53.348821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" event={"ID":"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9","Type":"ContainerStarted","Data":"a4b753d3b3503eb40f8d2c5d86c59c57f2640bcd948d1bcc147c2dba64c094bd"}
Jul 17 07:59:53.348965 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:53.348945    2584 scope.go:117] "RemoveContainer" containerID="56c6bc7ce1e356d7224c2b538bf3bff39cbe986fe750c40753c5d3564b4ecde0"
Jul 17 07:59:54.355293 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:54.355263    2584 generic.go:356] "Generic (PLEG): container finished" podID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerID="4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39" exitCode=6
Jul 17 07:59:54.355739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:54.355316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" event={"ID":"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9","Type":"ContainerDied","Data":"4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39"}
Jul 17 07:59:54.355739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:54.355380    2584 scope.go:117] "RemoveContainer" containerID="56c6bc7ce1e356d7224c2b538bf3bff39cbe986fe750c40753c5d3564b4ecde0"
Jul 17 07:59:54.355739 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:54.355550    2584 scope.go:117] "RemoveContainer" containerID="4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39"
Jul 17 07:59:54.355739 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:54.355731    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7_odh-ai-gateway-infra(480021a1-ecc4-4c1e-b3c7-deef8b87d2c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9"
Jul 17 07:59:54.672765 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:54.672680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:55.361620 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:55.361592    2584 scope.go:117] "RemoveContainer" containerID="4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39"
Jul 17 07:59:55.362013 ip-10-0-138-131 kubenswrapper[2584]: E0717 07:59:55.361784    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7_odh-ai-gateway-infra(480021a1-ecc4-4c1e-b3c7-deef8b87d2c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9"
Jul 17 07:59:59.076008 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:59.075974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 07:59:59.176299 ip-10-0-138-131 kubenswrapper[2584]: I0717 07:59:59.176266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:03.617108 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:03.617077    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:03.868408 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:03.868293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:04.002717 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.002682    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:04.006747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.006727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:04.020116 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.020086    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:04.045817 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.045781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn\" (UID: \"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:04.146314 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.146226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn\" (UID: \"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:04.146873 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.146851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn\" (UID: \"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:04.226571 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.226537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:04.316469 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.316440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:04.660743 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.660716    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:04.663046 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:00:04.663016    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0f6a3c_aa4d_4f69_88dc_36cf2d36cd38.slice/crio-701977cff6d7b2e972f92f43097e0e895da7e6bf4104994b3efdd46fb6991732 WatchSource:0}: Error finding container 701977cff6d7b2e972f92f43097e0e895da7e6bf4104994b3efdd46fb6991732: Status 404 returned error can't find the container with id 701977cff6d7b2e972f92f43097e0e895da7e6bf4104994b3efdd46fb6991732
Jul 17 08:00:04.804970 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.804933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:04.902114 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:04.902076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:05.409313 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:05.409278    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerID="e1e093aecbd669c74e91ada4e06d43a26882f165210914687139462ec7c0e698" exitCode=6
Jul 17 08:00:05.409546 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:05.409378    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerDied","Data":"e1e093aecbd669c74e91ada4e06d43a26882f165210914687139462ec7c0e698"}
Jul 17 08:00:05.409546 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:05.409429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerStarted","Data":"701977cff6d7b2e972f92f43097e0e895da7e6bf4104994b3efdd46fb6991732"}
Jul 17 08:00:05.409672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:05.409582    2584 scope.go:117] "RemoveContainer" containerID="e1e093aecbd669c74e91ada4e06d43a26882f165210914687139462ec7c0e698"
Jul 17 08:00:06.416159 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:06.416124    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerID="06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51" exitCode=6
Jul 17 08:00:06.416608 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:06.416214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerDied","Data":"06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51"}
Jul 17 08:00:06.416608 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:06.416277    2584 scope.go:117] "RemoveContainer" containerID="e1e093aecbd669c74e91ada4e06d43a26882f165210914687139462ec7c0e698"
Jul 17 08:00:06.416608 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:06.416482    2584 scope.go:117] "RemoveContainer" containerID="06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51"
Jul 17 08:00:06.416751 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:06.416714    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn_odh-ai-gateway-infra(ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38"
Jul 17 08:00:07.423385 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:07.423329    2584 scope.go:117] "RemoveContainer" containerID="06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51"
Jul 17 08:00:07.423789 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:07.423567    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn_odh-ai-gateway-infra(ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38"
Jul 17 08:00:08.000162 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.000132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.056833 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.056793    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:00:08.061380 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.061356    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.064032 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.064008    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jwckj\""
Jul 17 08:00:08.064377 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.064361    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 08:00:08.065023 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.065007    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d805bd-serving-cert\""
Jul 17 08:00:08.071019 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.070991    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:00:08.100764 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.100728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.184127 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.184088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.184364 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.184174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6m2\" (UniqueName: \"kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.184364 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.184246    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.184899 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.184869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.286789 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.286753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6m2\" (UniqueName: \"kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.286989 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.286836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.286989 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.286903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.287113 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:08.287041    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-serving-cert: secret "maas-api-e2e-shared-a-d805bd-serving-cert" not found
Jul 17 08:00:08.287207 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:08.287115    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls podName:48d8457f-404f-4590-a3fa-5165799aa088 nodeName:}" failed. No retries permitted until 2026-07-17 08:00:08.787093733 +0000 UTC m=+2905.040516542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls") pod "maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" (UID: "48d8457f-404f-4590-a3fa-5165799aa088") : secret "maas-api-e2e-shared-a-d805bd-serving-cert" not found
Jul 17 08:00:08.287968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.287946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.298465 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.298438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6m2\" (UniqueName: \"kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.424080 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.424016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.644427 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.644305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.743877 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.743804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:08.791975 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.791934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.794353 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.794319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") pod \"maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:08.973121 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:08.973029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:09.106173 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:09.106148    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:00:09.107923 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:00:09.107893    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d8457f_404f_4590_a3fa_5165799aa088.slice/crio-5d3177dfe86388ea6b8e91799869ba6f3a2e859e8e8498254e616ca458b72d98 WatchSource:0}: Error finding container 5d3177dfe86388ea6b8e91799869ba6f3a2e859e8e8498254e616ca458b72d98: Status 404 returned error can't find the container with id 5d3177dfe86388ea6b8e91799869ba6f3a2e859e8e8498254e616ca458b72d98
Jul 17 08:00:09.432638 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:09.432603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" event={"ID":"48d8457f-404f-4590-a3fa-5165799aa088","Type":"ContainerStarted","Data":"5d3177dfe86388ea6b8e91799869ba6f3a2e859e8e8498254e616ca458b72d98"}
Jul 17 08:00:10.290771 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:10.290737    2584 scope.go:117] "RemoveContainer" containerID="4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39"
Jul 17 08:00:10.439634 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:10.439599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" event={"ID":"48d8457f-404f-4590-a3fa-5165799aa088","Type":"ContainerStarted","Data":"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"}
Jul 17 08:00:10.440033 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:10.439666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:10.464114 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:10.464064    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" podStartSLOduration=1.766087414 podStartE2EDuration="2.464048378s" totalImagesPullingTime="697.960964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:00:10.463233707 +0000 UTC m=+2906.716656519" watchObservedRunningTime="2026-07-17 08:00:10.464048378 +0000 UTC m=+2906.717471189"
Jul 17 08:00:11.170275 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.170219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:11.272542 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.272455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:11.446288 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.446231    2584 generic.go:356] "Generic (PLEG): container finished" podID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerID="9ecb9fa62b57fa7e57a47d171a720580b42c49e1ae387447dac2404b0a449c04" exitCode=6
Jul 17 08:00:11.446780 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.446302    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" event={"ID":"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9","Type":"ContainerDied","Data":"9ecb9fa62b57fa7e57a47d171a720580b42c49e1ae387447dac2404b0a449c04"}
Jul 17 08:00:11.446780 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.446400    2584 scope.go:117] "RemoveContainer" containerID="4fc777ada6d19e64bb82291fb36390d7c8d05260671f263d22a99c5220e4ee39"
Jul 17 08:00:11.446921 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:11.446823    2584 scope.go:117] "RemoveContainer" containerID="9ecb9fa62b57fa7e57a47d171a720580b42c49e1ae387447dac2404b0a449c04"
Jul 17 08:00:11.447037 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:11.447017    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7_odh-ai-gateway-infra(480021a1-ecc4-4c1e-b3c7-deef8b87d2c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9"
Jul 17 08:00:12.336646 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.336615    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:00:12.474672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.474639    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 08:00:12.518467 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.518415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:12.606302 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.606276    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 08:00:12.723995 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.723964    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca\") pod \"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9\" (UID: \"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9\") "
Jul 17 08:00:12.724325 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.724300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" (UID: "480021a1-ecc4-4c1e-b3c7-deef8b87d2c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:00:12.825358 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.825293    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:00:12.939074 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:12.938944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:13.223551 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.223428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:13.459542 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.459506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7" event={"ID":"480021a1-ecc4-4c1e-b3c7-deef8b87d2c9","Type":"ContainerDied","Data":"a4b753d3b3503eb40f8d2c5d86c59c57f2640bcd948d1bcc147c2dba64c094bd"}
Jul 17 08:00:13.459542 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.459543    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"
Jul 17 08:00:13.459776 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.459581    2584 scope.go:117] "RemoveContainer" containerID="9ecb9fa62b57fa7e57a47d171a720580b42c49e1ae387447dac2404b0a449c04"
Jul 17 08:00:13.481166 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.481107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 08:00:13.488970 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.488949    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8x4j7"]
Jul 17 08:00:13.644635 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:13.644578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:14.289731 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:14.289701    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" path="/var/lib/kubelet/pods/480021a1-ecc4-4c1e-b3c7-deef8b87d2c9/volumes"
Jul 17 08:00:16.058012 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.057975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:16.292405 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292371    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:00:16.292812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292798    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.292880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292814    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.292880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292836    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.292880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292845    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.292880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292855    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.292880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292862    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.293157 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292953    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.293157 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.292968    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:16.297731 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.297707    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.300416 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.300392    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-04d1db-serving-cert\""
Jul 17 08:00:16.304190 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.304165    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:00:16.354013 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.353929    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.354013 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.353975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tnww\" (UniqueName: \"kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.354211 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.354041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.452317 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.452289    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:00:16.455002 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.454982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tnww\" (UniqueName: \"kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.455060 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.455031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.455106 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.455069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.455169 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:16.455156    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-serving-cert: secret "maas-api-e2e-shared-b-04d1db-serving-cert" not found
Jul 17 08:00:16.455222 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:16.455213    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls podName:cd7ce5d7-0042-4680-9235-108094dbcb1f nodeName:}" failed. No retries permitted until 2026-07-17 08:00:16.955195754 +0000 UTC m=+2913.208618545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls") pod "maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" (UID: "cd7ce5d7-0042-4680-9235-108094dbcb1f") : secret "maas-api-e2e-shared-b-04d1db-serving-cert" not found
Jul 17 08:00:16.455772 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.455748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.463234 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.463213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tnww\" (UniqueName: \"kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.567636 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.567605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:16.698988 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.698907    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:16.959822 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.959736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:16.962169 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:16.962142    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") pod \"maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:17.093927 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:17.093893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:17.194286 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:17.194244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:17.209924 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:17.209862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:17.376866 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:17.376795    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:00:17.379355 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:00:17.379315    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7ce5d7_0042_4680_9235_108094dbcb1f.slice/crio-a5f1582de394fe52a6a6f8c7e3ee631238987c9a5ee315a14e2ca928ea3bbd43 WatchSource:0}: Error finding container a5f1582de394fe52a6a6f8c7e3ee631238987c9a5ee315a14e2ca928ea3bbd43: Status 404 returned error can't find the container with id a5f1582de394fe52a6a6f8c7e3ee631238987c9a5ee315a14e2ca928ea3bbd43
Jul 17 08:00:17.478844 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:17.478758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" event={"ID":"cd7ce5d7-0042-4680-9235-108094dbcb1f","Type":"ContainerStarted","Data":"a5f1582de394fe52a6a6f8c7e3ee631238987c9a5ee315a14e2ca928ea3bbd43"}
Jul 17 08:00:18.484880 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:18.484846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" event={"ID":"cd7ce5d7-0042-4680-9235-108094dbcb1f","Type":"ContainerStarted","Data":"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"}
Jul 17 08:00:18.485255 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:18.484927    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:18.524222 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:18.524173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" podStartSLOduration=1.579509005 podStartE2EDuration="2.524157478s" totalImagesPullingTime="944.648473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:00:18.522364735 +0000 UTC m=+2914.775787538" watchObservedRunningTime="2026-07-17 08:00:18.524157478 +0000 UTC m=+2914.777580291"
Jul 17 08:00:20.760960 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:20.760919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:21.285958 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:21.285924    2584 scope.go:117] "RemoveContainer" containerID="06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51"
Jul 17 08:00:21.500096 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:21.500067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerStarted","Data":"ffb46414ff0e7b3d5730876441a8f041b56faa3405a0b6e525af4b4e523e80c3"}
Jul 17 08:00:21.519076 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:21.519029    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" podStartSLOduration=18.519015807 podStartE2EDuration="18.519015807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:00:21.516702446 +0000 UTC m=+2917.770125258" watchObservedRunningTime="2026-07-17 08:00:21.519015807 +0000 UTC m=+2917.772438618"
Jul 17 08:00:22.506122 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:22.506091    2584 generic.go:356] "Generic (PLEG): container finished" podID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerID="ffb46414ff0e7b3d5730876441a8f041b56faa3405a0b6e525af4b4e523e80c3" exitCode=6
Jul 17 08:00:22.506614 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:22.506144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerDied","Data":"ffb46414ff0e7b3d5730876441a8f041b56faa3405a0b6e525af4b4e523e80c3"}
Jul 17 08:00:22.506614 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:22.506205    2584 scope.go:117] "RemoveContainer" containerID="06cfc617527907bcbec1840689aca6a9aa9f551c5a7734d731378f52d57d2d51"
Jul 17 08:00:22.506614 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:22.506517    2584 scope.go:117] "RemoveContainer" containerID="ffb46414ff0e7b3d5730876441a8f041b56faa3405a0b6e525af4b4e523e80c3"
Jul 17 08:00:22.506770 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:22.506742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn_odh-ai-gateway-infra(ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38"
Jul 17 08:00:22.535799 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:22.535766    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:23.236077 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:23.236041    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:00:23.649720 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:23.649689    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:23.728922 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:23.728878    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca\") pod \"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38\" (UID: \"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38\") "
Jul 17 08:00:23.729284 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:23.729261    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" (UID: "ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:00:23.829971 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:23.829929    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:00:24.495328 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.495295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:00:24.519048 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.519022    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"
Jul 17 08:00:24.519259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.519063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn" event={"ID":"ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38","Type":"ContainerDied","Data":"701977cff6d7b2e972f92f43097e0e895da7e6bf4104994b3efdd46fb6991732"}
Jul 17 08:00:24.519259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.519126    2584 scope.go:117] "RemoveContainer" containerID="ffb46414ff0e7b3d5730876441a8f041b56faa3405a0b6e525af4b4e523e80c3"
Jul 17 08:00:24.540668 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.540634    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:24.546140 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:24.546113    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-rsdcn"]
Jul 17 08:00:26.289980 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:26.289939    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" path="/var/lib/kubelet/pods/ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38/volumes"
Jul 17 08:00:29.967482 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:29.967450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:30.843491 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:30.843457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:30.944909 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:30.944872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:32.237842 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:32.237806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:00:37.849223 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:37.849188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:37.948580 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:37.948548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:00:52.290748 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.290716    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.290996    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291006    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291025    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291030    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291083    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291091    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:52.291124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.291097    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="480021a1-ecc4-4c1e-b3c7-deef8b87d2c9" containerName="revoke-keys"
Jul 17 08:00:52.295658 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.295639    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:00:52.300287 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.300262    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:00:52.300627 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.300566    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:00:52.365876 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.365835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n\" (UID: \"3ce2b073-0527-4e82-83ad-9577163d34ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:00:52.466811 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.466773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n\" (UID: \"3ce2b073-0527-4e82-83ad-9577163d34ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:00:52.467464 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.467445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n\" (UID: \"3ce2b073-0527-4e82-83ad-9577163d34ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:00:52.608306 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.608209    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:00:52.750140 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:52.750108    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:00:52.752766 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:00:52.752740    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce2b073_0527_4e82_83ad_9577163d34ac.slice/crio-c45afb758495af7403ba53d3ad36c9520ece505fd2cda028e6f68346f485ff6c WatchSource:0}: Error finding container c45afb758495af7403ba53d3ad36c9520ece505fd2cda028e6f68346f485ff6c: Status 404 returned error can't find the container with id c45afb758495af7403ba53d3ad36c9520ece505fd2cda028e6f68346f485ff6c
Jul 17 08:00:53.656619 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:53.656583    2584 generic.go:356] "Generic (PLEG): container finished" podID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerID="bb1e3bf9239225233ae18ee66cfbd1d582ff69e0f1fc7954b0906364d70f133b" exitCode=6
Jul 17 08:00:53.656619 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:53.656622    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerDied","Data":"bb1e3bf9239225233ae18ee66cfbd1d582ff69e0f1fc7954b0906364d70f133b"}
Jul 17 08:00:53.657057 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:53.656661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerStarted","Data":"c45afb758495af7403ba53d3ad36c9520ece505fd2cda028e6f68346f485ff6c"}
Jul 17 08:00:53.657057 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:53.656825    2584 scope.go:117] "RemoveContainer" containerID="bb1e3bf9239225233ae18ee66cfbd1d582ff69e0f1fc7954b0906364d70f133b"
Jul 17 08:00:54.662805 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:54.662719    2584 generic.go:356] "Generic (PLEG): container finished" podID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerID="67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051" exitCode=6
Jul 17 08:00:54.663234 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:54.662797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerDied","Data":"67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051"}
Jul 17 08:00:54.663234 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:54.662891    2584 scope.go:117] "RemoveContainer" containerID="bb1e3bf9239225233ae18ee66cfbd1d582ff69e0f1fc7954b0906364d70f133b"
Jul 17 08:00:54.663234 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:54.663054    2584 scope.go:117] "RemoveContainer" containerID="67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051"
Jul 17 08:00:54.663395 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:54.663279    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n_odh-ai-gateway-infra(3ce2b073-0527-4e82-83ad-9577163d34ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac"
Jul 17 08:00:55.669282 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:55.669250    2584 scope.go:117] "RemoveContainer" containerID="67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051"
Jul 17 08:00:55.669714 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:55.669469    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n_odh-ai-gateway-infra(3ce2b073-0527-4e82-83ad-9577163d34ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac"
Jul 17 08:00:56.361449 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.361409    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:00:56.361830 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.361811    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:56.361921 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.361832    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:56.361975 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.361938    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0f6a3c-aa4d-4f69-88dc-36cf2d36cd38" containerName="revoke-keys"
Jul 17 08:00:56.365970 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.365948    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:00:56.377036 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.377014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:00:56.401819 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.401784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr\" (UID: \"4cd2cb36-2ad8-448a-8405-72fcb6e950e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:00:56.502597 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.502548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr\" (UID: \"4cd2cb36-2ad8-448a-8405-72fcb6e950e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:00:56.503172 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.503150    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr\" (UID: \"4cd2cb36-2ad8-448a-8405-72fcb6e950e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:00:56.676604 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.676515    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:00:56.804455 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:56.804431    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:00:56.807016 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:00:56.806992    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd2cb36_2ad8_448a_8405_72fcb6e950e9.slice/crio-b809b7aa1d515acc1381934c6657ece19e254db3950a6e40bc1fa5538cdacb1d WatchSource:0}: Error finding container b809b7aa1d515acc1381934c6657ece19e254db3950a6e40bc1fa5538cdacb1d: Status 404 returned error can't find the container with id b809b7aa1d515acc1381934c6657ece19e254db3950a6e40bc1fa5538cdacb1d
Jul 17 08:00:57.679004 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:57.678970    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerID="7e5f3d01401f738dc7eef0859679eb7378082ac0de2226a29d2847cfbce71017" exitCode=6
Jul 17 08:00:57.679415 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:57.679057    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerDied","Data":"7e5f3d01401f738dc7eef0859679eb7378082ac0de2226a29d2847cfbce71017"}
Jul 17 08:00:57.679415 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:57.679115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerStarted","Data":"b809b7aa1d515acc1381934c6657ece19e254db3950a6e40bc1fa5538cdacb1d"}
Jul 17 08:00:57.679415 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:57.679205    2584 scope.go:117] "RemoveContainer" containerID="7e5f3d01401f738dc7eef0859679eb7378082ac0de2226a29d2847cfbce71017"
Jul 17 08:00:58.685618 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:58.685532    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerID="624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf" exitCode=6
Jul 17 08:00:58.685618 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:58.685581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerDied","Data":"624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf"}
Jul 17 08:00:58.686071 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:58.685629    2584 scope.go:117] "RemoveContainer" containerID="7e5f3d01401f738dc7eef0859679eb7378082ac0de2226a29d2847cfbce71017"
Jul 17 08:00:58.686071 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:58.685832    2584 scope.go:117] "RemoveContainer" containerID="624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf"
Jul 17 08:00:58.686071 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:58.686029    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr_odh-ai-gateway-infra(4cd2cb36-2ad8-448a-8405-72fcb6e950e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9"
Jul 17 08:00:59.691205 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:00:59.691176    2584 scope.go:117] "RemoveContainer" containerID="624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf"
Jul 17 08:00:59.691603 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:00:59.691353    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr_odh-ai-gateway-infra(4cd2cb36-2ad8-448a-8405-72fcb6e950e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9"
Jul 17 08:01:04.061438 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.061401    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:04.065949 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.065931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:04.073278 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.073031    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:04.162522 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.162487    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4\" (UID: \"53f8a802-433f-42c6-ba25-c7acd03a3583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:04.263024 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.262992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4\" (UID: \"53f8a802-433f-42c6-ba25-c7acd03a3583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:04.263679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.263653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4\" (UID: \"53f8a802-433f-42c6-ba25-c7acd03a3583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:04.377080 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.376994    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:04.506888 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.506865    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:04.509412 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:01:04.509385    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f8a802_433f_42c6_ba25_c7acd03a3583.slice/crio-4e8db9b648521af73af43f4b4ed3854db8137e26c53e0c3e1960cda52dbbd2d1 WatchSource:0}: Error finding container 4e8db9b648521af73af43f4b4ed3854db8137e26c53e0c3e1960cda52dbbd2d1: Status 404 returned error can't find the container with id 4e8db9b648521af73af43f4b4ed3854db8137e26c53e0c3e1960cda52dbbd2d1
Jul 17 08:01:04.713191 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:04.713106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerStarted","Data":"4e8db9b648521af73af43f4b4ed3854db8137e26c53e0c3e1960cda52dbbd2d1"}
Jul 17 08:01:05.719757 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:05.719726    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerID="a7f4c1fc0af29a7ca82290473f777002421de4f809e3bb1014d6eec29a6ea61c" exitCode=6
Jul 17 08:01:05.720142 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:05.719777    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerDied","Data":"a7f4c1fc0af29a7ca82290473f777002421de4f809e3bb1014d6eec29a6ea61c"}
Jul 17 08:01:05.720142 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:05.719991    2584 scope.go:117] "RemoveContainer" containerID="a7f4c1fc0af29a7ca82290473f777002421de4f809e3bb1014d6eec29a6ea61c"
Jul 17 08:01:06.726567 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:06.726448    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerID="ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f" exitCode=6
Jul 17 08:01:06.726567 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:06.726501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerDied","Data":"ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f"}
Jul 17 08:01:06.727034 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:06.726560    2584 scope.go:117] "RemoveContainer" containerID="a7f4c1fc0af29a7ca82290473f777002421de4f809e3bb1014d6eec29a6ea61c"
Jul 17 08:01:06.727034 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:06.726745    2584 scope.go:117] "RemoveContainer" containerID="ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f"
Jul 17 08:01:06.727034 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:06.726962    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4_odh-ai-gateway-infra(53f8a802-433f-42c6-ba25-c7acd03a3583)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583"
Jul 17 08:01:07.285022 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:07.284986    2584 scope.go:117] "RemoveContainer" containerID="67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051"
Jul 17 08:01:07.732600 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:07.732508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerStarted","Data":"d789238c3f0ca78775ef9f3846d5838eb039c8fcfe5459910440f5ed466a4881"}
Jul 17 08:01:07.734091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:07.734072    2584 scope.go:117] "RemoveContainer" containerID="ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f"
Jul 17 08:01:07.734242 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:07.734225    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4_odh-ai-gateway-infra(53f8a802-433f-42c6-ba25-c7acd03a3583)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583"
Jul 17 08:01:07.752417 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:07.752373    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" podStartSLOduration=15.75233142 podStartE2EDuration="15.75233142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:01:07.750669786 +0000 UTC m=+2964.004092608" watchObservedRunningTime="2026-07-17 08:01:07.75233142 +0000 UTC m=+2964.005754252"
Jul 17 08:01:08.740224 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:08.740193    2584 generic.go:356] "Generic (PLEG): container finished" podID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerID="d789238c3f0ca78775ef9f3846d5838eb039c8fcfe5459910440f5ed466a4881" exitCode=6
Jul 17 08:01:08.740677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:08.740267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerDied","Data":"d789238c3f0ca78775ef9f3846d5838eb039c8fcfe5459910440f5ed466a4881"}
Jul 17 08:01:08.740677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:08.740326    2584 scope.go:117] "RemoveContainer" containerID="67362fac5058e670c71d31769c96aa4ed20b9040ba3a73b4cbfb8cb526e5b051"
Jul 17 08:01:08.740677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:08.740588    2584 scope.go:117] "RemoveContainer" containerID="d789238c3f0ca78775ef9f3846d5838eb039c8fcfe5459910440f5ed466a4881"
Jul 17 08:01:08.740818 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:08.740771    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n_odh-ai-gateway-infra(3ce2b073-0527-4e82-83ad-9577163d34ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac"
Jul 17 08:01:08.768724 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:08.768691    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:01:09.874413 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:09.874386    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:01:10.012306 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.012229    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca\") pod \"3ce2b073-0527-4e82-83ad-9577163d34ac\" (UID: \"3ce2b073-0527-4e82-83ad-9577163d34ac\") "
Jul 17 08:01:10.012602 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.012579    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ce2b073-0527-4e82-83ad-9577163d34ac" (UID: "3ce2b073-0527-4e82-83ad-9577163d34ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:01:10.113451 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.113421    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce2b073-0527-4e82-83ad-9577163d34ac-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:01:10.752012 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.751941    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"
Jul 17 08:01:10.752153 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.751937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n" event={"ID":"3ce2b073-0527-4e82-83ad-9577163d34ac","Type":"ContainerDied","Data":"c45afb758495af7403ba53d3ad36c9520ece505fd2cda028e6f68346f485ff6c"}
Jul 17 08:01:10.752153 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.752080    2584 scope.go:117] "RemoveContainer" containerID="d789238c3f0ca78775ef9f3846d5838eb039c8fcfe5459910440f5ed466a4881"
Jul 17 08:01:10.770628 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.770601    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:01:10.778243 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:10.778220    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-6fs5n"]
Jul 17 08:01:12.289729 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:12.289695    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" path="/var/lib/kubelet/pods/3ce2b073-0527-4e82-83ad-9577163d34ac/volumes"
Jul 17 08:01:13.285076 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:13.285042    2584 scope.go:117] "RemoveContainer" containerID="624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf"
Jul 17 08:01:13.768416 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:13.768319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerStarted","Data":"26b93b6725015359fd6637c9adab741059e9e92e8539c2ed4523c6afcddef613"}
Jul 17 08:01:14.239795 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.239758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:01:14.774809 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.774772    2584 generic.go:356] "Generic (PLEG): container finished" podID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerID="26b93b6725015359fd6637c9adab741059e9e92e8539c2ed4523c6afcddef613" exitCode=6
Jul 17 08:01:14.775215 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.774814    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerDied","Data":"26b93b6725015359fd6637c9adab741059e9e92e8539c2ed4523c6afcddef613"}
Jul 17 08:01:14.775215 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.774861    2584 scope.go:117] "RemoveContainer" containerID="624e6b317301e32e08bced1ed63ed22ff799f855aba572dfff070e5c56460abf"
Jul 17 08:01:14.775215 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.775178    2584 scope.go:117] "RemoveContainer" containerID="26b93b6725015359fd6637c9adab741059e9e92e8539c2ed4523c6afcddef613"
Jul 17 08:01:14.775442 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:14.775417    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr_odh-ai-gateway-infra(4cd2cb36-2ad8-448a-8405-72fcb6e950e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9"
Jul 17 08:01:14.802586 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:14.802558    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:01:15.915728 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:15.915705    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:01:16.068674 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.068643    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca\") pod \"4cd2cb36-2ad8-448a-8405-72fcb6e950e9\" (UID: \"4cd2cb36-2ad8-448a-8405-72fcb6e950e9\") "
Jul 17 08:01:16.069001 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.068979    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cd2cb36-2ad8-448a-8405-72fcb6e950e9" (UID: "4cd2cb36-2ad8-448a-8405-72fcb6e950e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:01:16.169786 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.169758    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd2cb36-2ad8-448a-8405-72fcb6e950e9-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:01:16.787013 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.786981    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"
Jul 17 08:01:16.787193 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.786985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr" event={"ID":"4cd2cb36-2ad8-448a-8405-72fcb6e950e9","Type":"ContainerDied","Data":"b809b7aa1d515acc1381934c6657ece19e254db3950a6e40bc1fa5538cdacb1d"}
Jul 17 08:01:16.787193 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.787117    2584 scope.go:117] "RemoveContainer" containerID="26b93b6725015359fd6637c9adab741059e9e92e8539c2ed4523c6afcddef613"
Jul 17 08:01:16.804701 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.804676    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:01:16.808718 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:16.808698    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-jmrrr"]
Jul 17 08:01:17.937912 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:17.937875    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:01:18.289879 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:18.289846    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" path="/var/lib/kubelet/pods/4cd2cb36-2ad8-448a-8405-72fcb6e950e9/volumes"
Jul 17 08:01:22.284901 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:22.284865    2584 scope.go:117] "RemoveContainer" containerID="ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f"
Jul 17 08:01:22.817956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:22.817920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerStarted","Data":"d3dbe3c28ea7d39bf06ebb929f0a1c3063081dca1ace653fbd4ff27298fc6820"}
Jul 17 08:01:22.837529 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:22.837486    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" podStartSLOduration=18.837469631 podStartE2EDuration="18.837469631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:01:22.835968953 +0000 UTC m=+2979.089391766" watchObservedRunningTime="2026-07-17 08:01:22.837469631 +0000 UTC m=+2979.090892442"
Jul 17 08:01:23.830535 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:23.830505    2584 generic.go:356] "Generic (PLEG): container finished" podID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerID="d3dbe3c28ea7d39bf06ebb929f0a1c3063081dca1ace653fbd4ff27298fc6820" exitCode=6
Jul 17 08:01:23.831017 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:23.830567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerDied","Data":"d3dbe3c28ea7d39bf06ebb929f0a1c3063081dca1ace653fbd4ff27298fc6820"}
Jul 17 08:01:23.831017 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:23.830640    2584 scope.go:117] "RemoveContainer" containerID="ea1cb8dc7646c3a81bac2f11215cdd30aad966e9b9d713405902e53333a90a3f"
Jul 17 08:01:23.831017 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:23.830955    2584 scope.go:117] "RemoveContainer" containerID="d3dbe3c28ea7d39bf06ebb929f0a1c3063081dca1ace653fbd4ff27298fc6820"
Jul 17 08:01:23.831200 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:23.831175    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4_odh-ai-gateway-infra(53f8a802-433f-42c6-ba25-c7acd03a3583)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583"
Jul 17 08:01:23.846953 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:23.846928    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:24.967324 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:24.967291    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:25.145207 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.145115    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca\") pod \"53f8a802-433f-42c6-ba25-c7acd03a3583\" (UID: \"53f8a802-433f-42c6-ba25-c7acd03a3583\") "
Jul 17 08:01:25.145505 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.145483    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53f8a802-433f-42c6-ba25-c7acd03a3583" (UID: "53f8a802-433f-42c6-ba25-c7acd03a3583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:01:25.246199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.246149    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f8a802-433f-42c6-ba25-c7acd03a3583-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:01:25.842041 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.842007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4" event={"ID":"53f8a802-433f-42c6-ba25-c7acd03a3583","Type":"ContainerDied","Data":"4e8db9b648521af73af43f4b4ed3854db8137e26c53e0c3e1960cda52dbbd2d1"}
Jul 17 08:01:25.842233 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.842064    2584 scope.go:117] "RemoveContainer" containerID="d3dbe3c28ea7d39bf06ebb929f0a1c3063081dca1ace653fbd4ff27298fc6820"
Jul 17 08:01:25.842233 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.842019    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"
Jul 17 08:01:25.868820 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.868780    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:25.870823 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:25.870802    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-2nhf4"]
Jul 17 08:01:26.290675 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:26.290644    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" path="/var/lib/kubelet/pods/53f8a802-433f-42c6-ba25-c7acd03a3583/volumes"
Jul 17 08:01:29.464459 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464422    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464730    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464741    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464751    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464756    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464765    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464771    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464778    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464784    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464792    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464798    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464810    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464815    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464822    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.464828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464827    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464878    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464886    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464893    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464898    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464904    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464911    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f8a802-433f-42c6-ba25-c7acd03a3583" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464916    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.465260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.464921    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:29.468951 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.468934    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:29.471969 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.471947    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:01:29.479473 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.479455    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:29.583245 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.583204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv\" (UID: \"11ff28ba-e5fa-455a-b41e-9d888b976aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:29.683788 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.683751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv\" (UID: \"11ff28ba-e5fa-455a-b41e-9d888b976aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:29.684391 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.684372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv\" (UID: \"11ff28ba-e5fa-455a-b41e-9d888b976aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:29.778622 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.778512    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:29.914590 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:29.914551    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:29.915411 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:01:29.915383    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ff28ba_e5fa_455a_b41e_9d888b976aba.slice/crio-f2567123396026c6295975b68eae5ab12a88f41d14027a6fe63268b0c06d394f WatchSource:0}: Error finding container f2567123396026c6295975b68eae5ab12a88f41d14027a6fe63268b0c06d394f: Status 404 returned error can't find the container with id f2567123396026c6295975b68eae5ab12a88f41d14027a6fe63268b0c06d394f
Jul 17 08:01:30.868957 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:30.868849    2584 generic.go:356] "Generic (PLEG): container finished" podID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerID="b52cdb3dcd1b3173d31b6661745301107015a57eba83ee1c5d81b6cf60a6ccf9" exitCode=6
Jul 17 08:01:30.868957 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:30.868928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" event={"ID":"11ff28ba-e5fa-455a-b41e-9d888b976aba","Type":"ContainerDied","Data":"b52cdb3dcd1b3173d31b6661745301107015a57eba83ee1c5d81b6cf60a6ccf9"}
Jul 17 08:01:30.869510 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:30.868983    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" event={"ID":"11ff28ba-e5fa-455a-b41e-9d888b976aba","Type":"ContainerStarted","Data":"f2567123396026c6295975b68eae5ab12a88f41d14027a6fe63268b0c06d394f"}
Jul 17 08:01:30.869510 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:30.869169    2584 scope.go:117] "RemoveContainer" containerID="b52cdb3dcd1b3173d31b6661745301107015a57eba83ee1c5d81b6cf60a6ccf9"
Jul 17 08:01:31.875785 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:31.875756    2584 generic.go:356] "Generic (PLEG): container finished" podID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerID="18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb" exitCode=6
Jul 17 08:01:31.876187 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:31.875850    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" event={"ID":"11ff28ba-e5fa-455a-b41e-9d888b976aba","Type":"ContainerDied","Data":"18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb"}
Jul 17 08:01:31.876187 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:31.875917    2584 scope.go:117] "RemoveContainer" containerID="b52cdb3dcd1b3173d31b6661745301107015a57eba83ee1c5d81b6cf60a6ccf9"
Jul 17 08:01:31.876187 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:31.876049    2584 scope.go:117] "RemoveContainer" containerID="18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb"
Jul 17 08:01:31.876395 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:31.876265    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv_odh-ai-gateway-infra(11ff28ba-e5fa-455a-b41e-9d888b976aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba"
Jul 17 08:01:32.882036 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:32.882003    2584 scope.go:117] "RemoveContainer" containerID="18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb"
Jul 17 08:01:32.882454 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:32.882182    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv_odh-ai-gateway-infra(11ff28ba-e5fa-455a-b41e-9d888b976aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba"
Jul 17 08:01:33.435585 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:33.435549    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:01:42.333128 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333089    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:01:42.333633 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333432    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:42.333633 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333443    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:42.333633 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333457    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:42.333633 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333462    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd2cb36-2ad8-448a-8405-72fcb6e950e9" containerName="revoke-keys"
Jul 17 08:01:42.333633 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.333522    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce2b073-0527-4e82-83ad-9577163d34ac" containerName="revoke-keys"
Jul 17 08:01:42.337869 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.337850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:42.350492 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.350459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:01:42.387472 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.387438    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp\" (UID: \"bcdb2d59-6073-4cf8-ab45-c403e3e469b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:42.488010 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.487972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp\" (UID: \"bcdb2d59-6073-4cf8-ab45-c403e3e469b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:42.488665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.488645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp\" (UID: \"bcdb2d59-6073-4cf8-ab45-c403e3e469b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:42.647504 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.647415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:42.786936 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.786910    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:01:42.788486 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:01:42.788455    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdb2d59_6073_4cf8_ab45_c403e3e469b8.slice/crio-c7b93d682a9c3c8a9ff3e642bbabcd7d43ccbf303dd931930b1b3a45ff862764 WatchSource:0}: Error finding container c7b93d682a9c3c8a9ff3e642bbabcd7d43ccbf303dd931930b1b3a45ff862764: Status 404 returned error can't find the container with id c7b93d682a9c3c8a9ff3e642bbabcd7d43ccbf303dd931930b1b3a45ff862764
Jul 17 08:01:42.928682 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:42.928593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" event={"ID":"bcdb2d59-6073-4cf8-ab45-c403e3e469b8","Type":"ContainerStarted","Data":"c7b93d682a9c3c8a9ff3e642bbabcd7d43ccbf303dd931930b1b3a45ff862764"}
Jul 17 08:01:43.284903 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.284880    2584 scope.go:117] "RemoveContainer" containerID="18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb"
Jul 17 08:01:43.934641 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.934543    2584 generic.go:356] "Generic (PLEG): container finished" podID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerID="2215bb55f81382f0127b942574247aaa200192b08ff7c8d7a01915c4108f3a9c" exitCode=6
Jul 17 08:01:43.934641 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.934621    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" event={"ID":"bcdb2d59-6073-4cf8-ab45-c403e3e469b8","Type":"ContainerDied","Data":"2215bb55f81382f0127b942574247aaa200192b08ff7c8d7a01915c4108f3a9c"}
Jul 17 08:01:43.935086 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.934821    2584 scope.go:117] "RemoveContainer" containerID="2215bb55f81382f0127b942574247aaa200192b08ff7c8d7a01915c4108f3a9c"
Jul 17 08:01:43.936582 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.936561    2584 generic.go:356] "Generic (PLEG): container finished" podID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerID="f1288e756981b4c84c5bd830897de8b76870b815e3a97699faac4ac8498c23ad" exitCode=6
Jul 17 08:01:43.936689 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.936596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" event={"ID":"11ff28ba-e5fa-455a-b41e-9d888b976aba","Type":"ContainerDied","Data":"f1288e756981b4c84c5bd830897de8b76870b815e3a97699faac4ac8498c23ad"}
Jul 17 08:01:43.936689 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.936635    2584 scope.go:117] "RemoveContainer" containerID="18b09c819d284405627672708de9d9ad4af425d2b6a01b7854938a7493dd70eb"
Jul 17 08:01:43.936881 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:43.936863    2584 scope.go:117] "RemoveContainer" containerID="f1288e756981b4c84c5bd830897de8b76870b815e3a97699faac4ac8498c23ad"
Jul 17 08:01:43.937083 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:43.937063    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv_odh-ai-gateway-infra(11ff28ba-e5fa-455a-b41e-9d888b976aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba"
Jul 17 08:01:44.423723 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.423695    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 08:01:44.427679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.427651    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 08:01:44.944363 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.944311    2584 generic.go:356] "Generic (PLEG): container finished" podID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerID="7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692" exitCode=6
Jul 17 08:01:44.944815 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.944403    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" event={"ID":"bcdb2d59-6073-4cf8-ab45-c403e3e469b8","Type":"ContainerDied","Data":"7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692"}
Jul 17 08:01:44.944815 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.944472    2584 scope.go:117] "RemoveContainer" containerID="2215bb55f81382f0127b942574247aaa200192b08ff7c8d7a01915c4108f3a9c"
Jul 17 08:01:44.944815 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.944669    2584 scope.go:117] "RemoveContainer" containerID="7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692"
Jul 17 08:01:44.945021 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:44.944923    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp_odh-ai-gateway-infra(bcdb2d59-6073-4cf8-ab45-c403e3e469b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8"
Jul 17 08:01:44.988999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:44.988916    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:45.129365 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.129322    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:45.207679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.207647    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca\") pod \"11ff28ba-e5fa-455a-b41e-9d888b976aba\" (UID: \"11ff28ba-e5fa-455a-b41e-9d888b976aba\") "
Jul 17 08:01:45.208009 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.207987    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11ff28ba-e5fa-455a-b41e-9d888b976aba" (UID: "11ff28ba-e5fa-455a-b41e-9d888b976aba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:01:45.308623 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.308593    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ff28ba-e5fa-455a-b41e-9d888b976aba-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:01:45.951765 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.951729    2584 scope.go:117] "RemoveContainer" containerID="7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692"
Jul 17 08:01:45.952178 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:45.951934    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp_odh-ai-gateway-infra(bcdb2d59-6073-4cf8-ab45-c403e3e469b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8"
Jul 17 08:01:45.952628 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.952608    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"
Jul 17 08:01:45.952747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.952634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv" event={"ID":"11ff28ba-e5fa-455a-b41e-9d888b976aba","Type":"ContainerDied","Data":"f2567123396026c6295975b68eae5ab12a88f41d14027a6fe63268b0c06d394f"}
Jul 17 08:01:45.952747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.952687    2584 scope.go:117] "RemoveContainer" containerID="f1288e756981b4c84c5bd830897de8b76870b815e3a97699faac4ac8498c23ad"
Jul 17 08:01:45.987978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.987938    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:45.995872 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:45.995838    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8jfwv"]
Jul 17 08:01:46.296113 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:46.296074    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" path="/var/lib/kubelet/pods/11ff28ba-e5fa-455a-b41e-9d888b976aba/volumes"
Jul 17 08:01:51.737870 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:51.737837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:01:56.415003 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.414958    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415273    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415284    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415306    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415311    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415384    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415392    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.415404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.415398    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:01:56.418361 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.418327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:01:56.426447 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.426422    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:01:56.497789 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.497748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4\" (UID: \"23c13174-e939-4f3c-a3a1-e965e7cf0358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:01:56.598723 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.598680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4\" (UID: \"23c13174-e939-4f3c-a3a1-e965e7cf0358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:01:56.599315 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.599294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4\" (UID: \"23c13174-e939-4f3c-a3a1-e965e7cf0358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:01:56.729041 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:56.728949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:01:57.067743 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:57.067713    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:01:57.069471 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:01:57.069442    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c13174_e939_4f3c_a3a1_e965e7cf0358.slice/crio-21d9a94ef3b3f55f6c8ec346969a33c316de876938d78ec992458c86484a624a WatchSource:0}: Error finding container 21d9a94ef3b3f55f6c8ec346969a33c316de876938d78ec992458c86484a624a: Status 404 returned error can't find the container with id 21d9a94ef3b3f55f6c8ec346969a33c316de876938d78ec992458c86484a624a
Jul 17 08:01:57.285207 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:57.285183    2584 scope.go:117] "RemoveContainer" containerID="7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692"
Jul 17 08:01:58.011497 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.011415    2584 generic.go:356] "Generic (PLEG): container finished" podID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerID="bf1b0546509aede3ceb21b67f40c316a2e663751b64a82ee42b9030534fec927" exitCode=6
Jul 17 08:01:58.011497 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.011484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" event={"ID":"bcdb2d59-6073-4cf8-ab45-c403e3e469b8","Type":"ContainerDied","Data":"bf1b0546509aede3ceb21b67f40c316a2e663751b64a82ee42b9030534fec927"}
Jul 17 08:01:58.011974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.011540    2584 scope.go:117] "RemoveContainer" containerID="7437a366f2932999ea5d15d9a8c8518b9dbe1b40a063a773fccafdff1bb91692"
Jul 17 08:01:58.011974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.011837    2584 scope.go:117] "RemoveContainer" containerID="bf1b0546509aede3ceb21b67f40c316a2e663751b64a82ee42b9030534fec927"
Jul 17 08:01:58.012093 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:58.012032    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp_odh-ai-gateway-infra(bcdb2d59-6073-4cf8-ab45-c403e3e469b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8"
Jul 17 08:01:58.013158 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.013137    2584 generic.go:356] "Generic (PLEG): container finished" podID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerID="ca6932c7a63ea60293216d3487d2c410f2043444b977b20a604cc2e4d8bf13a4" exitCode=6
Jul 17 08:01:58.013236 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.013196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" event={"ID":"23c13174-e939-4f3c-a3a1-e965e7cf0358","Type":"ContainerDied","Data":"ca6932c7a63ea60293216d3487d2c410f2043444b977b20a604cc2e4d8bf13a4"}
Jul 17 08:01:58.013236 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.013229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" event={"ID":"23c13174-e939-4f3c-a3a1-e965e7cf0358","Type":"ContainerStarted","Data":"21d9a94ef3b3f55f6c8ec346969a33c316de876938d78ec992458c86484a624a"}
Jul 17 08:01:58.013411 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:58.013366    2584 scope.go:117] "RemoveContainer" containerID="ca6932c7a63ea60293216d3487d2c410f2043444b977b20a604cc2e4d8bf13a4"
Jul 17 08:01:59.019971 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.019882    2584 generic.go:356] "Generic (PLEG): container finished" podID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerID="8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375" exitCode=6
Jul 17 08:01:59.020442 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.019971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" event={"ID":"23c13174-e939-4f3c-a3a1-e965e7cf0358","Type":"ContainerDied","Data":"8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375"}
Jul 17 08:01:59.020442 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.020035    2584 scope.go:117] "RemoveContainer" containerID="ca6932c7a63ea60293216d3487d2c410f2043444b977b20a604cc2e4d8bf13a4"
Jul 17 08:01:59.020442 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.020237    2584 scope.go:117] "RemoveContainer" containerID="8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375"
Jul 17 08:01:59.020581 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:01:59.020470    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4_odh-ai-gateway-infra(23c13174-e939-4f3c-a3a1-e965e7cf0358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358"
Jul 17 08:01:59.044467 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.044441    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:01:59.180522 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.180498    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:01:59.220144 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.220109    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca\") pod \"bcdb2d59-6073-4cf8-ab45-c403e3e469b8\" (UID: \"bcdb2d59-6073-4cf8-ab45-c403e3e469b8\") "
Jul 17 08:01:59.220462 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.220438    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bcdb2d59-6073-4cf8-ab45-c403e3e469b8" (UID: "bcdb2d59-6073-4cf8-ab45-c403e3e469b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:01:59.321406 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:01:59.321327    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb2d59-6073-4cf8-ab45-c403e3e469b8-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:00.027374 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.027324    2584 scope.go:117] "RemoveContainer" containerID="8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375"
Jul 17 08:02:00.027814 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:00.027564    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4_odh-ai-gateway-infra(23c13174-e939-4f3c-a3a1-e965e7cf0358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358"
Jul 17 08:02:00.028390 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.028366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp" event={"ID":"bcdb2d59-6073-4cf8-ab45-c403e3e469b8","Type":"ContainerDied","Data":"c7b93d682a9c3c8a9ff3e642bbabcd7d43ccbf303dd931930b1b3a45ff862764"}
Jul 17 08:02:00.028476 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.028433    2584 scope.go:117] "RemoveContainer" containerID="bf1b0546509aede3ceb21b67f40c316a2e663751b64a82ee42b9030534fec927"
Jul 17 08:02:00.028476 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.028377    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"
Jul 17 08:02:00.063433 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.063400    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:02:00.069978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.069948    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-wcbdp"]
Jul 17 08:02:00.290940 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:00.290897    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" path="/var/lib/kubelet/pods/bcdb2d59-6073-4cf8-ab45-c403e3e469b8/volumes"
Jul 17 08:02:04.115402 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115368    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115712    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115724    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115737    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115743    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ff28ba-e5fa-455a-b41e-9d888b976aba" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115755    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115761    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115768    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.115793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115773    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.116047 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115824    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.116047 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115833    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.116047 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.115841    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcdb2d59-6073-4cf8-ab45-c403e3e469b8" containerName="revoke-keys"
Jul 17 08:02:04.120202 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.120184    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:04.126913 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.126889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:04.168220 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.168186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m\" (UID: \"527582b1-37a5-4cc3-8e07-a85fba9b02e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:04.269432 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.269394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m\" (UID: \"527582b1-37a5-4cc3-8e07-a85fba9b02e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:04.270018 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.269997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m\" (UID: \"527582b1-37a5-4cc3-8e07-a85fba9b02e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:04.431879 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.431793    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:04.559918 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:04.559895    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:04.562370 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:02:04.562328    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527582b1_37a5_4cc3_8e07_a85fba9b02e3.slice/crio-49cef1730c5eff7d4bd61347b9ee793b383284ef758287f6a5c76adda95f2ddf WatchSource:0}: Error finding container 49cef1730c5eff7d4bd61347b9ee793b383284ef758287f6a5c76adda95f2ddf: Status 404 returned error can't find the container with id 49cef1730c5eff7d4bd61347b9ee793b383284ef758287f6a5c76adda95f2ddf
Jul 17 08:02:05.052300 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:05.052263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerStarted","Data":"4976eb22f78352870afcfd2873ef6d2881dced117aea56fb9d501a00bcf57ba7"}
Jul 17 08:02:05.052487 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:05.052371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerStarted","Data":"49cef1730c5eff7d4bd61347b9ee793b383284ef758287f6a5c76adda95f2ddf"}
Jul 17 08:02:05.070978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:05.070930    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" podStartSLOduration=1.070915988 podStartE2EDuration="1.070915988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:02:05.06894724 +0000 UTC m=+3021.322370061" watchObservedRunningTime="2026-07-17 08:02:05.070915988 +0000 UTC m=+3021.324338801"
Jul 17 08:02:05.934071 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:05.934030    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:02:06.058725 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:06.058691    2584 generic.go:356] "Generic (PLEG): container finished" podID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerID="4976eb22f78352870afcfd2873ef6d2881dced117aea56fb9d501a00bcf57ba7" exitCode=6
Jul 17 08:02:06.069313 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:06.058742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerDied","Data":"4976eb22f78352870afcfd2873ef6d2881dced117aea56fb9d501a00bcf57ba7"}
Jul 17 08:02:06.069313 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:06.058974    2584 scope.go:117] "RemoveContainer" containerID="4976eb22f78352870afcfd2873ef6d2881dced117aea56fb9d501a00bcf57ba7"
Jul 17 08:02:06.691981 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:06.691896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:02:07.064921 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:07.064887    2584 generic.go:356] "Generic (PLEG): container finished" podID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerID="eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924" exitCode=6
Jul 17 08:02:07.065409 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:07.064977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerDied","Data":"eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924"}
Jul 17 08:02:07.065409 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:07.065051    2584 scope.go:117] "RemoveContainer" containerID="4976eb22f78352870afcfd2873ef6d2881dced117aea56fb9d501a00bcf57ba7"
Jul 17 08:02:07.065409 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:07.065161    2584 scope.go:117] "RemoveContainer" containerID="eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924"
Jul 17 08:02:07.065409 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:07.065351    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m_odh-ai-gateway-infra(527582b1-37a5-4cc3-8e07-a85fba9b02e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3"
Jul 17 08:02:08.071458 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:08.071429    2584 scope.go:117] "RemoveContainer" containerID="eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924"
Jul 17 08:02:08.071834 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:08.071610    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m_odh-ai-gateway-infra(527582b1-37a5-4cc3-8e07-a85fba9b02e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3"
Jul 17 08:02:10.284997 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:10.284970    2584 scope.go:117] "RemoveContainer" containerID="8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375"
Jul 17 08:02:11.086778 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:11.086690    2584 generic.go:356] "Generic (PLEG): container finished" podID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerID="257e0cd7b5220cd64e60cd56d7b41c8859ce71ef822a21e827b528cec8e7d1d9" exitCode=6
Jul 17 08:02:11.086778 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:11.086763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" event={"ID":"23c13174-e939-4f3c-a3a1-e965e7cf0358","Type":"ContainerDied","Data":"257e0cd7b5220cd64e60cd56d7b41c8859ce71ef822a21e827b528cec8e7d1d9"}
Jul 17 08:02:11.087006 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:11.086835    2584 scope.go:117] "RemoveContainer" containerID="8390077c2e74579a10c659e8b67694df08f4bed96e98e79daf8039364876e375"
Jul 17 08:02:11.087206 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:11.087189    2584 scope.go:117] "RemoveContainer" containerID="257e0cd7b5220cd64e60cd56d7b41c8859ce71ef822a21e827b528cec8e7d1d9"
Jul 17 08:02:11.087449 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:11.087425    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4_odh-ai-gateway-infra(23c13174-e939-4f3c-a3a1-e965e7cf0358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358"
Jul 17 08:02:12.114700 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:12.114665    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:02:12.248417 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:12.248390    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:02:12.340593 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:12.340566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca\") pod \"23c13174-e939-4f3c-a3a1-e965e7cf0358\" (UID: \"23c13174-e939-4f3c-a3a1-e965e7cf0358\") "
Jul 17 08:02:12.340902 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:12.340882    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23c13174-e939-4f3c-a3a1-e965e7cf0358" (UID: "23c13174-e939-4f3c-a3a1-e965e7cf0358"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:02:12.441417 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:12.441321    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c13174-e939-4f3c-a3a1-e965e7cf0358-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:13.098014 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:13.097981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4" event={"ID":"23c13174-e939-4f3c-a3a1-e965e7cf0358","Type":"ContainerDied","Data":"21d9a94ef3b3f55f6c8ec346969a33c316de876938d78ec992458c86484a624a"}
Jul 17 08:02:13.098014 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:13.098013    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"
Jul 17 08:02:13.098227 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:13.098041    2584 scope.go:117] "RemoveContainer" containerID="257e0cd7b5220cd64e60cd56d7b41c8859ce71ef822a21e827b528cec8e7d1d9"
Jul 17 08:02:13.119481 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:13.119454    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:02:13.122219 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:13.122200    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-vz9r4"]
Jul 17 08:02:14.290577 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:14.290544    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" path="/var/lib/kubelet/pods/23c13174-e939-4f3c-a3a1-e965e7cf0358/volumes"
Jul 17 08:02:19.513984 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.513951    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:19.514470 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514451    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514473    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514491    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514498    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514524    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514533    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514814 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514612    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.514814 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.514624    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:19.518920 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.518899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:19.526392 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.526216    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:19.597616 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.597582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv\" (UID: \"2a68ffcf-a214-456e-b2b5-b4ff42d785dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:19.698263 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.698230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv\" (UID: \"2a68ffcf-a214-456e-b2b5-b4ff42d785dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:19.698866 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.698838    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv\" (UID: \"2a68ffcf-a214-456e-b2b5-b4ff42d785dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:19.830164 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.830133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:19.956017 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:19.955994    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:19.958731 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:02:19.958708    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a68ffcf_a214_456e_b2b5_b4ff42d785dc.slice/crio-7253daff44c14768f465a0f09aa838055bb450c3421b31fe999cde2ce479370d WatchSource:0}: Error finding container 7253daff44c14768f465a0f09aa838055bb450c3421b31fe999cde2ce479370d: Status 404 returned error can't find the container with id 7253daff44c14768f465a0f09aa838055bb450c3421b31fe999cde2ce479370d
Jul 17 08:02:20.132669 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:20.132595    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" event={"ID":"2a68ffcf-a214-456e-b2b5-b4ff42d785dc","Type":"ContainerStarted","Data":"7253daff44c14768f465a0f09aa838055bb450c3421b31fe999cde2ce479370d"}
Jul 17 08:02:21.138825 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:21.138792    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerID="e5ee7940dcf322b711b4460cc0a8342d46a38320942878741472dbc94084230d" exitCode=6
Jul 17 08:02:21.139207 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:21.138853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" event={"ID":"2a68ffcf-a214-456e-b2b5-b4ff42d785dc","Type":"ContainerDied","Data":"e5ee7940dcf322b711b4460cc0a8342d46a38320942878741472dbc94084230d"}
Jul 17 08:02:21.139207 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:21.139077    2584 scope.go:117] "RemoveContainer" containerID="e5ee7940dcf322b711b4460cc0a8342d46a38320942878741472dbc94084230d"
Jul 17 08:02:22.145450 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:22.145408    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerID="c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c" exitCode=6
Jul 17 08:02:22.145910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:22.145499    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" event={"ID":"2a68ffcf-a214-456e-b2b5-b4ff42d785dc","Type":"ContainerDied","Data":"c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c"}
Jul 17 08:02:22.145910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:22.145557    2584 scope.go:117] "RemoveContainer" containerID="e5ee7940dcf322b711b4460cc0a8342d46a38320942878741472dbc94084230d"
Jul 17 08:02:22.145910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:22.145793    2584 scope.go:117] "RemoveContainer" containerID="c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c"
Jul 17 08:02:22.146033 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:22.145985    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv_odh-ai-gateway-infra(2a68ffcf-a214-456e-b2b5-b4ff42d785dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc"
Jul 17 08:02:23.151302 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:23.151276    2584 scope.go:117] "RemoveContainer" containerID="c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c"
Jul 17 08:02:23.151687 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:23.151479    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv_odh-ai-gateway-infra(2a68ffcf-a214-456e-b2b5-b4ff42d785dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc"
Jul 17 08:02:23.286224 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:23.286175    2584 scope.go:117] "RemoveContainer" containerID="eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924"
Jul 17 08:02:24.157847 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:24.157815    2584 generic.go:356] "Generic (PLEG): container finished" podID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerID="1bf79b496f6034d64fcafeab2912e0bf5d6bd9bb4241d1390453d526db370272" exitCode=6
Jul 17 08:02:24.158217 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:24.157866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerDied","Data":"1bf79b496f6034d64fcafeab2912e0bf5d6bd9bb4241d1390453d526db370272"}
Jul 17 08:02:24.158217 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:24.157933    2584 scope.go:117] "RemoveContainer" containerID="eff06cb9edfa4845026d0a1b79f18389e2d1c062f2db01b8bd23367300619924"
Jul 17 08:02:24.158368 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:24.158224    2584 scope.go:117] "RemoveContainer" containerID="1bf79b496f6034d64fcafeab2912e0bf5d6bd9bb4241d1390453d526db370272"
Jul 17 08:02:24.158432 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:24.158414    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m_odh-ai-gateway-infra(527582b1-37a5-4cc3-8e07-a85fba9b02e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3"
Jul 17 08:02:24.740521 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:24.740485    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:02:25.207753 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:25.207708    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:25.338224 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:25.338193    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:25.450470 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:25.450436    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca\") pod \"527582b1-37a5-4cc3-8e07-a85fba9b02e3\" (UID: \"527582b1-37a5-4cc3-8e07-a85fba9b02e3\") "
Jul 17 08:02:25.450899 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:25.450868    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "527582b1-37a5-4cc3-8e07-a85fba9b02e3" (UID: "527582b1-37a5-4cc3-8e07-a85fba9b02e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:02:25.551481 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:25.551456    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/527582b1-37a5-4cc3-8e07-a85fba9b02e3-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:26.169096 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.169063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m" event={"ID":"527582b1-37a5-4cc3-8e07-a85fba9b02e3","Type":"ContainerDied","Data":"49cef1730c5eff7d4bd61347b9ee793b383284ef758287f6a5c76adda95f2ddf"}
Jul 17 08:02:26.169262 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.169097    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"
Jul 17 08:02:26.169262 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.169119    2584 scope.go:117] "RemoveContainer" containerID="1bf79b496f6034d64fcafeab2912e0bf5d6bd9bb4241d1390453d526db370272"
Jul 17 08:02:26.201711 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.201689    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:26.205523 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.205502    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-cqz4m"]
Jul 17 08:02:26.290181 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:26.290156    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" path="/var/lib/kubelet/pods/527582b1-37a5-4cc3-8e07-a85fba9b02e3/volumes"
Jul 17 08:02:32.382517 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382475    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382812    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382823    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382835    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382842    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382891    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c13174-e939-4f3c-a3a1-e965e7cf0358" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.382905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.382905    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:32.387361 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.387313    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:32.402590 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.402559    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:32.508832 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.508794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq\" (UID: \"546993a1-e2ab-43c9-ad95-bf1567ed0bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:32.610315 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.610278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq\" (UID: \"546993a1-e2ab-43c9-ad95-bf1567ed0bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:32.610945 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.610920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq\" (UID: \"546993a1-e2ab-43c9-ad95-bf1567ed0bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:32.697172 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.697086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:32.842121 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:32.842090    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:32.847310 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:02:32.847285    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546993a1_e2ab_43c9_ad95_bf1567ed0bd8.slice/crio-433d5c9e11ff072c765ee87cc973714bdfc80f2e437c85df5800b5e3e0672dab WatchSource:0}: Error finding container 433d5c9e11ff072c765ee87cc973714bdfc80f2e437c85df5800b5e3e0672dab: Status 404 returned error can't find the container with id 433d5c9e11ff072c765ee87cc973714bdfc80f2e437c85df5800b5e3e0672dab
Jul 17 08:02:33.201922 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:33.201887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerStarted","Data":"5f3ae3704bb8ae127bd6c09a8ee8500f016a39fbd684e2cd7bfb1c9267cc21a0"}
Jul 17 08:02:33.202072 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:33.201943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerStarted","Data":"433d5c9e11ff072c765ee87cc973714bdfc80f2e437c85df5800b5e3e0672dab"}
Jul 17 08:02:34.207645 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:34.207601    2584 generic.go:356] "Generic (PLEG): container finished" podID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerID="5f3ae3704bb8ae127bd6c09a8ee8500f016a39fbd684e2cd7bfb1c9267cc21a0" exitCode=6
Jul 17 08:02:34.208033 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:34.207660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerDied","Data":"5f3ae3704bb8ae127bd6c09a8ee8500f016a39fbd684e2cd7bfb1c9267cc21a0"}
Jul 17 08:02:34.208033 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:34.207896    2584 scope.go:117] "RemoveContainer" containerID="5f3ae3704bb8ae127bd6c09a8ee8500f016a39fbd684e2cd7bfb1c9267cc21a0"
Jul 17 08:02:35.214629 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:35.214546    2584 generic.go:356] "Generic (PLEG): container finished" podID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerID="9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b" exitCode=6
Jul 17 08:02:35.215050 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:35.214632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerDied","Data":"9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b"}
Jul 17 08:02:35.215050 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:35.214722    2584 scope.go:117] "RemoveContainer" containerID="5f3ae3704bb8ae127bd6c09a8ee8500f016a39fbd684e2cd7bfb1c9267cc21a0"
Jul 17 08:02:35.215050 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:35.214945    2584 scope.go:117] "RemoveContainer" containerID="9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b"
Jul 17 08:02:35.215168 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:35.215149    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq_odh-ai-gateway-infra(546993a1-e2ab-43c9-ad95-bf1567ed0bd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8"
Jul 17 08:02:36.221810 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:36.221770    2584 scope.go:117] "RemoveContainer" containerID="9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b"
Jul 17 08:02:36.222738 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:36.222701    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq_odh-ai-gateway-infra(546993a1-e2ab-43c9-ad95-bf1567ed0bd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8"
Jul 17 08:02:36.292288 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:36.292258    2584 scope.go:117] "RemoveContainer" containerID="c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c"
Jul 17 08:02:37.227663 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:37.227579    2584 generic.go:356] "Generic (PLEG): container finished" podID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerID="b9e0db965d876b24b7c3a515022351378f022c85864d306cb500e96793059e1a" exitCode=6
Jul 17 08:02:37.227663 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:37.227651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" event={"ID":"2a68ffcf-a214-456e-b2b5-b4ff42d785dc","Type":"ContainerDied","Data":"b9e0db965d876b24b7c3a515022351378f022c85864d306cb500e96793059e1a"}
Jul 17 08:02:37.228166 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:37.227713    2584 scope.go:117] "RemoveContainer" containerID="c11141221c5914837e88fd63b3513dfdb178a5edbf2eb4179082ee79ef0bf02c"
Jul 17 08:02:37.228166 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:37.228027    2584 scope.go:117] "RemoveContainer" containerID="b9e0db965d876b24b7c3a515022351378f022c85864d306cb500e96793059e1a"
Jul 17 08:02:37.228293 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:37.228273    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv_odh-ai-gateway-infra(2a68ffcf-a214-456e-b2b5-b4ff42d785dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc"
Jul 17 08:02:38.275922 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:38.275888    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:38.412303 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:38.412279    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:38.452973 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:38.452944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca\") pod \"2a68ffcf-a214-456e-b2b5-b4ff42d785dc\" (UID: \"2a68ffcf-a214-456e-b2b5-b4ff42d785dc\") "
Jul 17 08:02:38.453295 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:38.453275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a68ffcf-a214-456e-b2b5-b4ff42d785dc" (UID: "2a68ffcf-a214-456e-b2b5-b4ff42d785dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:02:38.554330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:38.554295    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a68ffcf-a214-456e-b2b5-b4ff42d785dc-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:39.240549 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.240515    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv" event={"ID":"2a68ffcf-a214-456e-b2b5-b4ff42d785dc","Type":"ContainerDied","Data":"7253daff44c14768f465a0f09aa838055bb450c3421b31fe999cde2ce479370d"}
Jul 17 08:02:39.240549 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.240540    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"
Jul 17 08:02:39.240804 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.240580    2584 scope.go:117] "RemoveContainer" containerID="b9e0db965d876b24b7c3a515022351378f022c85864d306cb500e96793059e1a"
Jul 17 08:02:39.264455 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.264427    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:39.268748 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.268726    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-rdcbv"]
Jul 17 08:02:39.738544 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:39.738492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:02:40.290404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:40.290372    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" path="/var/lib/kubelet/pods/2a68ffcf-a214-456e-b2b5-b4ff42d785dc/volumes"
Jul 17 08:02:43.114300 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114259    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114609    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114621    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114634    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114639    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114658    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114664    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114708    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:43.114741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.114725    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="527582b1-37a5-4cc3-8e07-a85fba9b02e3" containerName="revoke-keys"
Jul 17 08:02:43.118795 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.118775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:43.121957 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.121936    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kbw87\""
Jul 17 08:02:43.129849 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.129826    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:43.193143 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.193107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc46v\" (UniqueName: \"kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v\") pod \"kuadrant-operator-catalog-j47pd\" (UID: \"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b\") " pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:43.294362 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.294307    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc46v\" (UniqueName: \"kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v\") pod \"kuadrant-operator-catalog-j47pd\" (UID: \"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b\") " pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:43.313625 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.313592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc46v\" (UniqueName: \"kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v\") pod \"kuadrant-operator-catalog-j47pd\" (UID: \"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b\") " pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:43.427923 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.427835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:43.568171 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:43.568142    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:43.569799 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:02:43.569768    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec3f519_5944_4c9a_b8bc_3a3e6a95922b.slice/crio-7e7d205969d72cd71bc22ccf7bf5a87f0bc33857609cfeb07ece3934f3a0108a WatchSource:0}: Error finding container 7e7d205969d72cd71bc22ccf7bf5a87f0bc33857609cfeb07ece3934f3a0108a: Status 404 returned error can't find the container with id 7e7d205969d72cd71bc22ccf7bf5a87f0bc33857609cfeb07ece3934f3a0108a
Jul 17 08:02:44.265197 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:44.265162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" event={"ID":"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b","Type":"ContainerStarted","Data":"7e7d205969d72cd71bc22ccf7bf5a87f0bc33857609cfeb07ece3934f3a0108a"}
Jul 17 08:02:46.277124 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.277085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" event={"ID":"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b","Type":"ContainerStarted","Data":"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"}
Jul 17 08:02:46.297277 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.297219    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" podStartSLOduration=1.134451918 podStartE2EDuration="3.297203541s" totalImagesPullingTime="2.162751623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:02:46.295959173 +0000 UTC m=+3062.549381996" watchObservedRunningTime="2026-07-17 08:02:46.297203541 +0000 UTC m=+3062.550626352"
Jul 17 08:02:46.501730 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.501694    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:02:46.502099 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.502085    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:46.502146 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.502100    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:46.502181 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.502171    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a68ffcf-a214-456e-b2b5-b4ff42d785dc" containerName="revoke-keys"
Jul 17 08:02:46.505431 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.505412    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:02:46.523183 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.523148    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh\" (UID: \"eef31146-e46a-4c30-8e16-281a4c524609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:02:46.526908 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.526879    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:02:46.623780 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.623744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh\" (UID: \"eef31146-e46a-4c30-8e16-281a4c524609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:02:46.624426 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.624402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh\" (UID: \"eef31146-e46a-4c30-8e16-281a4c524609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:02:46.815479 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.815442    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:02:46.969196 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:46.969131    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:02:46.971249 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:02:46.971214    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef31146_e46a_4c30_8e16_281a4c524609.slice/crio-98c371a0960311d1aae62a0654e965f16f515c1461dfa7671a9c794cd43f1b0b WatchSource:0}: Error finding container 98c371a0960311d1aae62a0654e965f16f515c1461dfa7671a9c794cd43f1b0b: Status 404 returned error can't find the container with id 98c371a0960311d1aae62a0654e965f16f515c1461dfa7671a9c794cd43f1b0b
Jul 17 08:02:47.283584 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:47.283548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerStarted","Data":"442bd833232b7305ec5f29a90b95b619c22692b5d5eab46e9575ccf3a2b35169"}
Jul 17 08:02:47.284000 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:47.283603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerStarted","Data":"98c371a0960311d1aae62a0654e965f16f515c1461dfa7671a9c794cd43f1b0b"}
Jul 17 08:02:47.314489 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:47.314440    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" podStartSLOduration=1.314424267 podStartE2EDuration="1.314424267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:02:47.312773367 +0000 UTC m=+3063.566196186" watchObservedRunningTime="2026-07-17 08:02:47.314424267 +0000 UTC m=+3063.567847119"
Jul 17 08:02:48.289805 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:48.289771    2584 generic.go:356] "Generic (PLEG): container finished" podID="eef31146-e46a-4c30-8e16-281a4c524609" containerID="442bd833232b7305ec5f29a90b95b619c22692b5d5eab46e9575ccf3a2b35169" exitCode=6
Jul 17 08:02:48.290189 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:48.289823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerDied","Data":"442bd833232b7305ec5f29a90b95b619c22692b5d5eab46e9575ccf3a2b35169"}
Jul 17 08:02:48.290189 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:48.290000    2584 scope.go:117] "RemoveContainer" containerID="442bd833232b7305ec5f29a90b95b619c22692b5d5eab46e9575ccf3a2b35169"
Jul 17 08:02:49.284915 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:49.284887    2584 scope.go:117] "RemoveContainer" containerID="9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b"
Jul 17 08:02:49.298243 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:49.298213    2584 generic.go:356] "Generic (PLEG): container finished" podID="eef31146-e46a-4c30-8e16-281a4c524609" containerID="8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5" exitCode=6
Jul 17 08:02:49.298647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:49.298292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerDied","Data":"8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5"}
Jul 17 08:02:49.298647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:49.298369    2584 scope.go:117] "RemoveContainer" containerID="442bd833232b7305ec5f29a90b95b619c22692b5d5eab46e9575ccf3a2b35169"
Jul 17 08:02:49.298647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:49.298528    2584 scope.go:117] "RemoveContainer" containerID="8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5"
Jul 17 08:02:49.298810 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:49.298753    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh_odh-ai-gateway-infra(eef31146-e46a-4c30-8e16-281a4c524609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" podUID="eef31146-e46a-4c30-8e16-281a4c524609"
Jul 17 08:02:50.304285 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:50.304259    2584 scope.go:117] "RemoveContainer" containerID="8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5"
Jul 17 08:02:50.304885 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:50.304480    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh_odh-ai-gateway-infra(eef31146-e46a-4c30-8e16-281a4c524609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" podUID="eef31146-e46a-4c30-8e16-281a4c524609"
Jul 17 08:02:50.305739 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:50.305718    2584 generic.go:356] "Generic (PLEG): container finished" podID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerID="b2801aef87f971670c89adb3417d599dfaf502430a4cd9c9626bfaa3b43cee06" exitCode=6
Jul 17 08:02:50.305828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:50.305772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerDied","Data":"b2801aef87f971670c89adb3417d599dfaf502430a4cd9c9626bfaa3b43cee06"}
Jul 17 08:02:50.305828 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:50.305812    2584 scope.go:117] "RemoveContainer" containerID="9838c5b2adc275241932020b4d0cc91c547e6b3294261092d69b9d37542a393b"
Jul 17 08:02:50.306044 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:50.306027    2584 scope.go:117] "RemoveContainer" containerID="b2801aef87f971670c89adb3417d599dfaf502430a4cd9c9626bfaa3b43cee06"
Jul 17 08:02:50.306207 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:50.306187    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq_odh-ai-gateway-infra(546993a1-e2ab-43c9-ad95-bf1567ed0bd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8"
Jul 17 08:02:51.370275 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:51.370233    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:51.495932 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:51.495906    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:51.569463 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:51.569429    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca\") pod \"546993a1-e2ab-43c9-ad95-bf1567ed0bd8\" (UID: \"546993a1-e2ab-43c9-ad95-bf1567ed0bd8\") "
Jul 17 08:02:51.569826 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:51.569799    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "546993a1-e2ab-43c9-ad95-bf1567ed0bd8" (UID: "546993a1-e2ab-43c9-ad95-bf1567ed0bd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:02:51.670476 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:51.670381    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546993a1-e2ab-43c9-ad95-bf1567ed0bd8-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:52.317461 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:52.317429    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"
Jul 17 08:02:52.317635 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:52.317427    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq" event={"ID":"546993a1-e2ab-43c9-ad95-bf1567ed0bd8","Type":"ContainerDied","Data":"433d5c9e11ff072c765ee87cc973714bdfc80f2e437c85df5800b5e3e0672dab"}
Jul 17 08:02:52.317635 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:52.317574    2584 scope.go:117] "RemoveContainer" containerID="b2801aef87f971670c89adb3417d599dfaf502430a4cd9c9626bfaa3b43cee06"
Jul 17 08:02:52.338038 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:52.338006    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:52.343167 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:52.343142    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-lz9sq"]
Jul 17 08:02:53.428605 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:53.428570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:53.429010 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:53.428661    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:53.451474 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:53.451446    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:54.290281 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:54.290249    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" path="/var/lib/kubelet/pods/546993a1-e2ab-43c9-ad95-bf1567ed0bd8/volumes"
Jul 17 08:02:54.351570 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:54.351543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:54.417109 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:54.417078    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:56.340029 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:56.339964    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" podUID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" containerName="registry-server" containerID="cri-o://228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293" gracePeriod=2
Jul 17 08:02:56.594859 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:56.594804    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:56.709957 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:56.709919    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc46v\" (UniqueName: \"kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v\") pod \"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b\" (UID: \"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b\") "
Jul 17 08:02:56.712069 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:56.712044    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v" (OuterVolumeSpecName: "kube-api-access-fc46v") pod "9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" (UID: "9ec3f519-5944-4c9a-b8bc-3a3e6a95922b"). InnerVolumeSpecName "kube-api-access-fc46v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 08:02:56.810799 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:56.810762    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc46v\" (UniqueName: \"kubernetes.io/projected/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b-kube-api-access-fc46v\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:02:57.345833 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.345804    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" containerID="228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293" exitCode=0
Jul 17 08:02:57.346259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.345847    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j47pd"
Jul 17 08:02:57.346259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.345896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" event={"ID":"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b","Type":"ContainerDied","Data":"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"}
Jul 17 08:02:57.346259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.345948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j47pd" event={"ID":"9ec3f519-5944-4c9a-b8bc-3a3e6a95922b","Type":"ContainerDied","Data":"7e7d205969d72cd71bc22ccf7bf5a87f0bc33857609cfeb07ece3934f3a0108a"}
Jul 17 08:02:57.346259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.345969    2584 scope.go:117] "RemoveContainer" containerID="228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"
Jul 17 08:02:57.355726 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.355700    2584 scope.go:117] "RemoveContainer" containerID="228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"
Jul 17 08:02:57.355993 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:02:57.355974    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293\": container with ID starting with 228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293 not found: ID does not exist" containerID="228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"
Jul 17 08:02:57.356063 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.356008    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293"} err="failed to get container status \"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293\": rpc error: code = NotFound desc = could not find container \"228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293\": container with ID starting with 228f974cae64e61c4093627ff5caed24b6b6394d4e026ea5a44ad89be0da7293 not found: ID does not exist"
Jul 17 08:02:57.371831 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.371797    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:57.378481 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:57.378453    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j47pd"]
Jul 17 08:02:58.289666 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:58.289633    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" path="/var/lib/kubelet/pods/9ec3f519-5944-4c9a-b8bc-3a3e6a95922b/volumes"
Jul 17 08:02:58.447527 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:02:58.447489    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:03:04.180603 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180566    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180887    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180899    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180913    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180918    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180925    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180931    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180942    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" containerName="registry-server"
Jul 17 08:03:04.180974 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.180948    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" containerName="registry-server"
Jul 17 08:03:04.181236 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.181000    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.181236 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.181011    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ec3f519-5944-4c9a-b8bc-3a3e6a95922b" containerName="registry-server"
Jul 17 08:03:04.181236 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.181018    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:04.185117 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.185097    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:04.193713 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.193668    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:04.274211 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.274168    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg\" (UID: \"f8da4196-ad80-4c73-a0c4-d1a65480275e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:04.374784 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.374738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg\" (UID: \"f8da4196-ad80-4c73-a0c4-d1a65480275e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:04.375398 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.375375    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg\" (UID: \"f8da4196-ad80-4c73-a0c4-d1a65480275e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:04.496611 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.496529    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:04.654468 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:04.654443    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:04.656682 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:03:04.656657    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8da4196_ad80_4c73_a0c4_d1a65480275e.slice/crio-ddeb83042e512547292a6113c42734e4b93fecdba64f6c0b2bf490eab36eac97 WatchSource:0}: Error finding container ddeb83042e512547292a6113c42734e4b93fecdba64f6c0b2bf490eab36eac97: Status 404 returned error can't find the container with id ddeb83042e512547292a6113c42734e4b93fecdba64f6c0b2bf490eab36eac97
Jul 17 08:03:05.285197 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:05.285168    2584 scope.go:117] "RemoveContainer" containerID="8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5"
Jul 17 08:03:05.384586 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:05.384557    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerID="e05e3e85e29d3beff53f337cdff425e04f7df97dde0c4f3b6de494a09184722f" exitCode=6
Jul 17 08:03:05.384738 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:05.384605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerDied","Data":"e05e3e85e29d3beff53f337cdff425e04f7df97dde0c4f3b6de494a09184722f"}
Jul 17 08:03:05.384738 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:05.384642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerStarted","Data":"ddeb83042e512547292a6113c42734e4b93fecdba64f6c0b2bf490eab36eac97"}
Jul 17 08:03:05.384821 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:05.384789    2584 scope.go:117] "RemoveContainer" containerID="e05e3e85e29d3beff53f337cdff425e04f7df97dde0c4f3b6de494a09184722f"
Jul 17 08:03:06.390677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.390645    2584 generic.go:356] "Generic (PLEG): container finished" podID="eef31146-e46a-4c30-8e16-281a4c524609" containerID="520378b25faee3b0ef8ecf02b61b1b2cdf901a1a4cfabfdaf13aaf81774d5df2" exitCode=6
Jul 17 08:03:06.391144 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.390721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerDied","Data":"520378b25faee3b0ef8ecf02b61b1b2cdf901a1a4cfabfdaf13aaf81774d5df2"}
Jul 17 08:03:06.391144 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.390778    2584 scope.go:117] "RemoveContainer" containerID="8cde8d43e84fa6182d7f7aa7b63a1180a93a1ae167c9baa490e9e69a59e5b5a5"
Jul 17 08:03:06.391144 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.391096    2584 scope.go:117] "RemoveContainer" containerID="520378b25faee3b0ef8ecf02b61b1b2cdf901a1a4cfabfdaf13aaf81774d5df2"
Jul 17 08:03:06.391324 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:06.391304    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh_odh-ai-gateway-infra(eef31146-e46a-4c30-8e16-281a4c524609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" podUID="eef31146-e46a-4c30-8e16-281a4c524609"
Jul 17 08:03:06.392529 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.392507    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerID="31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845" exitCode=6
Jul 17 08:03:06.392620 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.392578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerDied","Data":"31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845"}
Jul 17 08:03:06.392721 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.392708    2584 scope.go:117] "RemoveContainer" containerID="31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845"
Jul 17 08:03:06.392864 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:06.392849    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg_odh-ai-gateway-infra(f8da4196-ad80-4c73-a0c4-d1a65480275e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e"
Jul 17 08:03:06.401635 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:06.401618    2584 scope.go:117] "RemoveContainer" containerID="e05e3e85e29d3beff53f337cdff425e04f7df97dde0c4f3b6de494a09184722f"
Jul 17 08:03:07.400531 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.400499    2584 scope.go:117] "RemoveContainer" containerID="31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845"
Jul 17 08:03:07.400925 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:07.400665    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg_odh-ai-gateway-infra(f8da4196-ad80-4c73-a0c4-d1a65480275e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e"
Jul 17 08:03:07.422085 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.422050    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:03:07.553752 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.553727    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:03:07.601324 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.601290    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca\") pod \"eef31146-e46a-4c30-8e16-281a4c524609\" (UID: \"eef31146-e46a-4c30-8e16-281a4c524609\") "
Jul 17 08:03:07.601642 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.601621    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eef31146-e46a-4c30-8e16-281a4c524609" (UID: "eef31146-e46a-4c30-8e16-281a4c524609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:03:07.702276 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:07.702185    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef31146-e46a-4c30-8e16-281a4c524609-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:03:08.407250 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:08.407219    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"
Jul 17 08:03:08.407250 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:08.407237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh" event={"ID":"eef31146-e46a-4c30-8e16-281a4c524609","Type":"ContainerDied","Data":"98c371a0960311d1aae62a0654e965f16f515c1461dfa7671a9c794cd43f1b0b"}
Jul 17 08:03:08.407743 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:08.407296    2584 scope.go:117] "RemoveContainer" containerID="520378b25faee3b0ef8ecf02b61b1b2cdf901a1a4cfabfdaf13aaf81774d5df2"
Jul 17 08:03:08.429372 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:08.429329    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:03:08.436725 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:08.436704    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-f2vjh"]
Jul 17 08:03:09.566715 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.566676    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567009    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567020    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567029    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567035    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567046    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567051    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567309 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567101    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567309 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567109    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567309 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567115    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef31146-e46a-4c30-8e16-281a4c524609" containerName="revoke-keys"
Jul 17 08:03:09.567309 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.567123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="546993a1-e2ab-43c9-ad95-bf1567ed0bd8" containerName="revoke-keys"
Jul 17 08:03:09.571377 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.571358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:09.581818 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.581783    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:09.720682 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.720641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt\" (UID: \"601269da-756b-4b63-ba75-daaa1276081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:09.821939 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.821851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt\" (UID: \"601269da-756b-4b63-ba75-daaa1276081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:09.822541 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.822524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt\" (UID: \"601269da-756b-4b63-ba75-daaa1276081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:09.881365 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:09.881310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:10.015691 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:10.015669    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:10.017589 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:03:10.017566    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601269da_756b_4b63_ba75_daaa1276081e.slice/crio-628030b875c3ee1f94e04f886d57a27ed251be19de7ebbf3b06c1f04f238807e WatchSource:0}: Error finding container 628030b875c3ee1f94e04f886d57a27ed251be19de7ebbf3b06c1f04f238807e: Status 404 returned error can't find the container with id 628030b875c3ee1f94e04f886d57a27ed251be19de7ebbf3b06c1f04f238807e
Jul 17 08:03:10.289947 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:10.289917    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef31146-e46a-4c30-8e16-281a4c524609" path="/var/lib/kubelet/pods/eef31146-e46a-4c30-8e16-281a4c524609/volumes"
Jul 17 08:03:10.418821 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:10.418787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerStarted","Data":"35fd2695af3f598c8454d7760fcfc049ae91db3bfe69cdce544f8fb8195ab93f"}
Jul 17 08:03:10.418990 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:10.418844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerStarted","Data":"628030b875c3ee1f94e04f886d57a27ed251be19de7ebbf3b06c1f04f238807e"}
Jul 17 08:03:10.442788 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:10.442739    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" podStartSLOduration=1.44272494 podStartE2EDuration="1.44272494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:03:10.44092044 +0000 UTC m=+3086.694343251" watchObservedRunningTime="2026-07-17 08:03:10.44272494 +0000 UTC m=+3086.696148062"
Jul 17 08:03:11.424765 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:11.424729    2584 generic.go:356] "Generic (PLEG): container finished" podID="601269da-756b-4b63-ba75-daaa1276081e" containerID="35fd2695af3f598c8454d7760fcfc049ae91db3bfe69cdce544f8fb8195ab93f" exitCode=6
Jul 17 08:03:11.425168 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:11.424821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerDied","Data":"35fd2695af3f598c8454d7760fcfc049ae91db3bfe69cdce544f8fb8195ab93f"}
Jul 17 08:03:11.425168 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:11.424978    2584 scope.go:117] "RemoveContainer" containerID="35fd2695af3f598c8454d7760fcfc049ae91db3bfe69cdce544f8fb8195ab93f"
Jul 17 08:03:12.431451 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:12.431332    2584 generic.go:356] "Generic (PLEG): container finished" podID="601269da-756b-4b63-ba75-daaa1276081e" containerID="3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b" exitCode=6
Jul 17 08:03:12.431451 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:12.431380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerDied","Data":"3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b"}
Jul 17 08:03:12.431451 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:12.431451    2584 scope.go:117] "RemoveContainer" containerID="35fd2695af3f598c8454d7760fcfc049ae91db3bfe69cdce544f8fb8195ab93f"
Jul 17 08:03:12.431944 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:12.431593    2584 scope.go:117] "RemoveContainer" containerID="3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b"
Jul 17 08:03:12.431944 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:12.431812    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt_odh-ai-gateway-infra(601269da-756b-4b63-ba75-daaa1276081e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" podUID="601269da-756b-4b63-ba75-daaa1276081e"
Jul 17 08:03:13.437573 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:13.437546    2584 scope.go:117] "RemoveContainer" containerID="3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b"
Jul 17 08:03:13.437973 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:13.437738    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt_odh-ai-gateway-infra(601269da-756b-4b63-ba75-daaa1276081e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" podUID="601269da-756b-4b63-ba75-daaa1276081e"
Jul 17 08:03:17.164258 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:17.164223    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:03:21.284733 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:21.284700    2584 scope.go:117] "RemoveContainer" containerID="31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845"
Jul 17 08:03:21.480681 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:21.480643    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerStarted","Data":"4ecdc5439660e8870480ed1feffedd42cf9f8a26b1bd031070b54b01cc0ba413"}
Jul 17 08:03:21.501887 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:21.501836    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" podStartSLOduration=17.501821832 podStartE2EDuration="17.501821832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:03:21.500723119 +0000 UTC m=+3097.754145930" watchObservedRunningTime="2026-07-17 08:03:21.501821832 +0000 UTC m=+3097.755244644"
Jul 17 08:03:22.487261 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:22.487226    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerID="4ecdc5439660e8870480ed1feffedd42cf9f8a26b1bd031070b54b01cc0ba413" exitCode=6
Jul 17 08:03:22.487687 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:22.487304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerDied","Data":"4ecdc5439660e8870480ed1feffedd42cf9f8a26b1bd031070b54b01cc0ba413"}
Jul 17 08:03:22.487687 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:22.487388    2584 scope.go:117] "RemoveContainer" containerID="31059c9cec697c774232e3ee3f94edb429f20f82574d3340373d3eef6948e845"
Jul 17 08:03:22.487687 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:22.487638    2584 scope.go:117] "RemoveContainer" containerID="4ecdc5439660e8870480ed1feffedd42cf9f8a26b1bd031070b54b01cc0ba413"
Jul 17 08:03:22.487855 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:22.487837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg_odh-ai-gateway-infra(f8da4196-ad80-4c73-a0c4-d1a65480275e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e"
Jul 17 08:03:22.525262 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:22.525230    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:23.630065 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:23.630038    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:23.747082 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:23.747004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca\") pod \"f8da4196-ad80-4c73-a0c4-d1a65480275e\" (UID: \"f8da4196-ad80-4c73-a0c4-d1a65480275e\") "
Jul 17 08:03:23.747332 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:23.747311    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8da4196-ad80-4c73-a0c4-d1a65480275e" (UID: "f8da4196-ad80-4c73-a0c4-d1a65480275e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:03:23.848282 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:23.848248    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8da4196-ad80-4c73-a0c4-d1a65480275e-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:03:24.499889 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:24.499856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg" event={"ID":"f8da4196-ad80-4c73-a0c4-d1a65480275e","Type":"ContainerDied","Data":"ddeb83042e512547292a6113c42734e4b93fecdba64f6c0b2bf490eab36eac97"}
Jul 17 08:03:24.499889 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:24.499879    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"
Jul 17 08:03:24.500126 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:24.499917    2584 scope.go:117] "RemoveContainer" containerID="4ecdc5439660e8870480ed1feffedd42cf9f8a26b1bd031070b54b01cc0ba413"
Jul 17 08:03:24.527372 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:24.527323    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:24.540853 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:24.540822    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-9t7gg"]
Jul 17 08:03:26.290022 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:26.289989    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" path="/var/lib/kubelet/pods/f8da4196-ad80-4c73-a0c4-d1a65480275e/volumes"
Jul 17 08:03:27.285404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:27.285368    2584 scope.go:117] "RemoveContainer" containerID="3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b"
Jul 17 08:03:28.197620 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:28.197536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:03:28.520681 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:28.520584    2584 generic.go:356] "Generic (PLEG): container finished" podID="601269da-756b-4b63-ba75-daaa1276081e" containerID="63afd4b9b53913a16a6371f04c279aab433be483d4edf78d7becdfa4cb3767cf" exitCode=6
Jul 17 08:03:28.520681 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:28.520635    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerDied","Data":"63afd4b9b53913a16a6371f04c279aab433be483d4edf78d7becdfa4cb3767cf"}
Jul 17 08:03:28.520901 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:28.520694    2584 scope.go:117] "RemoveContainer" containerID="3c4404a2a3b9a21ae5fd3c2a3c4d0f6e3fec83925a746edcbad1c0c5010b687b"
Jul 17 08:03:28.520997 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:28.520968    2584 scope.go:117] "RemoveContainer" containerID="63afd4b9b53913a16a6371f04c279aab433be483d4edf78d7becdfa4cb3767cf"
Jul 17 08:03:28.521184 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:28.521163    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt_odh-ai-gateway-infra(601269da-756b-4b63-ba75-daaa1276081e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" podUID="601269da-756b-4b63-ba75-daaa1276081e"
Jul 17 08:03:29.552495 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:29.552459    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:29.692242 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:29.692213    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:29.798990 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:29.798954    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca\") pod \"601269da-756b-4b63-ba75-daaa1276081e\" (UID: \"601269da-756b-4b63-ba75-daaa1276081e\") "
Jul 17 08:03:29.799397 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:29.799331    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "601269da-756b-4b63-ba75-daaa1276081e" (UID: "601269da-756b-4b63-ba75-daaa1276081e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:03:29.900571 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:29.900481    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601269da-756b-4b63-ba75-daaa1276081e-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:03:30.252494 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.252389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:03:30.532931 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.532847    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"
Jul 17 08:03:30.533105 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.532849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt" event={"ID":"601269da-756b-4b63-ba75-daaa1276081e","Type":"ContainerDied","Data":"628030b875c3ee1f94e04f886d57a27ed251be19de7ebbf3b06c1f04f238807e"}
Jul 17 08:03:30.533105 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.532978    2584 scope.go:117] "RemoveContainer" containerID="63afd4b9b53913a16a6371f04c279aab433be483d4edf78d7becdfa4cb3767cf"
Jul 17 08:03:30.551263 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.551237    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:30.555174 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:30.555149    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-4rtxt"]
Jul 17 08:03:31.735006 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:31.734970    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:03:32.290136 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.290104    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601269da-756b-4b63-ba75-daaa1276081e" path="/var/lib/kubelet/pods/601269da-756b-4b63-ba75-daaa1276081e/volumes"
Jul 17 08:03:32.438625 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438587    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:32.438934 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438921    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.438978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438936    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.438978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438947    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.438978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438953    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.438978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438969    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.438978 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438975    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438982    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438990    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.438996    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.439001    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.439056    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.439066    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.439076    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:32.439129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.439082    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:32.443505 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.443485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:32.446140 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.446119    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:03:32.450792 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.450767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:32.622768 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.622676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w74vv\" (UID: \"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:32.723318 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.723284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w74vv\" (UID: \"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:32.723904 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.723883    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w74vv\" (UID: \"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:32.755669 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.755644    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:32.894594 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:32.889550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:33.550023 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:33.549994    2584 generic.go:356] "Generic (PLEG): container finished" podID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerID="310db58304cc3b88283c00007664b4ff5e98a77584a6cb11a722ec48c64ab8e8" exitCode=6
Jul 17 08:03:33.550161 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:33.550048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" event={"ID":"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b","Type":"ContainerDied","Data":"310db58304cc3b88283c00007664b4ff5e98a77584a6cb11a722ec48c64ab8e8"}
Jul 17 08:03:33.550161 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:33.550099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" event={"ID":"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b","Type":"ContainerStarted","Data":"7ed8d530f7f290ad66685f45987c9b8d1677f95f4dd06c1c4943bb18c40b153f"}
Jul 17 08:03:33.550248 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:33.550194    2584 scope.go:117] "RemoveContainer" containerID="310db58304cc3b88283c00007664b4ff5e98a77584a6cb11a722ec48c64ab8e8"
Jul 17 08:03:34.557528 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:34.557495    2584 generic.go:356] "Generic (PLEG): container finished" podID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerID="2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f" exitCode=6
Jul 17 08:03:34.557967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:34.557565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" event={"ID":"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b","Type":"ContainerDied","Data":"2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f"}
Jul 17 08:03:34.557967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:34.557608    2584 scope.go:117] "RemoveContainer" containerID="310db58304cc3b88283c00007664b4ff5e98a77584a6cb11a722ec48c64ab8e8"
Jul 17 08:03:34.557967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:34.557767    2584 scope.go:117] "RemoveContainer" containerID="2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f"
Jul 17 08:03:34.558107 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:34.557968    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w74vv_odh-ai-gateway-infra(86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b"
Jul 17 08:03:35.563878 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:35.563849    2584 scope.go:117] "RemoveContainer" containerID="2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f"
Jul 17 08:03:35.564384 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:35.564077    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w74vv_odh-ai-gateway-infra(86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b"
Jul 17 08:03:40.755440 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.755408    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:03:40.755791 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.755722    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:40.755791 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.755733    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:40.755874 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.755798    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="601269da-756b-4b63-ba75-daaa1276081e" containerName="revoke-keys"
Jul 17 08:03:40.755874 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.755809    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8da4196-ad80-4c73-a0c4-d1a65480275e" containerName="revoke-keys"
Jul 17 08:03:40.759993 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.759974    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.762776 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.762756    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d805bd\"/\"default-dockercfg-fzqfz\""
Jul 17 08:03:40.763112 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.763091    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d805bd\"/\"openshift-service-ca.crt\""
Jul 17 08:03:40.763899 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.763881    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d805bd\"/\"sub-test-model-d805bd-kserve-self-signed-certs\""
Jul 17 08:03:40.763953 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.763894    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d805bd\"/\"kube-root-ca.crt\""
Jul 17 08:03:40.769107 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.769084    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:03:40.779352 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779312    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.779452 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wkkf\" (UniqueName: \"kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.779672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779487    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.779672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.779672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.779672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.779610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.827666 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.827635    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:03:40.880684 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wkkf\" (UniqueName: \"kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.880863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.880863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.880863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.880863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.880863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.880807    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.881133 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.881112    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.881197 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.881135    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.881250 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.881223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.883085 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.883066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.883251 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.883235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:40.891716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:40.891688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wkkf\" (UniqueName: \"kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf\") pod \"sub-test-model-d805bd-kserve-b6f75958f-hjvxr\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") " pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:41.069835 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:41.069798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:41.216571 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:41.216546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:03:41.218063 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:03:41.218033    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23b89554_5247_4eb2_bfd4_7cf5964a93af.slice/crio-667c2733e07f66128c61d5492e44ba749c00f3d09cd3d20feb50a915cc7eb094 WatchSource:0}: Error finding container 667c2733e07f66128c61d5492e44ba749c00f3d09cd3d20feb50a915cc7eb094: Status 404 returned error can't find the container with id 667c2733e07f66128c61d5492e44ba749c00f3d09cd3d20feb50a915cc7eb094
Jul 17 08:03:41.592320 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:41.592285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" event={"ID":"23b89554-5247-4eb2-bfd4-7cf5964a93af","Type":"ContainerStarted","Data":"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"}
Jul 17 08:03:41.592494 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:41.592328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" event={"ID":"23b89554-5247-4eb2-bfd4-7cf5964a93af","Type":"ContainerStarted","Data":"667c2733e07f66128c61d5492e44ba749c00f3d09cd3d20feb50a915cc7eb094"}
Jul 17 08:03:41.618017 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:41.617965    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" podStartSLOduration=1.617949088 podStartE2EDuration="1.617949088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:03:41.615321267 +0000 UTC m=+3117.868744080" watchObservedRunningTime="2026-07-17 08:03:41.617949088 +0000 UTC m=+3117.871371894"
Jul 17 08:03:47.285032 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:47.284999    2584 scope.go:117] "RemoveContainer" containerID="2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f"
Jul 17 08:03:47.339201 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:47.339173    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:03:48.625706 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:48.625673    2584 generic.go:356] "Generic (PLEG): container finished" podID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerID="f66098c1ad7169671a06fe606ae69bf7edf194de3b2662175b2f550f45a0b9e4" exitCode=6
Jul 17 08:03:48.625706 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:48.625710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" event={"ID":"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b","Type":"ContainerDied","Data":"f66098c1ad7169671a06fe606ae69bf7edf194de3b2662175b2f550f45a0b9e4"}
Jul 17 08:03:48.626244 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:48.625756    2584 scope.go:117] "RemoveContainer" containerID="2a81dd5bb98e43d9a045e849c6860687fe0fb721d83cb595823550a09678134f"
Jul 17 08:03:48.626244 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:48.626055    2584 scope.go:117] "RemoveContainer" containerID="f66098c1ad7169671a06fe606ae69bf7edf194de3b2662175b2f550f45a0b9e4"
Jul 17 08:03:48.626353 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:03:48.626286    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w74vv_odh-ai-gateway-infra(86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b"
Jul 17 08:03:49.658994 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:49.658959    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:49.791287 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:49.791261    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:49.841456 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:49.841423    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca\") pod \"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b\" (UID: \"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b\") "
Jul 17 08:03:49.841793 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:49.841769    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" (UID: "86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:03:49.942111 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:49.942020    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:03:50.636688 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:50.636609    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"
Jul 17 08:03:50.636870 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:50.636604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv" event={"ID":"86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b","Type":"ContainerDied","Data":"7ed8d530f7f290ad66685f45987c9b8d1677f95f4dd06c1c4943bb18c40b153f"}
Jul 17 08:03:50.636870 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:50.636750    2584 scope.go:117] "RemoveContainer" containerID="f66098c1ad7169671a06fe606ae69bf7edf194de3b2662175b2f550f45a0b9e4"
Jul 17 08:03:50.656760 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:50.656701    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:50.658681 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:50.658655    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w74vv"]
Jul 17 08:03:51.069947 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:51.069906    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:51.069947 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:51.069956    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:51.082665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:51.082640    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:51.659243 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:51.659209    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:03:52.289983 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:03:52.289949    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" path="/var/lib/kubelet/pods/86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b/volumes"
Jul 17 08:04:04.363796 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:04.363758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:08.360809 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:08.360690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:08.643405 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:08.643303    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:04:20.393374 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:20.393318    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:20.698823 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:20.698735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:22.498507 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498472    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498801    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498813    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498825    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498831    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498840    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.498884 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498846    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.499082 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.499082 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.498905    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:22.502170 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.502153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:22.504785 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.504764    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:04:22.511589 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.511561    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:22.616060 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.616022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-ntls2\" (UID: \"c66a075a-9780-428f-9ada-6ba56e558759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:22.717464 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.717429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-ntls2\" (UID: \"c66a075a-9780-428f-9ada-6ba56e558759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:22.718068 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.718048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-ntls2\" (UID: \"c66a075a-9780-428f-9ada-6ba56e558759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:22.813129 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.813097    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:22.939955 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:22.939931    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:22.942246 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:04:22.942215    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66a075a_9780_428f_9ada_6ba56e558759.slice/crio-f559ca68205d36dd6814c805a221daecc24b3ae671672497b156dffb58f00c3c WatchSource:0}: Error finding container f559ca68205d36dd6814c805a221daecc24b3ae671672497b156dffb58f00c3c: Status 404 returned error can't find the container with id f559ca68205d36dd6814c805a221daecc24b3ae671672497b156dffb58f00c3c
Jul 17 08:04:23.794494 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:23.794454    2584 generic.go:356] "Generic (PLEG): container finished" podID="c66a075a-9780-428f-9ada-6ba56e558759" containerID="2fe20d72c92910c5247ea016f6669c9550845133014e9793ee67bb5b225e9ed8" exitCode=6
Jul 17 08:04:23.794924 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:23.794545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerDied","Data":"2fe20d72c92910c5247ea016f6669c9550845133014e9793ee67bb5b225e9ed8"}
Jul 17 08:04:23.794924 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:23.794609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerStarted","Data":"f559ca68205d36dd6814c805a221daecc24b3ae671672497b156dffb58f00c3c"}
Jul 17 08:04:23.794924 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:23.794678    2584 scope.go:117] "RemoveContainer" containerID="2fe20d72c92910c5247ea016f6669c9550845133014e9793ee67bb5b225e9ed8"
Jul 17 08:04:24.759646 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:24.759607    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:04:24.800535 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:24.800502    2584 generic.go:356] "Generic (PLEG): container finished" podID="c66a075a-9780-428f-9ada-6ba56e558759" containerID="795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e" exitCode=6
Jul 17 08:04:24.800910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:24.800580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerDied","Data":"795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e"}
Jul 17 08:04:24.800910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:24.800638    2584 scope.go:117] "RemoveContainer" containerID="2fe20d72c92910c5247ea016f6669c9550845133014e9793ee67bb5b225e9ed8"
Jul 17 08:04:24.800910 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:24.800722    2584 scope.go:117] "RemoveContainer" containerID="795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e"
Jul 17 08:04:24.801055 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:24.800928    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-ntls2_odh-ai-gateway-infra(c66a075a-9780-428f-9ada-6ba56e558759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" podUID="c66a075a-9780-428f-9ada-6ba56e558759"
Jul 17 08:04:25.807244 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:25.807218    2584 scope.go:117] "RemoveContainer" containerID="795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e"
Jul 17 08:04:25.807645 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:25.807433    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-ntls2_odh-ai-gateway-infra(c66a075a-9780-428f-9ada-6ba56e558759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" podUID="c66a075a-9780-428f-9ada-6ba56e558759"
Jul 17 08:04:36.595112 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.595078    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:36.595537 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.595521    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="86411a9e-bdd3-4c44-bec5-bd6ae5e38f5b" containerName="revoke-keys"
Jul 17 08:04:36.598455 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.598439    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:36.607407 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.607240    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:36.738159 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.738123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf\" (UID: \"9132f0f9-12ba-41a9-9bac-5cb119b75a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:36.838859 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.838830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf\" (UID: \"9132f0f9-12ba-41a9-9bac-5cb119b75a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:36.839451 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.839433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf\" (UID: \"9132f0f9-12ba-41a9-9bac-5cb119b75a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:36.909483 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:36.909411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:37.033791 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.033769    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:37.036660 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:04:37.036633    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9132f0f9_12ba_41a9_9bac_5cb119b75a88.slice/crio-a6d6e74724a5b3704cd7ac0c6988d873b8994e1ef82a812d3c7781e622856ba3 WatchSource:0}: Error finding container a6d6e74724a5b3704cd7ac0c6988d873b8994e1ef82a812d3c7781e622856ba3: Status 404 returned error can't find the container with id a6d6e74724a5b3704cd7ac0c6988d873b8994e1ef82a812d3c7781e622856ba3
Jul 17 08:04:37.284900 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.284870    2584 scope.go:117] "RemoveContainer" containerID="795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e"
Jul 17 08:04:37.861183 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.861148    2584 generic.go:356] "Generic (PLEG): container finished" podID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerID="171cda5b5951df5efcdb1a441e8bdd1ae2971408e9e597422dab3b746a5774cb" exitCode=6
Jul 17 08:04:37.861665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.861196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" event={"ID":"9132f0f9-12ba-41a9-9bac-5cb119b75a88","Type":"ContainerDied","Data":"171cda5b5951df5efcdb1a441e8bdd1ae2971408e9e597422dab3b746a5774cb"}
Jul 17 08:04:37.861665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.861251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" event={"ID":"9132f0f9-12ba-41a9-9bac-5cb119b75a88","Type":"ContainerStarted","Data":"a6d6e74724a5b3704cd7ac0c6988d873b8994e1ef82a812d3c7781e622856ba3"}
Jul 17 08:04:37.861665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.861534    2584 scope.go:117] "RemoveContainer" containerID="171cda5b5951df5efcdb1a441e8bdd1ae2971408e9e597422dab3b746a5774cb"
Jul 17 08:04:37.863087 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.863065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerStarted","Data":"4a4f3a381270b296e70b54403985072182598221b54a02f92faaada9fc27ee26"}
Jul 17 08:04:37.895659 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:37.895618    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" podStartSLOduration=15.89560544 podStartE2EDuration="15.89560544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:04:37.894817338 +0000 UTC m=+3174.148240171" watchObservedRunningTime="2026-07-17 08:04:37.89560544 +0000 UTC m=+3174.149028252"
Jul 17 08:04:38.869283 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.869249    2584 generic.go:356] "Generic (PLEG): container finished" podID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerID="fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0" exitCode=6
Jul 17 08:04:38.869782 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.869357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" event={"ID":"9132f0f9-12ba-41a9-9bac-5cb119b75a88","Type":"ContainerDied","Data":"fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0"}
Jul 17 08:04:38.869782 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.869431    2584 scope.go:117] "RemoveContainer" containerID="171cda5b5951df5efcdb1a441e8bdd1ae2971408e9e597422dab3b746a5774cb"
Jul 17 08:04:38.869782 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.869603    2584 scope.go:117] "RemoveContainer" containerID="fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0"
Jul 17 08:04:38.869921 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:38.869811    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf_odh-ai-gateway-infra(9132f0f9-12ba-41a9-9bac-5cb119b75a88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88"
Jul 17 08:04:38.871220 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.871199    2584 generic.go:356] "Generic (PLEG): container finished" podID="c66a075a-9780-428f-9ada-6ba56e558759" containerID="4a4f3a381270b296e70b54403985072182598221b54a02f92faaada9fc27ee26" exitCode=6
Jul 17 08:04:38.871350 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.871256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerDied","Data":"4a4f3a381270b296e70b54403985072182598221b54a02f92faaada9fc27ee26"}
Jul 17 08:04:38.871555 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.871540    2584 scope.go:117] "RemoveContainer" containerID="4a4f3a381270b296e70b54403985072182598221b54a02f92faaada9fc27ee26"
Jul 17 08:04:38.871730 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:38.871713    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-ntls2_odh-ai-gateway-infra(c66a075a-9780-428f-9ada-6ba56e558759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" podUID="c66a075a-9780-428f-9ada-6ba56e558759"
Jul 17 08:04:38.880369 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.880322    2584 scope.go:117] "RemoveContainer" containerID="795c61dfc1b5e669d15ee788048ec43055a023ea35e69b8bc5d0e7eac7c7249e"
Jul 17 08:04:38.906625 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:38.906600    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:39.433035 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:39.432988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:39.497400 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:39.497369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:39.877989 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:39.877962    2584 scope.go:117] "RemoveContainer" containerID="fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0"
Jul 17 08:04:39.878436 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:39.878143    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf_odh-ai-gateway-infra(9132f0f9-12ba-41a9-9bac-5cb119b75a88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88"
Jul 17 08:04:40.012648 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.012621    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:40.168467 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.168381    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca\") pod \"c66a075a-9780-428f-9ada-6ba56e558759\" (UID: \"c66a075a-9780-428f-9ada-6ba56e558759\") "
Jul 17 08:04:40.168794 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.168763    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c66a075a-9780-428f-9ada-6ba56e558759" (UID: "c66a075a-9780-428f-9ada-6ba56e558759"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:04:40.269803 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.269776    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66a075a-9780-428f-9ada-6ba56e558759-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:04:40.886239 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.886215    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"
Jul 17 08:04:40.886708 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.886210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2" event={"ID":"c66a075a-9780-428f-9ada-6ba56e558759","Type":"ContainerDied","Data":"f559ca68205d36dd6814c805a221daecc24b3ae671672497b156dffb58f00c3c"}
Jul 17 08:04:40.886708 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.886365    2584 scope.go:117] "RemoveContainer" containerID="4a4f3a381270b296e70b54403985072182598221b54a02f92faaada9fc27ee26"
Jul 17 08:04:40.906027 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.906000    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:40.910896 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:40.910875    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-ntls2"]
Jul 17 08:04:42.290652 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:42.290618    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66a075a-9780-428f-9ada-6ba56e558759" path="/var/lib/kubelet/pods/c66a075a-9780-428f-9ada-6ba56e558759/volumes"
Jul 17 08:04:44.530814 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:44.530777    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:04:51.249607 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:51.249571    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:04:51.272370 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:51.272321    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:52.189022 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:52.188947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:04:55.285069 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:55.285037    2584 scope.go:117] "RemoveContainer" containerID="fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0"
Jul 17 08:04:55.955602 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:55.955574    2584 generic.go:356] "Generic (PLEG): container finished" podID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerID="660bb788a9ce0cdd828ed46f4e47609aac4efe16207700116554e73d0f7c4509" exitCode=6
Jul 17 08:04:55.955734 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:55.955653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" event={"ID":"9132f0f9-12ba-41a9-9bac-5cb119b75a88","Type":"ContainerDied","Data":"660bb788a9ce0cdd828ed46f4e47609aac4efe16207700116554e73d0f7c4509"}
Jul 17 08:04:55.955734 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:55.955726    2584 scope.go:117] "RemoveContainer" containerID="fd32f373fa4106c785ad9fc466e9b21273c8296d2615b4e4b92e870d988202f0"
Jul 17 08:04:55.956040 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:55.956021    2584 scope.go:117] "RemoveContainer" containerID="660bb788a9ce0cdd828ed46f4e47609aac4efe16207700116554e73d0f7c4509"
Jul 17 08:04:55.956264 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:04:55.956245    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf_odh-ai-gateway-infra(9132f0f9-12ba-41a9-9bac-5cb119b75a88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88"
Jul 17 08:04:56.983162 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:56.983129    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:57.116195 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.116169    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:57.212075 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.212043    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca\") pod \"9132f0f9-12ba-41a9-9bac-5cb119b75a88\" (UID: \"9132f0f9-12ba-41a9-9bac-5cb119b75a88\") "
Jul 17 08:04:57.212433 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.212411    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9132f0f9-12ba-41a9-9bac-5cb119b75a88" (UID: "9132f0f9-12ba-41a9-9bac-5cb119b75a88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:04:57.312772 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.312725    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9132f0f9-12ba-41a9-9bac-5cb119b75a88-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:04:57.830511 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.830467    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:04:57.968524 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.968494    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"
Jul 17 08:04:57.968718 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.968531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf" event={"ID":"9132f0f9-12ba-41a9-9bac-5cb119b75a88","Type":"ContainerDied","Data":"a6d6e74724a5b3704cd7ac0c6988d873b8994e1ef82a812d3c7781e622856ba3"}
Jul 17 08:04:57.968718 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.968612    2584 scope.go:117] "RemoveContainer" containerID="660bb788a9ce0cdd828ed46f4e47609aac4efe16207700116554e73d0f7c4509"
Jul 17 08:04:57.989600 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.989567    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:57.995288 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:57.995266    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-28cnf"]
Jul 17 08:04:58.290199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:04:58.290168    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" path="/var/lib/kubelet/pods/9132f0f9-12ba-41a9-9bac-5cb119b75a88/volumes"
Jul 17 08:05:06.269468 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.269421    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" podUID="23b89554-5247-4eb2-bfd4-7cf5964a93af" containerName="main" containerID="cri-o://c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71" gracePeriod=45
Jul 17 08:05:06.511738 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.511712    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:05:06.590141 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590106    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590389 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590171    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wkkf\" (UniqueName: \"kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590389 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590200    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590389 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590236    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590389 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590389 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590335    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs\") pod \"23b89554-5247-4eb2-bfd4-7cf5964a93af\" (UID: \"23b89554-5247-4eb2-bfd4-7cf5964a93af\") "
Jul 17 08:05:06.590647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590479    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 08:05:06.590647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590527    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home" (OuterVolumeSpecName: "home") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 08:05:06.590647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590564    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache" (OuterVolumeSpecName: "model-cache") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 08:05:06.590647 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590645    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-tmp-dir\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:06.590788 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590658    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-home\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:06.590788 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.590667    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-model-cache\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:06.592578 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.592550    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 08:05:06.592686 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.592593    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf" (OuterVolumeSpecName: "kube-api-access-7wkkf") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "kube-api-access-7wkkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 08:05:06.592686 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.592612    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm" (OuterVolumeSpecName: "dshm") pod "23b89554-5247-4eb2-bfd4-7cf5964a93af" (UID: "23b89554-5247-4eb2-bfd4-7cf5964a93af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 08:05:06.691283 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.691241    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b89554-5247-4eb2-bfd4-7cf5964a93af-tls-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:06.691283 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.691273    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wkkf\" (UniqueName: \"kubernetes.io/projected/23b89554-5247-4eb2-bfd4-7cf5964a93af-kube-api-access-7wkkf\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:06.691283 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:06.691285    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b89554-5247-4eb2-bfd4-7cf5964a93af-dshm\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:07.018684 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.018594    2584 generic.go:356] "Generic (PLEG): container finished" podID="23b89554-5247-4eb2-bfd4-7cf5964a93af" containerID="c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71" exitCode=0
Jul 17 08:05:07.018684 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.018637    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"
Jul 17 08:05:07.018877 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.018678    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" event={"ID":"23b89554-5247-4eb2-bfd4-7cf5964a93af","Type":"ContainerDied","Data":"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"}
Jul 17 08:05:07.018877 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.018723    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr" event={"ID":"23b89554-5247-4eb2-bfd4-7cf5964a93af","Type":"ContainerDied","Data":"667c2733e07f66128c61d5492e44ba749c00f3d09cd3d20feb50a915cc7eb094"}
Jul 17 08:05:07.018877 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.018741    2584 scope.go:117] "RemoveContainer" containerID="c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"
Jul 17 08:05:07.027443 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.027427    2584 scope.go:117] "RemoveContainer" containerID="c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"
Jul 17 08:05:07.027709 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:07.027683    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71\": container with ID starting with c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71 not found: ID does not exist" containerID="c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"
Jul 17 08:05:07.027808 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.027713    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71"} err="failed to get container status \"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71\": rpc error: code = NotFound desc = could not find container \"c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71\": container with ID starting with c49d807b9e6e6032fc415fb9db84fec095f098668b13151dbd1fef3e9c86ba71 not found: ID does not exist"
Jul 17 08:05:07.040493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.040456    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:05:07.044206 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:07.044185    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d805bd/sub-test-model-d805bd-kserve-b6f75958f-hjvxr"]
Jul 17 08:05:08.290287 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:08.290242    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b89554-5247-4eb2-bfd4-7cf5964a93af" path="/var/lib/kubelet/pods/23b89554-5247-4eb2-bfd4-7cf5964a93af/volumes"
Jul 17 08:05:09.363783 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:09.363742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:12.109319 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.109284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:12.210220 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.210187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:12.545007 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.544971    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:12.545362 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545325    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.545362 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545358    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545382    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545391    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545401    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545409    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545419    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545426    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545437    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b89554-5247-4eb2-bfd4-7cf5964a93af" containerName="main"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545445    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b89554-5247-4eb2-bfd4-7cf5964a93af" containerName="main"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545459    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545467    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.545519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545497    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.546073 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545581    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.546073 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545596    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:12.546073 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545605    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.546073 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545614    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b89554-5247-4eb2-bfd4-7cf5964a93af" containerName="main"
Jul 17 08:05:12.546073 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.545626    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:12.549965 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.549943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:12.552311 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.552291    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:05:12.555681 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.555656    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:12.646618 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.646585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq\" (UID: \"533bc8b5-90c5-4bfd-814c-3c6e400f2f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:12.747355 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.747300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq\" (UID: \"533bc8b5-90c5-4bfd-814c-3c6e400f2f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:12.747926 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.747907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq\" (UID: \"533bc8b5-90c5-4bfd-814c-3c6e400f2f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:12.862225 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.862141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:12.989321 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:12.989296    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:12.992059 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:05:12.992033    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533bc8b5_90c5_4bfd_814c_3c6e400f2f15.slice/crio-5634d8dafc1f6a6cd8a2c3afc54796510e1b21d75140f9370c6a89289f91ebef WatchSource:0}: Error finding container 5634d8dafc1f6a6cd8a2c3afc54796510e1b21d75140f9370c6a89289f91ebef: Status 404 returned error can't find the container with id 5634d8dafc1f6a6cd8a2c3afc54796510e1b21d75140f9370c6a89289f91ebef
Jul 17 08:05:13.033151 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:13.033122    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:05:13.051290 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:13.051260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" event={"ID":"533bc8b5-90c5-4bfd-814c-3c6e400f2f15","Type":"ContainerStarted","Data":"5634d8dafc1f6a6cd8a2c3afc54796510e1b21d75140f9370c6a89289f91ebef"}
Jul 17 08:05:14.058161 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:14.058126    2584 generic.go:356] "Generic (PLEG): container finished" podID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerID="0fe50bff56eaf6273afd742f59af8b852d2c12c01e062c4eb91584ecb649c7b6" exitCode=6
Jul 17 08:05:14.058682 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:14.058177    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" event={"ID":"533bc8b5-90c5-4bfd-814c-3c6e400f2f15","Type":"ContainerDied","Data":"0fe50bff56eaf6273afd742f59af8b852d2c12c01e062c4eb91584ecb649c7b6"}
Jul 17 08:05:14.058682 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:14.058486    2584 scope.go:117] "RemoveContainer" containerID="0fe50bff56eaf6273afd742f59af8b852d2c12c01e062c4eb91584ecb649c7b6"
Jul 17 08:05:15.064905 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:15.064869    2584 generic.go:356] "Generic (PLEG): container finished" podID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerID="1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c" exitCode=6
Jul 17 08:05:15.065358 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:15.064926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" event={"ID":"533bc8b5-90c5-4bfd-814c-3c6e400f2f15","Type":"ContainerDied","Data":"1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c"}
Jul 17 08:05:15.065358 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:15.064977    2584 scope.go:117] "RemoveContainer" containerID="0fe50bff56eaf6273afd742f59af8b852d2c12c01e062c4eb91584ecb649c7b6"
Jul 17 08:05:15.065358 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:15.065150    2584 scope.go:117] "RemoveContainer" containerID="1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c"
Jul 17 08:05:15.065542 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:15.065388    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq_odh-ai-gateway-infra(533bc8b5-90c5-4bfd-814c-3c6e400f2f15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15"
Jul 17 08:05:16.070654 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:16.070626    2584 scope.go:117] "RemoveContainer" containerID="1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c"
Jul 17 08:05:16.071037 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:16.070805    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq_odh-ai-gateway-infra(533bc8b5-90c5-4bfd-814c-3c6e400f2f15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15"
Jul 17 08:05:30.284739 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:30.284702    2584 scope.go:117] "RemoveContainer" containerID="1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c"
Jul 17 08:05:31.138218 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.138180    2584 generic.go:356] "Generic (PLEG): container finished" podID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerID="1c12c887529777862a5e65275914b42d97f552d02c36b2fbf547f5f942ce05ee" exitCode=6
Jul 17 08:05:31.138413 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.138256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" event={"ID":"533bc8b5-90c5-4bfd-814c-3c6e400f2f15","Type":"ContainerDied","Data":"1c12c887529777862a5e65275914b42d97f552d02c36b2fbf547f5f942ce05ee"}
Jul 17 08:05:31.138413 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.138361    2584 scope.go:117] "RemoveContainer" containerID="1eaedaa0ca5c4317ad2351b37469cb4a064bf808becd8bfdab7616188c471a0c"
Jul 17 08:05:31.138652 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.138627    2584 scope.go:117] "RemoveContainer" containerID="1c12c887529777862a5e65275914b42d97f552d02c36b2fbf547f5f942ce05ee"
Jul 17 08:05:31.138838 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:31.138816    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq_odh-ai-gateway-infra(533bc8b5-90c5-4bfd-814c-3c6e400f2f15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15"
Jul 17 08:05:31.638360 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.638307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:31.738101 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:31.738068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:32.165995 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.165958    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:32.297419 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.297395    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:32.371132 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.371106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:32.418032 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.417947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca\") pod \"533bc8b5-90c5-4bfd-814c-3c6e400f2f15\" (UID: \"533bc8b5-90c5-4bfd-814c-3c6e400f2f15\") "
Jul 17 08:05:32.418298 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.418274    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "533bc8b5-90c5-4bfd-814c-3c6e400f2f15" (UID: "533bc8b5-90c5-4bfd-814c-3c6e400f2f15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:05:32.519445 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:32.519398    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533bc8b5-90c5-4bfd-814c-3c6e400f2f15-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:33.149967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:33.149932    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"
Jul 17 08:05:33.150456 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:33.149965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq" event={"ID":"533bc8b5-90c5-4bfd-814c-3c6e400f2f15","Type":"ContainerDied","Data":"5634d8dafc1f6a6cd8a2c3afc54796510e1b21d75140f9370c6a89289f91ebef"}
Jul 17 08:05:33.150456 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:33.150026    2584 scope.go:117] "RemoveContainer" containerID="1c12c887529777862a5e65275914b42d97f552d02c36b2fbf547f5f942ce05ee"
Jul 17 08:05:33.171404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:33.171379    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:33.177504 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:33.177483    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-tq8fq"]
Jul 17 08:05:34.290015 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:34.289981    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" path="/var/lib/kubelet/pods/533bc8b5-90c5-4bfd-814c-3c6e400f2f15/volumes"
Jul 17 08:05:36.636792 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.636758    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:05:36.658361 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658282    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:36.658665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658651    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658667    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658682    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658690    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658699    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658716 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658704    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658872 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658752    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658872 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658759    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:36.658872 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658766    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66a075a-9780-428f-9ada-6ba56e558759" containerName="revoke-keys"
Jul 17 08:05:36.658872 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.658773    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9132f0f9-12ba-41a9-9bac-5cb119b75a88" containerName="revoke-keys"
Jul 17 08:05:36.662418 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.662401    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:36.664864 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.664847    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:05:36.670328 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.670294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:36.757140 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.757102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9\" (UID: \"58f38086-82f3-45d7-86ed-01097ee61554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:36.857973 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.857934    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9\" (UID: \"58f38086-82f3-45d7-86ed-01097ee61554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:36.858665 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.858636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9\" (UID: \"58f38086-82f3-45d7-86ed-01097ee61554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:36.973955 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:36.973854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:37.116782 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:37.116754    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:37.119058 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:05:37.119029    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f38086_82f3_45d7_86ed_01097ee61554.slice/crio-55920e092e46ea9ad3f25285bb9b9c61b676ec411fc9930d1c0921595bfe893a WatchSource:0}: Error finding container 55920e092e46ea9ad3f25285bb9b9c61b676ec411fc9930d1c0921595bfe893a: Status 404 returned error can't find the container with id 55920e092e46ea9ad3f25285bb9b9c61b676ec411fc9930d1c0921595bfe893a
Jul 17 08:05:37.169895 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:37.169867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" event={"ID":"58f38086-82f3-45d7-86ed-01097ee61554","Type":"ContainerStarted","Data":"55920e092e46ea9ad3f25285bb9b9c61b676ec411fc9930d1c0921595bfe893a"}
Jul 17 08:05:38.175608 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:38.175576    2584 generic.go:356] "Generic (PLEG): container finished" podID="58f38086-82f3-45d7-86ed-01097ee61554" containerID="8632df045f21dbdb6f5b427d7ecaac5ecb3b12f90c4c539f611c071c4dd32074" exitCode=6
Jul 17 08:05:38.176007 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:38.175670    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" event={"ID":"58f38086-82f3-45d7-86ed-01097ee61554","Type":"ContainerDied","Data":"8632df045f21dbdb6f5b427d7ecaac5ecb3b12f90c4c539f611c071c4dd32074"}
Jul 17 08:05:38.176007 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:38.175822    2584 scope.go:117] "RemoveContainer" containerID="8632df045f21dbdb6f5b427d7ecaac5ecb3b12f90c4c539f611c071c4dd32074"
Jul 17 08:05:38.699311 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:38.699221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:38.803860 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:38.803828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:05:39.181822 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:39.181790    2584 generic.go:356] "Generic (PLEG): container finished" podID="58f38086-82f3-45d7-86ed-01097ee61554" containerID="d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900" exitCode=6
Jul 17 08:05:39.182302 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:39.181849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" event={"ID":"58f38086-82f3-45d7-86ed-01097ee61554","Type":"ContainerDied","Data":"d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900"}
Jul 17 08:05:39.182302 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:39.181897    2584 scope.go:117] "RemoveContainer" containerID="8632df045f21dbdb6f5b427d7ecaac5ecb3b12f90c4c539f611c071c4dd32074"
Jul 17 08:05:39.182302 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:39.182119    2584 scope.go:117] "RemoveContainer" containerID="d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900"
Jul 17 08:05:39.182491 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:39.182366    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9_odh-ai-gateway-infra(58f38086-82f3-45d7-86ed-01097ee61554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" podUID="58f38086-82f3-45d7-86ed-01097ee61554"
Jul 17 08:05:40.187643 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:40.187614    2584 scope.go:117] "RemoveContainer" containerID="d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900"
Jul 17 08:05:40.188020 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:40.187764    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9_odh-ai-gateway-infra(58f38086-82f3-45d7-86ed-01097ee61554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" podUID="58f38086-82f3-45d7-86ed-01097ee61554"
Jul 17 08:05:49.713290 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.713258    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:05:49.713704 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.713634    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="533bc8b5-90c5-4bfd-814c-3c6e400f2f15" containerName="revoke-keys"
Jul 17 08:05:49.717999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.717977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:05:49.724677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.724652    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:05:49.764184 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.764153    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms\" (UID: \"6e955a89-50a7-4e07-9067-982400dfcb74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:05:49.865506 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.865472    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms\" (UID: \"6e955a89-50a7-4e07-9067-982400dfcb74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:05:49.866066 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:49.866047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms\" (UID: \"6e955a89-50a7-4e07-9067-982400dfcb74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:05:50.029769 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:50.029690    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:05:50.161629 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:50.161600    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:05:50.163372 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:05:50.163323    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e955a89_50a7_4e07_9067_982400dfcb74.slice/crio-46c290c97b362e9eb248d5fc0fd6e6c3f3a90655ab240da8cabee9c020607900 WatchSource:0}: Error finding container 46c290c97b362e9eb248d5fc0fd6e6c3f3a90655ab240da8cabee9c020607900: Status 404 returned error can't find the container with id 46c290c97b362e9eb248d5fc0fd6e6c3f3a90655ab240da8cabee9c020607900
Jul 17 08:05:50.232521 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:50.232489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" event={"ID":"6e955a89-50a7-4e07-9067-982400dfcb74","Type":"ContainerStarted","Data":"46c290c97b362e9eb248d5fc0fd6e6c3f3a90655ab240da8cabee9c020607900"}
Jul 17 08:05:51.239094 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:51.239060    2584 generic.go:356] "Generic (PLEG): container finished" podID="6e955a89-50a7-4e07-9067-982400dfcb74" containerID="68adf8dfb154b3c7d9f498425b9050fcf3200d6904d69e889e78eee163370c5b" exitCode=6
Jul 17 08:05:51.239520 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:51.239101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" event={"ID":"6e955a89-50a7-4e07-9067-982400dfcb74","Type":"ContainerDied","Data":"68adf8dfb154b3c7d9f498425b9050fcf3200d6904d69e889e78eee163370c5b"}
Jul 17 08:05:51.239520 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:51.239282    2584 scope.go:117] "RemoveContainer" containerID="68adf8dfb154b3c7d9f498425b9050fcf3200d6904d69e889e78eee163370c5b"
Jul 17 08:05:51.285229 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:51.285210    2584 scope.go:117] "RemoveContainer" containerID="d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900"
Jul 17 08:05:52.245708 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.245626    2584 generic.go:356] "Generic (PLEG): container finished" podID="58f38086-82f3-45d7-86ed-01097ee61554" containerID="7926dc81739114c4c72ae42fe38b172a3c51ff9e2903db2848a950b0aee28470" exitCode=6
Jul 17 08:05:52.246112 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.245707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" event={"ID":"58f38086-82f3-45d7-86ed-01097ee61554","Type":"ContainerDied","Data":"7926dc81739114c4c72ae42fe38b172a3c51ff9e2903db2848a950b0aee28470"}
Jul 17 08:05:52.246112 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.245765    2584 scope.go:117] "RemoveContainer" containerID="d7864145bdb389ef3a7b3963def7a480cba1993c63f836d9c5d4a52008939900"
Jul 17 08:05:52.246112 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.246047    2584 scope.go:117] "RemoveContainer" containerID="7926dc81739114c4c72ae42fe38b172a3c51ff9e2903db2848a950b0aee28470"
Jul 17 08:05:52.246282 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:52.246249    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9_odh-ai-gateway-infra(58f38086-82f3-45d7-86ed-01097ee61554)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" podUID="58f38086-82f3-45d7-86ed-01097ee61554"
Jul 17 08:05:52.247459 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.247437    2584 generic.go:356] "Generic (PLEG): container finished" podID="6e955a89-50a7-4e07-9067-982400dfcb74" containerID="a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751" exitCode=6
Jul 17 08:05:52.247564 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.247479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" event={"ID":"6e955a89-50a7-4e07-9067-982400dfcb74","Type":"ContainerDied","Data":"a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751"}
Jul 17 08:05:52.247672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.247660    2584 scope.go:117] "RemoveContainer" containerID="a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751"
Jul 17 08:05:52.247824 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:52.247809    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms_odh-ai-gateway-infra(6e955a89-50a7-4e07-9067-982400dfcb74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" podUID="6e955a89-50a7-4e07-9067-982400dfcb74"
Jul 17 08:05:52.256217 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.256201    2584 scope.go:117] "RemoveContainer" containerID="68adf8dfb154b3c7d9f498425b9050fcf3200d6904d69e889e78eee163370c5b"
Jul 17 08:05:52.734372 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:52.734317    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:05:53.255561 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.255536    2584 scope.go:117] "RemoveContainer" containerID="a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751"
Jul 17 08:05:53.255934 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:53.255720    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms_odh-ai-gateway-infra(6e955a89-50a7-4e07-9067-982400dfcb74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" podUID="6e955a89-50a7-4e07-9067-982400dfcb74"
Jul 17 08:05:53.276202 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.276166    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:53.414062 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.414042    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:53.496848 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.496815    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca\") pod \"58f38086-82f3-45d7-86ed-01097ee61554\" (UID: \"58f38086-82f3-45d7-86ed-01097ee61554\") "
Jul 17 08:05:53.497168 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.497145    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58f38086-82f3-45d7-86ed-01097ee61554" (UID: "58f38086-82f3-45d7-86ed-01097ee61554"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:05:53.598272 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:53.598240    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f38086-82f3-45d7-86ed-01097ee61554-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:05:54.260599 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.260563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9" event={"ID":"58f38086-82f3-45d7-86ed-01097ee61554","Type":"ContainerDied","Data":"55920e092e46ea9ad3f25285bb9b9c61b676ec411fc9930d1c0921595bfe893a"}
Jul 17 08:05:54.260599 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.260598    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"
Jul 17 08:05:54.261075 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.260651    2584 scope.go:117] "RemoveContainer" containerID="7926dc81739114c4c72ae42fe38b172a3c51ff9e2903db2848a950b0aee28470"
Jul 17 08:05:54.284197 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.284175    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:54.292285 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.292261    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-kk5c9"]
Jul 17 08:05:54.353285 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353261    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:05:54.353613 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353600    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353615    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353627    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353635    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353774 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353703    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353774 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353714    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.353774 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.353722    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:05:54.357901 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.357886    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:05:54.365701 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.365677    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:05:54.405059 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.405031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg\" (UID: \"8d967440-9708-494d-9e8e-c6cc8304cfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:05:54.506003 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.505957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg\" (UID: \"8d967440-9708-494d-9e8e-c6cc8304cfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:05:54.506595 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.506572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg\" (UID: \"8d967440-9708-494d-9e8e-c6cc8304cfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:05:54.668618 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.668591    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:05:54.798485 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:54.798455    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:05:54.800691 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:05:54.800659    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d967440_9708_494d_9e8e_c6cc8304cfc9.slice/crio-1b5e5c57793196460213c93db6f0992d824ce01e8feab6ab6d165be67a3abffa WatchSource:0}: Error finding container 1b5e5c57793196460213c93db6f0992d824ce01e8feab6ab6d165be67a3abffa: Status 404 returned error can't find the container with id 1b5e5c57793196460213c93db6f0992d824ce01e8feab6ab6d165be67a3abffa
Jul 17 08:05:55.268626 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:55.268590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerStarted","Data":"6545e27f866e39377a1974a032507f17bf733d3e4bbfde40991ef3f456cdc68b"}
Jul 17 08:05:55.269022 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:55.268647    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerStarted","Data":"1b5e5c57793196460213c93db6f0992d824ce01e8feab6ab6d165be67a3abffa"}
Jul 17 08:05:55.289164 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:55.289108    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" podStartSLOduration=1.289089887 podStartE2EDuration="1.289089887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:05:55.288379392 +0000 UTC m=+3251.541802206" watchObservedRunningTime="2026-07-17 08:05:55.289089887 +0000 UTC m=+3251.542512698"
Jul 17 08:05:56.274449 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:56.274413    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerID="6545e27f866e39377a1974a032507f17bf733d3e4bbfde40991ef3f456cdc68b" exitCode=6
Jul 17 08:05:56.274449 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:56.274446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerDied","Data":"6545e27f866e39377a1974a032507f17bf733d3e4bbfde40991ef3f456cdc68b"}
Jul 17 08:05:56.274881 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:56.274689    2584 scope.go:117] "RemoveContainer" containerID="6545e27f866e39377a1974a032507f17bf733d3e4bbfde40991ef3f456cdc68b"
Jul 17 08:05:56.291054 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:56.291034    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f38086-82f3-45d7-86ed-01097ee61554" path="/var/lib/kubelet/pods/58f38086-82f3-45d7-86ed-01097ee61554/volumes"
Jul 17 08:05:57.280744 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:57.280665    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerID="ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff" exitCode=6
Jul 17 08:05:57.280744 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:57.280707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerDied","Data":"ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff"}
Jul 17 08:05:57.281199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:57.280755    2584 scope.go:117] "RemoveContainer" containerID="6545e27f866e39377a1974a032507f17bf733d3e4bbfde40991ef3f456cdc68b"
Jul 17 08:05:57.281199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:57.280918    2584 scope.go:117] "RemoveContainer" containerID="ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff"
Jul 17 08:05:57.281199 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:57.281128    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg_odh-ai-gateway-infra(8d967440-9708-494d-9e8e-c6cc8304cfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9"
Jul 17 08:05:58.289830 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:05:58.289802    2584 scope.go:117] "RemoveContainer" containerID="ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff"
Jul 17 08:05:58.290205 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:05:58.290039    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg_odh-ai-gateway-infra(8d967440-9708-494d-9e8e-c6cc8304cfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9"
Jul 17 08:06:06.285388 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:06.285320    2584 scope.go:117] "RemoveContainer" containerID="a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751"
Jul 17 08:06:07.331299 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:07.331258    2584 generic.go:356] "Generic (PLEG): container finished" podID="6e955a89-50a7-4e07-9067-982400dfcb74" containerID="97f3add6db3130a107ea57996ca3583bfccb357e3a671f9538a1f842bebb6a4c" exitCode=6
Jul 17 08:06:07.331709 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:07.331320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" event={"ID":"6e955a89-50a7-4e07-9067-982400dfcb74","Type":"ContainerDied","Data":"97f3add6db3130a107ea57996ca3583bfccb357e3a671f9538a1f842bebb6a4c"}
Jul 17 08:06:07.331709 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:07.331397    2584 scope.go:117] "RemoveContainer" containerID="a810f5dd590cf1af81827bbc1841ee8c49100d9e0324b68588825173170e3751"
Jul 17 08:06:07.331709 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:07.331677    2584 scope.go:117] "RemoveContainer" containerID="97f3add6db3130a107ea57996ca3583bfccb357e3a671f9538a1f842bebb6a4c"
Jul 17 08:06:07.331891 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:07.331873    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms_odh-ai-gateway-infra(6e955a89-50a7-4e07-9067-982400dfcb74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" podUID="6e955a89-50a7-4e07-9067-982400dfcb74"
Jul 17 08:06:08.367203 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:08.367169    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:06:08.500683 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:08.500657    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:06:08.631085 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:08.631004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca\") pod \"6e955a89-50a7-4e07-9067-982400dfcb74\" (UID: \"6e955a89-50a7-4e07-9067-982400dfcb74\") "
Jul 17 08:06:08.631386 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:08.631334    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e955a89-50a7-4e07-9067-982400dfcb74" (UID: "6e955a89-50a7-4e07-9067-982400dfcb74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:06:08.732318 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:08.732279    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e955a89-50a7-4e07-9067-982400dfcb74-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:09.343280 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:09.343251    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"
Jul 17 08:06:09.343280 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:09.343277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms" event={"ID":"6e955a89-50a7-4e07-9067-982400dfcb74","Type":"ContainerDied","Data":"46c290c97b362e9eb248d5fc0fd6e6c3f3a90655ab240da8cabee9c020607900"}
Jul 17 08:06:09.343538 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:09.343332    2584 scope.go:117] "RemoveContainer" containerID="97f3add6db3130a107ea57996ca3583bfccb357e3a671f9538a1f842bebb6a4c"
Jul 17 08:06:09.367556 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:09.367533    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:06:09.371292 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:09.371273    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-8b7ms"]
Jul 17 08:06:10.290293 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:10.290260    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" path="/var/lib/kubelet/pods/6e955a89-50a7-4e07-9067-982400dfcb74/volumes"
Jul 17 08:06:12.267515 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.267475    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:06:12.284835 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.284805    2584 scope.go:117] "RemoveContainer" containerID="ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff"
Jul 17 08:06:12.605158 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605124    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:12.605480 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605467    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:06:12.605525 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605482    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f38086-82f3-45d7-86ed-01097ee61554" containerName="revoke-keys"
Jul 17 08:06:12.605525 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605493    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605525 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605499    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605525 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605514    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605525 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605519    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605678 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605529    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605678 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605535    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605678 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605583    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605678 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605590    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.605678 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.605598    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e955a89-50a7-4e07-9067-982400dfcb74" containerName="revoke-keys"
Jul 17 08:06:12.609605 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.609585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:12.618436 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.618400    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:12.773927 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.773887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx\" (UID: \"edd910ec-f7ac-411e-8bd0-30effa413e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:12.874359 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.874261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx\" (UID: \"edd910ec-f7ac-411e-8bd0-30effa413e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:12.874948 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.874929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx\" (UID: \"edd910ec-f7ac-411e-8bd0-30effa413e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:12.920392 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:12.920359    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:13.058169 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.058114    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:13.060219 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:06:13.060192    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd910ec_f7ac_411e_8bd0_30effa413e40.slice/crio-38b5d55bd1a0b67ab97e72ff7a994803402f96ca92f503e77b54f75da17f6564 WatchSource:0}: Error finding container 38b5d55bd1a0b67ab97e72ff7a994803402f96ca92f503e77b54f75da17f6564: Status 404 returned error can't find the container with id 38b5d55bd1a0b67ab97e72ff7a994803402f96ca92f503e77b54f75da17f6564
Jul 17 08:06:13.365830 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.365793    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerID="261aff15da2c547234b59e32d7dad5c76529b599495f687b30111d731d85bc2d" exitCode=6
Jul 17 08:06:13.366230 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.365861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerDied","Data":"261aff15da2c547234b59e32d7dad5c76529b599495f687b30111d731d85bc2d"}
Jul 17 08:06:13.366230 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.365921    2584 scope.go:117] "RemoveContainer" containerID="ce03082bcbfaf24eb2297ed8fa13ea8176d356dd265a3e201521102a1ca226ff"
Jul 17 08:06:13.366230 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.366220    2584 scope.go:117] "RemoveContainer" containerID="261aff15da2c547234b59e32d7dad5c76529b599495f687b30111d731d85bc2d"
Jul 17 08:06:13.366499 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:13.366476    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg_odh-ai-gateway-infra(8d967440-9708-494d-9e8e-c6cc8304cfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9"
Jul 17 08:06:13.367601 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.367579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerStarted","Data":"f5f5f36529e81ea97e65db9b85b1dc9bd825a3b58efc70b265610aef45cb2b9b"}
Jul 17 08:06:13.367689 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.367628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerStarted","Data":"38b5d55bd1a0b67ab97e72ff7a994803402f96ca92f503e77b54f75da17f6564"}
Jul 17 08:06:13.405091 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:13.405039    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" podStartSLOduration=1.405023981 podStartE2EDuration="1.405023981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:06:13.403022125 +0000 UTC m=+3269.656444937" watchObservedRunningTime="2026-07-17 08:06:13.405023981 +0000 UTC m=+3269.658446792"
Jul 17 08:06:14.374622 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.374585    2584 generic.go:356] "Generic (PLEG): container finished" podID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerID="f5f5f36529e81ea97e65db9b85b1dc9bd825a3b58efc70b265610aef45cb2b9b" exitCode=6
Jul 17 08:06:14.375010 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.374659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerDied","Data":"f5f5f36529e81ea97e65db9b85b1dc9bd825a3b58efc70b265610aef45cb2b9b"}
Jul 17 08:06:14.375010 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.374856    2584 scope.go:117] "RemoveContainer" containerID="f5f5f36529e81ea97e65db9b85b1dc9bd825a3b58efc70b265610aef45cb2b9b"
Jul 17 08:06:14.397873 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.397850    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:06:14.546599 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.546577    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:06:14.689650 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.689610    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca\") pod \"8d967440-9708-494d-9e8e-c6cc8304cfc9\" (UID: \"8d967440-9708-494d-9e8e-c6cc8304cfc9\") "
Jul 17 08:06:14.689962 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.689939    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d967440-9708-494d-9e8e-c6cc8304cfc9" (UID: "8d967440-9708-494d-9e8e-c6cc8304cfc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:06:14.791122 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:14.791083    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d967440-9708-494d-9e8e-c6cc8304cfc9-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:15.380500 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.380468    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"
Jul 17 08:06:15.380962 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.380467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg" event={"ID":"8d967440-9708-494d-9e8e-c6cc8304cfc9","Type":"ContainerDied","Data":"1b5e5c57793196460213c93db6f0992d824ce01e8feab6ab6d165be67a3abffa"}
Jul 17 08:06:15.380962 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.380621    2584 scope.go:117] "RemoveContainer" containerID="261aff15da2c547234b59e32d7dad5c76529b599495f687b30111d731d85bc2d"
Jul 17 08:06:15.383210 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.383173    2584 generic.go:356] "Generic (PLEG): container finished" podID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerID="7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced" exitCode=6
Jul 17 08:06:15.383368 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.383216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerDied","Data":"7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced"}
Jul 17 08:06:15.383697 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.383678    2584 scope.go:117] "RemoveContainer" containerID="7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced"
Jul 17 08:06:15.384058 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:15.384034    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx_odh-ai-gateway-infra(edd910ec-f7ac-411e-8bd0-30effa413e40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40"
Jul 17 08:06:15.397308 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.397292    2584 scope.go:117] "RemoveContainer" containerID="f5f5f36529e81ea97e65db9b85b1dc9bd825a3b58efc70b265610aef45cb2b9b"
Jul 17 08:06:15.421086 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.421058    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:06:15.430225 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:15.430198    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-st6sg"]
Jul 17 08:06:16.290577 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:16.290545    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" path="/var/lib/kubelet/pods/8d967440-9708-494d-9e8e-c6cc8304cfc9/volumes"
Jul 17 08:06:16.382060 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:16.382020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:16.390300 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:16.390279    2584 scope.go:117] "RemoveContainer" containerID="7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced"
Jul 17 08:06:16.390497 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:16.390478    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx_odh-ai-gateway-infra(edd910ec-f7ac-411e-8bd0-30effa413e40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40"
Jul 17 08:06:17.284834 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:17.284802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:26.698597 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698562    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698879    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698890    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698901    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698907    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698942    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.698968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.698948    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.699182 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.699000    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.699182 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.699007    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:26.701920 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.701904    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:26.713071 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.713047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:26.776909 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.776879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv\" (UID: \"d74b99a6-d518-4e94-8351-113f1c3c806a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:26.877572 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.877541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv\" (UID: \"d74b99a6-d518-4e94-8351-113f1c3c806a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:26.878199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:26.878175    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv\" (UID: \"d74b99a6-d518-4e94-8351-113f1c3c806a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:27.011858 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:27.011785    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:27.142043 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:27.142016    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:27.144136 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:06:27.144098    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74b99a6_d518_4e94_8351_113f1c3c806a.slice/crio-841dbb73cada181d1cd58179613631ebe1fdcb6a8ab65392274aa523a83dcf82 WatchSource:0}: Error finding container 841dbb73cada181d1cd58179613631ebe1fdcb6a8ab65392274aa523a83dcf82: Status 404 returned error can't find the container with id 841dbb73cada181d1cd58179613631ebe1fdcb6a8ab65392274aa523a83dcf82
Jul 17 08:06:27.449165 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:27.449129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerStarted","Data":"eba837867522205674ac0b4bd826f3d633372806b2e0d85d773354937353a74a"}
Jul 17 08:06:27.449445 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:27.449424    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerStarted","Data":"841dbb73cada181d1cd58179613631ebe1fdcb6a8ab65392274aa523a83dcf82"}
Jul 17 08:06:27.468946 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:27.468900    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" podStartSLOduration=1.468886174 podStartE2EDuration="1.468886174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:06:27.467061288 +0000 UTC m=+3283.720484099" watchObservedRunningTime="2026-07-17 08:06:27.468886174 +0000 UTC m=+3283.722309021"
Jul 17 08:06:28.455563 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:28.455532    2584 generic.go:356] "Generic (PLEG): container finished" podID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerID="eba837867522205674ac0b4bd826f3d633372806b2e0d85d773354937353a74a" exitCode=6
Jul 17 08:06:28.455936 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:28.455626    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerDied","Data":"eba837867522205674ac0b4bd826f3d633372806b2e0d85d773354937353a74a"}
Jul 17 08:06:28.455936 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:28.455807    2584 scope.go:117] "RemoveContainer" containerID="eba837867522205674ac0b4bd826f3d633372806b2e0d85d773354937353a74a"
Jul 17 08:06:29.047108 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:29.047082    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:06:29.461552 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:29.461517    2584 generic.go:356] "Generic (PLEG): container finished" podID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerID="ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f" exitCode=6
Jul 17 08:06:29.461977 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:29.461584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerDied","Data":"ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f"}
Jul 17 08:06:29.461977 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:29.461627    2584 scope.go:117] "RemoveContainer" containerID="eba837867522205674ac0b4bd826f3d633372806b2e0d85d773354937353a74a"
Jul 17 08:06:29.461977 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:29.461883    2584 scope.go:117] "RemoveContainer" containerID="ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f"
Jul 17 08:06:29.462116 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:29.462093    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv_odh-ai-gateway-infra(d74b99a6-d518-4e94-8351-113f1c3c806a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a"
Jul 17 08:06:30.286293 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:30.285939    2584 scope.go:117] "RemoveContainer" containerID="7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced"
Jul 17 08:06:30.468402 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:30.468372    2584 scope.go:117] "RemoveContainer" containerID="ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f"
Jul 17 08:06:30.468790 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:30.468571    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv_odh-ai-gateway-infra(d74b99a6-d518-4e94-8351-113f1c3c806a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a"
Jul 17 08:06:31.474383 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:31.474330    2584 generic.go:356] "Generic (PLEG): container finished" podID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerID="66ab4a87b3ed2ea20afc261cb2b41c81db854bd49a22a31aa12de1e48e673c1e" exitCode=6
Jul 17 08:06:31.474383 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:31.474377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerDied","Data":"66ab4a87b3ed2ea20afc261cb2b41c81db854bd49a22a31aa12de1e48e673c1e"}
Jul 17 08:06:31.474897 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:31.474440    2584 scope.go:117] "RemoveContainer" containerID="7aef521b2be9a8d72f308efb7af7cc89284ad1d6496cbddfe08d1a0e61eb0ced"
Jul 17 08:06:31.474897 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:31.474700    2584 scope.go:117] "RemoveContainer" containerID="66ab4a87b3ed2ea20afc261cb2b41c81db854bd49a22a31aa12de1e48e673c1e"
Jul 17 08:06:31.475013 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:31.474901    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx_odh-ai-gateway-infra(edd910ec-f7ac-411e-8bd0-30effa413e40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40"
Jul 17 08:06:32.506233 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:32.506199    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:32.637199 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:32.637172    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:32.727833 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:32.727805    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca\") pod \"edd910ec-f7ac-411e-8bd0-30effa413e40\" (UID: \"edd910ec-f7ac-411e-8bd0-30effa413e40\") "
Jul 17 08:06:32.728182 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:32.728161    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edd910ec-f7ac-411e-8bd0-30effa413e40" (UID: "edd910ec-f7ac-411e-8bd0-30effa413e40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:06:32.828309 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:32.828282    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd910ec-f7ac-411e-8bd0-30effa413e40-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:33.486540 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:33.486508    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"
Jul 17 08:06:33.486540 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:33.486527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx" event={"ID":"edd910ec-f7ac-411e-8bd0-30effa413e40","Type":"ContainerDied","Data":"38b5d55bd1a0b67ab97e72ff7a994803402f96ca92f503e77b54f75da17f6564"}
Jul 17 08:06:33.486755 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:33.486592    2584 scope.go:117] "RemoveContainer" containerID="66ab4a87b3ed2ea20afc261cb2b41c81db854bd49a22a31aa12de1e48e673c1e"
Jul 17 08:06:33.510954 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:33.510930    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:33.516860 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:33.516835    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-8gwnx"]
Jul 17 08:06:34.289927 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:34.289893    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" path="/var/lib/kubelet/pods/edd910ec-f7ac-411e-8bd0-30effa413e40/volumes"
Jul 17 08:06:37.988352 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:37.988300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:38.089841 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:38.089786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:43.285111 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:43.285074    2584 scope.go:117] "RemoveContainer" containerID="ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f"
Jul 17 08:06:43.532885 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:43.532852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerStarted","Data":"198e625d45c79585eb9523fbf10d8b41a59de97ee4e2c53f352e59e14c5fb92a"}
Jul 17 08:06:44.458264 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.458237    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 08:06:44.461692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.461663    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2tp8r_6aca15ef-883e-4aa7-9711-050622a04421/ovn-acl-logging/0.log"
Jul 17 08:06:44.539089 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.539055    2584 generic.go:356] "Generic (PLEG): container finished" podID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerID="198e625d45c79585eb9523fbf10d8b41a59de97ee4e2c53f352e59e14c5fb92a" exitCode=6
Jul 17 08:06:44.539238 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.539125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerDied","Data":"198e625d45c79585eb9523fbf10d8b41a59de97ee4e2c53f352e59e14c5fb92a"}
Jul 17 08:06:44.539238 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.539207    2584 scope.go:117] "RemoveContainer" containerID="ac6df067404408acdd8cd04859c7185afaa538092cf848ec15c2e04a4fd9e21f"
Jul 17 08:06:44.539545 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.539523    2584 scope.go:117] "RemoveContainer" containerID="198e625d45c79585eb9523fbf10d8b41a59de97ee4e2c53f352e59e14c5fb92a"
Jul 17 08:06:44.539762 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:44.539744    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv_odh-ai-gateway-infra(d74b99a6-d518-4e94-8351-113f1c3c806a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a"
Jul 17 08:06:44.568181 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:44.568149    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:45.682473 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:45.682450    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:45.737208 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:45.737173    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca\") pod \"d74b99a6-d518-4e94-8351-113f1c3c806a\" (UID: \"d74b99a6-d518-4e94-8351-113f1c3c806a\") "
Jul 17 08:06:45.737540 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:45.737517    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d74b99a6-d518-4e94-8351-113f1c3c806a" (UID: "d74b99a6-d518-4e94-8351-113f1c3c806a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:06:45.837812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:45.837719    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b99a6-d518-4e94-8351-113f1c3c806a-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:46.551700 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:46.551619    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"
Jul 17 08:06:46.551850 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:46.551620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv" event={"ID":"d74b99a6-d518-4e94-8351-113f1c3c806a","Type":"ContainerDied","Data":"841dbb73cada181d1cd58179613631ebe1fdcb6a8ab65392274aa523a83dcf82"}
Jul 17 08:06:46.551850 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:46.551756    2584 scope.go:117] "RemoveContainer" containerID="198e625d45c79585eb9523fbf10d8b41a59de97ee4e2c53f352e59e14c5fb92a"
Jul 17 08:06:46.570724 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:46.570695    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:46.574506 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:46.574477    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-hgchv"]
Jul 17 08:06:47.140959 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.140931    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:06:47.141445 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.141204    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" podUID="48d8457f-404f-4590-a3fa-5165799aa088" containerName="maas-api" containerID="cri-o://82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007" gracePeriod=30
Jul 17 08:06:47.183692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.183663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:47.393452 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.393389    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:06:47.454413 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.454373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s6m2\" (UniqueName: \"kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2\") pod \"48d8457f-404f-4590-a3fa-5165799aa088\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") "
Jul 17 08:06:47.454601 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.454460    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca\") pod \"48d8457f-404f-4590-a3fa-5165799aa088\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") "
Jul 17 08:06:47.454601 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.454519    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") pod \"48d8457f-404f-4590-a3fa-5165799aa088\" (UID: \"48d8457f-404f-4590-a3fa-5165799aa088\") "
Jul 17 08:06:47.454832 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.454804    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48d8457f-404f-4590-a3fa-5165799aa088" (UID: "48d8457f-404f-4590-a3fa-5165799aa088"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:06:47.456533 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.456508    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48d8457f-404f-4590-a3fa-5165799aa088" (UID: "48d8457f-404f-4590-a3fa-5165799aa088"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 08:06:47.456621 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.456544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2" (OuterVolumeSpecName: "kube-api-access-2s6m2") pod "48d8457f-404f-4590-a3fa-5165799aa088" (UID: "48d8457f-404f-4590-a3fa-5165799aa088"). InnerVolumeSpecName "kube-api-access-2s6m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 08:06:47.555510 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.555472    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d8457f-404f-4590-a3fa-5165799aa088-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:47.555510 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.555508    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6m2\" (UniqueName: \"kubernetes.io/projected/48d8457f-404f-4590-a3fa-5165799aa088-kube-api-access-2s6m2\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:47.555727 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.555524    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d8457f-404f-4590-a3fa-5165799aa088-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:06:47.557452 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.557421    2584 generic.go:356] "Generic (PLEG): container finished" podID="48d8457f-404f-4590-a3fa-5165799aa088" containerID="82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007" exitCode=0
Jul 17 08:06:47.557576 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.557470    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"
Jul 17 08:06:47.557576 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.557513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" event={"ID":"48d8457f-404f-4590-a3fa-5165799aa088","Type":"ContainerDied","Data":"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"}
Jul 17 08:06:47.557576 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.557567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr" event={"ID":"48d8457f-404f-4590-a3fa-5165799aa088","Type":"ContainerDied","Data":"5d3177dfe86388ea6b8e91799869ba6f3a2e859e8e8498254e616ca458b72d98"}
Jul 17 08:06:47.557717 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.557588    2584 scope.go:117] "RemoveContainer" containerID="82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"
Jul 17 08:06:47.566381 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.566358    2584 scope.go:117] "RemoveContainer" containerID="82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"
Jul 17 08:06:47.566637 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:47.566618    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007\": container with ID starting with 82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007 not found: ID does not exist" containerID="82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"
Jul 17 08:06:47.566685 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.566646    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007"} err="failed to get container status \"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007\": rpc error: code = NotFound desc = could not find container \"82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007\": container with ID starting with 82857dbf449cbc5a338c7258d559f3af5e37b216200161dbcf8ad61335af1007 not found: ID does not exist"
Jul 17 08:06:47.581235 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.581201    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:06:47.585101 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:47.585077    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d805bd-76c5ddb649-6sxsr"]
Jul 17 08:06:48.289621 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:48.289578    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d8457f-404f-4590-a3fa-5165799aa088" path="/var/lib/kubelet/pods/48d8457f-404f-4590-a3fa-5165799aa088/volumes"
Jul 17 08:06:48.290116 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:48.290042    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" path="/var/lib/kubelet/pods/d74b99a6-d518-4e94-8351-113f1c3c806a/volumes"
Jul 17 08:06:51.030007 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:51.029966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:06:51.659640 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:51.659596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:54.401154 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401113    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401469    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401482    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401498    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401504    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401513    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401519    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401525    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d8457f-404f-4590-a3fa-5165799aa088" containerName="maas-api"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401530    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d8457f-404f-4590-a3fa-5165799aa088" containerName="maas-api"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401540    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.401581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401545    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401600    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401611    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d8457f-404f-4590-a3fa-5165799aa088" containerName="maas-api"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401617    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401624    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d967440-9708-494d-9e8e-c6cc8304cfc9" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401630    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401635    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:06:54.401892 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.401640    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:06:54.405894 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.405874    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:06:54.408677 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.408655    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:06:54.412483 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.412458    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:06:54.519171 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.519121    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85\" (UID: \"c97e8ae6-cbb2-465f-9f1b-716e4ed97408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:06:54.620035 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.619993    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85\" (UID: \"c97e8ae6-cbb2-465f-9f1b-716e4ed97408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:06:54.620675 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.620652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85\" (UID: \"c97e8ae6-cbb2-465f-9f1b-716e4ed97408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:06:54.718305 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:54.718226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:06:55.055163 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:55.055137    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:06:55.057711 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:06:55.057685    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97e8ae6_cbb2_465f_9f1b_716e4ed97408.slice/crio-11be0e3a1f16bbf1a79dee3f8955135c5e6a10c24867ffc2128c7893aca9ee0a WatchSource:0}: Error finding container 11be0e3a1f16bbf1a79dee3f8955135c5e6a10c24867ffc2128c7893aca9ee0a: Status 404 returned error can't find the container with id 11be0e3a1f16bbf1a79dee3f8955135c5e6a10c24867ffc2128c7893aca9ee0a
Jul 17 08:06:55.597148 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:55.597049    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerStarted","Data":"4ec28e3f1b46e6ede72df35c3494beefa52138031032b8b1ba3ce93c1512723a"}
Jul 17 08:06:55.597148 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:55.597138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerStarted","Data":"11be0e3a1f16bbf1a79dee3f8955135c5e6a10c24867ffc2128c7893aca9ee0a"}
Jul 17 08:06:55.613592 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:55.613540    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" podStartSLOduration=1.613528182 podStartE2EDuration="1.613528182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:06:55.612853452 +0000 UTC m=+3311.866276264" watchObservedRunningTime="2026-07-17 08:06:55.613528182 +0000 UTC m=+3311.866950997"
Jul 17 08:06:56.602659 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:56.602621    2584 generic.go:356] "Generic (PLEG): container finished" podID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerID="4ec28e3f1b46e6ede72df35c3494beefa52138031032b8b1ba3ce93c1512723a" exitCode=6
Jul 17 08:06:56.603021 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:56.602675    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerDied","Data":"4ec28e3f1b46e6ede72df35c3494beefa52138031032b8b1ba3ce93c1512723a"}
Jul 17 08:06:56.603021 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:56.602899    2584 scope.go:117] "RemoveContainer" containerID="4ec28e3f1b46e6ede72df35c3494beefa52138031032b8b1ba3ce93c1512723a"
Jul 17 08:06:57.609920 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:57.609884    2584 generic.go:356] "Generic (PLEG): container finished" podID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerID="054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64" exitCode=6
Jul 17 08:06:57.610404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:57.609964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerDied","Data":"054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64"}
Jul 17 08:06:57.610404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:57.610042    2584 scope.go:117] "RemoveContainer" containerID="4ec28e3f1b46e6ede72df35c3494beefa52138031032b8b1ba3ce93c1512723a"
Jul 17 08:06:57.610404 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:57.610293    2584 scope.go:117] "RemoveContainer" containerID="054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64"
Jul 17 08:06:57.610555 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:57.610499    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85_odh-ai-gateway-infra(c97e8ae6-cbb2-465f-9f1b-716e4ed97408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408"
Jul 17 08:06:57.955260 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:57.955188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:58.057227 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:58.057196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:06:58.616879 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:06:58.616849    2584 scope.go:117] "RemoveContainer" containerID="054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64"
Jul 17 08:06:58.617263 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:06:58.617112    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85_odh-ai-gateway-infra(c97e8ae6-cbb2-465f-9f1b-716e4ed97408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408"
Jul 17 08:07:02.458622 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.458588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:02.676680 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.676644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:02.804218 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804187    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"]
Jul 17 08:07:02.804537 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804524    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:07:02.804588 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804539    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd910ec-f7ac-411e-8bd0-30effa413e40" containerName="revoke-keys"
Jul 17 08:07:02.804588 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804555    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:07:02.804588 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804561    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:07:02.804679 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.804624    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74b99a6-d518-4e94-8351-113f1c3c806a" containerName="revoke-keys"
Jul 17 08:07:02.808715 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.808699    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:02.816422 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.816394    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"]
Jul 17 08:07:02.895094 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.895053    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7\" (UID: \"0eb948e0-e651-4b60-a09e-6f27efbd5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:02.996118 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.996079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7\" (UID: \"0eb948e0-e651-4b60-a09e-6f27efbd5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:02.996763 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:02.996742    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7\" (UID: \"0eb948e0-e651-4b60-a09e-6f27efbd5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:03.120817 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.120729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:03.259035 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.258995    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"]
Jul 17 08:07:03.261219 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:07:03.261189    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb948e0_e651_4b60_a09e_6f27efbd5ff9.slice/crio-c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2 WatchSource:0}: Error finding container c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2: Status 404 returned error can't find the container with id c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2
Jul 17 08:07:03.640589 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.640555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7" event={"ID":"0eb948e0-e651-4b60-a09e-6f27efbd5ff9","Type":"ContainerStarted","Data":"14ccb28c32785bd9fa19b69eeedd01932efa67c77c8ec963f67d5b6cbc3a70db"}
Jul 17 08:07:03.640954 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.640609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7" event={"ID":"0eb948e0-e651-4b60-a09e-6f27efbd5ff9","Type":"ContainerStarted","Data":"c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2"}
Jul 17 08:07:03.664218 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.664169    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7" podStartSLOduration=1.664151578 podStartE2EDuration="1.664151578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:07:03.661809204 +0000 UTC m=+3319.915232018" watchObservedRunningTime="2026-07-17 08:07:03.664151578 +0000 UTC m=+3319.917574390"
Jul 17 08:07:03.722625 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.722542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:03.823566 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:03.823531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:04.646273 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:04.646235    2584 generic.go:356] "Generic (PLEG): container finished" podID="0eb948e0-e651-4b60-a09e-6f27efbd5ff9" containerID="14ccb28c32785bd9fa19b69eeedd01932efa67c77c8ec963f67d5b6cbc3a70db" exitCode=0
Jul 17 08:07:04.646664 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:04.646319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7" event={"ID":"0eb948e0-e651-4b60-a09e-6f27efbd5ff9","Type":"ContainerDied","Data":"14ccb28c32785bd9fa19b69eeedd01932efa67c77c8ec963f67d5b6cbc3a70db"}
Jul 17 08:07:05.780725 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:05.780701    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:05.921729 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:05.921643    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca\") pod \"0eb948e0-e651-4b60-a09e-6f27efbd5ff9\" (UID: \"0eb948e0-e651-4b60-a09e-6f27efbd5ff9\") "
Jul 17 08:07:05.922040 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:05.922019    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0eb948e0-e651-4b60-a09e-6f27efbd5ff9" (UID: "0eb948e0-e651-4b60-a09e-6f27efbd5ff9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:07:06.022318 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:06.022292    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb948e0-e651-4b60-a09e-6f27efbd5ff9-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:06.656487 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:06.656370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7" event={"ID":"0eb948e0-e651-4b60-a09e-6f27efbd5ff9","Type":"ContainerDied","Data":"c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2"}
Jul 17 08:07:06.656487 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:06.656453    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c859e7a016192dec847a4c26de1154871be7bcf4aca44ef8cfd46fc996a26fd2"
Jul 17 08:07:06.656487 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:06.656385    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7"
Jul 17 08:07:12.661469 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.661434    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:12.661878 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.661748    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb948e0-e651-4b60-a09e-6f27efbd5ff9" containerName="revoke-keys"
Jul 17 08:07:12.661878 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.661759    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb948e0-e651-4b60-a09e-6f27efbd5ff9" containerName="revoke-keys"
Jul 17 08:07:12.661878 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.661824    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb948e0-e651-4b60-a09e-6f27efbd5ff9" containerName="revoke-keys"
Jul 17 08:07:12.664612 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.664595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:12.673620 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.673594    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:12.677057 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.677030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq\" (UID: \"64f46d6c-3849-4c2a-a481-76dd959f4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:12.777958 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.777915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq\" (UID: \"64f46d6c-3849-4c2a-a481-76dd959f4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:12.778652 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.778627    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq\" (UID: \"64f46d6c-3849-4c2a-a481-76dd959f4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:12.975400 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:12.975290    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:13.103581 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.103543    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:13.106367 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:07:13.106320    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f46d6c_3849_4c2a_a481_76dd959f4268.slice/crio-2bfac23a224c3d6e8578fa96d9b41f1957e27dedb0048e092553691b6749ec83 WatchSource:0}: Error finding container 2bfac23a224c3d6e8578fa96d9b41f1957e27dedb0048e092553691b6749ec83: Status 404 returned error can't find the container with id 2bfac23a224c3d6e8578fa96d9b41f1957e27dedb0048e092553691b6749ec83
Jul 17 08:07:13.285068 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.285033    2584 scope.go:117] "RemoveContainer" containerID="054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64"
Jul 17 08:07:13.371081 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.371044    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:07:13.371461 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.371430    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" podUID="cd7ce5d7-0042-4680-9235-108094dbcb1f" containerName="maas-api" containerID="cri-o://4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec" gracePeriod=30
Jul 17 08:07:13.400903 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.400868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:13.607747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.607722    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:07:13.684131 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.684100    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca\") pod \"cd7ce5d7-0042-4680-9235-108094dbcb1f\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") "
Jul 17 08:07:13.684658 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.684146    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") pod \"cd7ce5d7-0042-4680-9235-108094dbcb1f\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") "
Jul 17 08:07:13.684658 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.684220    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tnww\" (UniqueName: \"kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww\") pod \"cd7ce5d7-0042-4680-9235-108094dbcb1f\" (UID: \"cd7ce5d7-0042-4680-9235-108094dbcb1f\") "
Jul 17 08:07:13.684658 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.684537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd7ce5d7-0042-4680-9235-108094dbcb1f" (UID: "cd7ce5d7-0042-4680-9235-108094dbcb1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:07:13.686381 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.686352    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd7ce5d7-0042-4680-9235-108094dbcb1f" (UID: "cd7ce5d7-0042-4680-9235-108094dbcb1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 08:07:13.686520 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.686419    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww" (OuterVolumeSpecName: "kube-api-access-5tnww") pod "cd7ce5d7-0042-4680-9235-108094dbcb1f" (UID: "cd7ce5d7-0042-4680-9235-108094dbcb1f"). InnerVolumeSpecName "kube-api-access-5tnww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 08:07:13.690059 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.690031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerStarted","Data":"7beff277c38b33bc1eb85bf4b3695b1c7adaf778abf22e8c59b7118aa15c84d0"}
Jul 17 08:07:13.691492 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.691463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerStarted","Data":"6960191e42e7c0dd28e8cd977d4514090189fea07b5aad7ae1bcb2bb114f45e0"}
Jul 17 08:07:13.691594 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.691515    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerStarted","Data":"2bfac23a224c3d6e8578fa96d9b41f1957e27dedb0048e092553691b6749ec83"}
Jul 17 08:07:13.692579 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.692556    2584 generic.go:356] "Generic (PLEG): container finished" podID="cd7ce5d7-0042-4680-9235-108094dbcb1f" containerID="4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec" exitCode=0
Jul 17 08:07:13.692692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.692588    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"
Jul 17 08:07:13.692692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.692594    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" event={"ID":"cd7ce5d7-0042-4680-9235-108094dbcb1f","Type":"ContainerDied","Data":"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"}
Jul 17 08:07:13.692692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.692624    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p" event={"ID":"cd7ce5d7-0042-4680-9235-108094dbcb1f","Type":"ContainerDied","Data":"a5f1582de394fe52a6a6f8c7e3ee631238987c9a5ee315a14e2ca928ea3bbd43"}
Jul 17 08:07:13.692692 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.692648    2584 scope.go:117] "RemoveContainer" containerID="4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"
Jul 17 08:07:13.700830 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.700810    2584 scope.go:117] "RemoveContainer" containerID="4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"
Jul 17 08:07:13.701076 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:13.701059    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec\": container with ID starting with 4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec not found: ID does not exist" containerID="4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"
Jul 17 08:07:13.701122 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.701085    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec"} err="failed to get container status \"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec\": rpc error: code = NotFound desc = could not find container \"4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec\": container with ID starting with 4137611e0e2e179bc77696757a49be99a449a75102b842a46959475e6bc523ec not found: ID does not exist"
Jul 17 08:07:13.750694 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.750627    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:07:13.760909 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.760884    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-04d1db-7f8bbb5dfd-dnz8p"]
Jul 17 08:07:13.779526 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.779481    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" podStartSLOduration=1.779466381 podStartE2EDuration="1.779466381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:07:13.778970289 +0000 UTC m=+3330.032393101" watchObservedRunningTime="2026-07-17 08:07:13.779466381 +0000 UTC m=+3330.032889193"
Jul 17 08:07:13.784943 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.784919    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ce5d7-0042-4680-9235-108094dbcb1f-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:13.784943 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.784943    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7ce5d7-0042-4680-9235-108094dbcb1f-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:13.785265 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:13.784952    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tnww\" (UniqueName: \"kubernetes.io/projected/cd7ce5d7-0042-4680-9235-108094dbcb1f-kube-api-access-5tnww\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:14.289801 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.289772    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7ce5d7-0042-4680-9235-108094dbcb1f" path="/var/lib/kubelet/pods/cd7ce5d7-0042-4680-9235-108094dbcb1f/volumes"
Jul 17 08:07:14.698580 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.698502    2584 generic.go:356] "Generic (PLEG): container finished" podID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerID="7beff277c38b33bc1eb85bf4b3695b1c7adaf778abf22e8c59b7118aa15c84d0" exitCode=6
Jul 17 08:07:14.699051 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.698577    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerDied","Data":"7beff277c38b33bc1eb85bf4b3695b1c7adaf778abf22e8c59b7118aa15c84d0"}
Jul 17 08:07:14.699051 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.698639    2584 scope.go:117] "RemoveContainer" containerID="054ee3d8a82e044fda5410239bbf73a9021890107b834a831016dfdecd107d64"
Jul 17 08:07:14.699051 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.698893    2584 scope.go:117] "RemoveContainer" containerID="7beff277c38b33bc1eb85bf4b3695b1c7adaf778abf22e8c59b7118aa15c84d0"
Jul 17 08:07:14.699204 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:14.699103    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85_odh-ai-gateway-infra(c97e8ae6-cbb2-465f-9f1b-716e4ed97408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408"
Jul 17 08:07:14.702610 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.702513    2584 generic.go:356] "Generic (PLEG): container finished" podID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerID="6960191e42e7c0dd28e8cd977d4514090189fea07b5aad7ae1bcb2bb114f45e0" exitCode=6
Jul 17 08:07:14.702724 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.702650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerDied","Data":"6960191e42e7c0dd28e8cd977d4514090189fea07b5aad7ae1bcb2bb114f45e0"}
Jul 17 08:07:14.702954 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.702937    2584 scope.go:117] "RemoveContainer" containerID="6960191e42e7c0dd28e8cd977d4514090189fea07b5aad7ae1bcb2bb114f45e0"
Jul 17 08:07:14.737030 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:14.737004    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:07:15.711999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.711970    2584 generic.go:356] "Generic (PLEG): container finished" podID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerID="2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b" exitCode=6
Jul 17 08:07:15.712330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.712051    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerDied","Data":"2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b"}
Jul 17 08:07:15.712330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.712109    2584 scope.go:117] "RemoveContainer" containerID="6960191e42e7c0dd28e8cd977d4514090189fea07b5aad7ae1bcb2bb114f45e0"
Jul 17 08:07:15.712330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.712315    2584 scope.go:117] "RemoveContainer" containerID="2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b"
Jul 17 08:07:15.712546 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:15.712528    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq_odh-ai-gateway-infra(64f46d6c-3849-4c2a-a481-76dd959f4268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268"
Jul 17 08:07:15.842421 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.842398    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:07:15.901866 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.901834    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca\") pod \"c97e8ae6-cbb2-465f-9f1b-716e4ed97408\" (UID: \"c97e8ae6-cbb2-465f-9f1b-716e4ed97408\") "
Jul 17 08:07:15.902212 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:15.902194    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c97e8ae6-cbb2-465f-9f1b-716e4ed97408" (UID: "c97e8ae6-cbb2-465f-9f1b-716e4ed97408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:07:16.002803 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.002715    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97e8ae6-cbb2-465f-9f1b-716e4ed97408-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:16.718412 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.718308    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"
Jul 17 08:07:16.718412 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.718329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85" event={"ID":"c97e8ae6-cbb2-465f-9f1b-716e4ed97408","Type":"ContainerDied","Data":"11be0e3a1f16bbf1a79dee3f8955135c5e6a10c24867ffc2128c7893aca9ee0a"}
Jul 17 08:07:16.718928 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.718414    2584 scope.go:117] "RemoveContainer" containerID="7beff277c38b33bc1eb85bf4b3695b1c7adaf778abf22e8c59b7118aa15c84d0"
Jul 17 08:07:16.720419 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.720398    2584 scope.go:117] "RemoveContainer" containerID="2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b"
Jul 17 08:07:16.720656 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:16.720629    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq_odh-ai-gateway-infra(64f46d6c-3849-4c2a-a481-76dd959f4268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268"
Jul 17 08:07:16.736877 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.736850    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:07:16.740912 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.740892    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-wdz85"]
Jul 17 08:07:16.769393 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769355    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:16.769724 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769712    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7ce5d7-0042-4680-9235-108094dbcb1f" containerName="maas-api"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769726    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7ce5d7-0042-4680-9235-108094dbcb1f" containerName="maas-api"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769741    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769747    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769755    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769763    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769775 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769775    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769780    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769830    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7ce5d7-0042-4680-9235-108094dbcb1f" containerName="maas-api"
Jul 17 08:07:16.769956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769842    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769851    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.769956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.769859    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" containerName="revoke-keys"
Jul 17 08:07:16.774287 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.774269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:16.780070 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.780035    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:16.818748 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.818712    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6\" (UID: \"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:16.919272 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.919236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6\" (UID: \"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:16.919874 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:16.919853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6\" (UID: \"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:17.032116 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.032030    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sh6l"]
Jul 17 08:07:17.086989 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.086954    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:17.228203 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.228172    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:17.229404 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:07:17.229375    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f78a72_09bd_4fb6_bdd5_211d8c3f47d2.slice/crio-f190ab2d2b9d3d3ec928e50c02c1decc412d3040b27e915e5e12b47bab24ecb0 WatchSource:0}: Error finding container f190ab2d2b9d3d3ec928e50c02c1decc412d3040b27e915e5e12b47bab24ecb0: Status 404 returned error can't find the container with id f190ab2d2b9d3d3ec928e50c02c1decc412d3040b27e915e5e12b47bab24ecb0
Jul 17 08:07:17.728273 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.728179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerStarted","Data":"6f912834e3cb3c4aca6c4d8871b065cc439208c9197e0f4e00a9b85d3fabae48"}
Jul 17 08:07:17.728273 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.728235    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerStarted","Data":"f190ab2d2b9d3d3ec928e50c02c1decc412d3040b27e915e5e12b47bab24ecb0"}
Jul 17 08:07:17.751558 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.751506    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" podStartSLOduration=1.751489453 podStartE2EDuration="1.751489453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:07:17.750629449 +0000 UTC m=+3334.004052261" watchObservedRunningTime="2026-07-17 08:07:17.751489453 +0000 UTC m=+3334.004912308"
Jul 17 08:07:17.903380 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:17.903324    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:18.290310 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:18.290277    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c97e8ae6-cbb2-465f-9f1b-716e4ed97408" path="/var/lib/kubelet/pods/c97e8ae6-cbb2-465f-9f1b-716e4ed97408/volumes"
Jul 17 08:07:18.734696 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:18.734596    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerID="6f912834e3cb3c4aca6c4d8871b065cc439208c9197e0f4e00a9b85d3fabae48" exitCode=6
Jul 17 08:07:18.734696 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:18.734685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerDied","Data":"6f912834e3cb3c4aca6c4d8871b065cc439208c9197e0f4e00a9b85d3fabae48"}
Jul 17 08:07:18.735212 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:18.734900    2584 scope.go:117] "RemoveContainer" containerID="6f912834e3cb3c4aca6c4d8871b065cc439208c9197e0f4e00a9b85d3fabae48"
Jul 17 08:07:19.740851 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:19.740764    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerID="bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a" exitCode=6
Jul 17 08:07:19.740851 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:19.740829    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerDied","Data":"bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a"}
Jul 17 08:07:19.741313 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:19.740887    2584 scope.go:117] "RemoveContainer" containerID="6f912834e3cb3c4aca6c4d8871b065cc439208c9197e0f4e00a9b85d3fabae48"
Jul 17 08:07:19.741313 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:19.741059    2584 scope.go:117] "RemoveContainer" containerID="bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a"
Jul 17 08:07:19.741313 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:19.741254    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6_odh-ai-gateway-infra(13f78a72-09bd-4fb6-bdd5-211d8c3f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2"
Jul 17 08:07:20.746540 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:20.746512    2584 scope.go:117] "RemoveContainer" containerID="bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a"
Jul 17 08:07:20.746920 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:20.746698    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6_odh-ai-gateway-infra(13f78a72-09bd-4fb6-bdd5-211d8c3f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2"
Jul 17 08:07:24.148695 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:24.148659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:24.250588 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:24.250547    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:28.284939 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:28.284900    2584 scope.go:117] "RemoveContainer" containerID="2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b"
Jul 17 08:07:28.698484 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:28.698394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:28.786888 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:28.786857    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerStarted","Data":"3402ad9e9884496de0bb6fab61faefb36bc2a196a50d0561a1256df79b43c7fa"}
Jul 17 08:07:28.945725 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:28.945690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:29.793985 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:29.793944    2584 generic.go:356] "Generic (PLEG): container finished" podID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerID="3402ad9e9884496de0bb6fab61faefb36bc2a196a50d0561a1256df79b43c7fa" exitCode=6
Jul 17 08:07:29.794458 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:29.794017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerDied","Data":"3402ad9e9884496de0bb6fab61faefb36bc2a196a50d0561a1256df79b43c7fa"}
Jul 17 08:07:29.794458 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:29.794080    2584 scope.go:117] "RemoveContainer" containerID="2faed645217cfb591961e23e368e255dab2c520a3adcff39b7524b7ddc8e073b"
Jul 17 08:07:29.794458 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:29.794318    2584 scope.go:117] "RemoveContainer" containerID="3402ad9e9884496de0bb6fab61faefb36bc2a196a50d0561a1256df79b43c7fa"
Jul 17 08:07:29.794596 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:29.794566    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq_odh-ai-gateway-infra(64f46d6c-3849-4c2a-a481-76dd959f4268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268"
Jul 17 08:07:29.824756 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:29.824724    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:30.933420 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:30.933393    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:31.020836 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.020804    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca\") pod \"64f46d6c-3849-4c2a-a481-76dd959f4268\" (UID: \"64f46d6c-3849-4c2a-a481-76dd959f4268\") "
Jul 17 08:07:31.021147 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.021112    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64f46d6c-3849-4c2a-a481-76dd959f4268" (UID: "64f46d6c-3849-4c2a-a481-76dd959f4268"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:07:31.122096 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.122002    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f46d6c-3849-4c2a-a481-76dd959f4268-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:31.805796 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.805767    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"
Jul 17 08:07:31.805796 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.805780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq" event={"ID":"64f46d6c-3849-4c2a-a481-76dd959f4268","Type":"ContainerDied","Data":"2bfac23a224c3d6e8578fa96d9b41f1957e27dedb0048e092553691b6749ec83"}
Jul 17 08:07:31.806058 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.805837    2584 scope.go:117] "RemoveContainer" containerID="3402ad9e9884496de0bb6fab61faefb36bc2a196a50d0561a1256df79b43c7fa"
Jul 17 08:07:31.834009 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.833977    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:31.838460 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:31.838435    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-rv9bq"]
Jul 17 08:07:32.290176 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:32.290144    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" path="/var/lib/kubelet/pods/64f46d6c-3849-4c2a-a481-76dd959f4268/volumes"
Jul 17 08:07:33.284569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:33.284540    2584 scope.go:117] "RemoveContainer" containerID="bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a"
Jul 17 08:07:33.817802 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:33.817764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerStarted","Data":"940560d44ce2aef902d1fe07ae7300d3c03d6b45aea4adb3a4e7cfa6aee7459e"}
Jul 17 08:07:34.824116 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:34.824082    2584 generic.go:356] "Generic (PLEG): container finished" podID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerID="940560d44ce2aef902d1fe07ae7300d3c03d6b45aea4adb3a4e7cfa6aee7459e" exitCode=6
Jul 17 08:07:34.824669 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:34.824159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerDied","Data":"940560d44ce2aef902d1fe07ae7300d3c03d6b45aea4adb3a4e7cfa6aee7459e"}
Jul 17 08:07:34.824669 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:34.824223    2584 scope.go:117] "RemoveContainer" containerID="bb08d6a09c8fce8fb051084feaf2104c2c79a0a292e543ee857d32bd7e1a013a"
Jul 17 08:07:34.824669 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:34.824531    2584 scope.go:117] "RemoveContainer" containerID="940560d44ce2aef902d1fe07ae7300d3c03d6b45aea4adb3a4e7cfa6aee7459e"
Jul 17 08:07:34.824793 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:34.824729    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6_odh-ai-gateway-infra(13f78a72-09bd-4fb6-bdd5-211d8c3f47d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2"
Jul 17 08:07:34.853628 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:34.853600    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:35.959637 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:35.959608    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:36.061729 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.061695    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca\") pod \"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2\" (UID: \"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2\") "
Jul 17 08:07:36.062019 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.061997    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" (UID: "13f78a72-09bd-4fb6-bdd5-211d8c3f47d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:07:36.162414 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.162328    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:07:36.323265 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.323236    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2_fb05d8cf-34b6-4b5b-8be8-d19d288b3802/copy-manifests/0.log"
Jul 17 08:07:36.331600 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.331581    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2_fb05d8cf-34b6-4b5b-8be8-d19d288b3802/manager/0.log"
Jul 17 08:07:36.835689 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.835662    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"
Jul 17 08:07:36.835873 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.835700    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6" event={"ID":"13f78a72-09bd-4fb6-bdd5-211d8c3f47d2","Type":"ContainerDied","Data":"f190ab2d2b9d3d3ec928e50c02c1decc412d3040b27e915e5e12b47bab24ecb0"}
Jul 17 08:07:36.835873 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.835753    2584 scope.go:117] "RemoveContainer" containerID="940560d44ce2aef902d1fe07ae7300d3c03d6b45aea4adb3a4e7cfa6aee7459e"
Jul 17 08:07:36.856836 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.856812    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:36.861369 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:36.861330    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-xcbp6"]
Jul 17 08:07:37.117519 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:37.117448    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737905-b544n_a0ec5d82-16ae-449d-b61b-f00aa11d829b/cleanup/0.log"
Jul 17 08:07:37.478672 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:37.478598    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-5cf58d-h5cw8_7240de31-df71-4ec4-9dc2-5906b6b1bfc8/revoke-keys/0.log"
Jul 17 08:07:37.599259 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:37.599225    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-38e02d-jr8gx_cbfcecbb-fd49-411a-9e03-36afb3aac785/revoke-keys/0.log"
Jul 17 08:07:37.725741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:37.725712    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-1f268b-t4dhw_bdc40031-d917-49a8-9ee8-f4c83090ce0b/revoke-keys/0.log"
Jul 17 08:07:37.849416 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:37.849388    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-cab91c-l2g64_c9e64d70-3a41-495d-a36c-43d56784df99/revoke-keys/0.log"
Jul 17 08:07:38.290419 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:38.290391    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" path="/var/lib/kubelet/pods/13f78a72-09bd-4fb6-bdd5-211d8c3f47d2/volumes"
Jul 17 08:07:38.344460 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:38.344430    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-04d1db-92bt7_0eb948e0-e651-4b60-a09e-6f27efbd5ff9/revoke-keys/0.log"
Jul 17 08:07:39.810501 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810460    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810910    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810929    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810939    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810947    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810960    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810967    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810976    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.810985    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.810999 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811002    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.811493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811011    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.811493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811127    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.811493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811140    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:39.811493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811150    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.811493 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.811159    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:39.815578 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.815556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:07:39.818460 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.818438    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c65vn\""
Jul 17 08:07:39.818591 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.818438    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 08:07:39.823142 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.823117    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:07:39.893536 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.893508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm\" (UID: \"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:07:39.994370 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.994303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm\" (UID: \"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:07:39.994993 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:39.994968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm\" (UID: \"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:07:40.127359 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.127269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:07:40.255620 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.255597    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:07:40.257767 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:07:40.257741    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5186cc_f380_4bfb_a07f_d4fd366b2cb1.slice/crio-c9147bc3acc51a51157bf46e73a6791522abfb51e8715250a30ba0817b440b80 WatchSource:0}: Error finding container c9147bc3acc51a51157bf46e73a6791522abfb51e8715250a30ba0817b440b80: Status 404 returned error can't find the container with id c9147bc3acc51a51157bf46e73a6791522abfb51e8715250a30ba0817b440b80
Jul 17 08:07:40.717931 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.717852    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2sh6l_d3e910ce-7d62-4baa-ad7b-e506ab53ce37/limitador/0.log"
Jul 17 08:07:40.858357 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.858302    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerStarted","Data":"c7499094a1a537c2efc732a5287adb62b5828b05a54961afa5d2ccc49f945ad4"}
Jul 17 08:07:40.858732 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.858375    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerStarted","Data":"c9147bc3acc51a51157bf46e73a6791522abfb51e8715250a30ba0817b440b80"}
Jul 17 08:07:40.879321 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:40.879267    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" podStartSLOduration=1.879251987 podStartE2EDuration="1.879251987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 08:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:07:40.876278393 +0000 UTC m=+3357.129701204" watchObservedRunningTime="2026-07-17 08:07:40.879251987 +0000 UTC m=+3357.132674798"
Jul 17 08:07:41.369590 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:41.369556    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6nrdp_8b522f30-8d75-48da-b8ec-edf7f90e359b/discovery/0.log"
Jul 17 08:07:41.864629 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:41.864595    2584 generic.go:356] "Generic (PLEG): container finished" podID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerID="c7499094a1a537c2efc732a5287adb62b5828b05a54961afa5d2ccc49f945ad4" exitCode=6
Jul 17 08:07:41.865038 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:41.864679    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerDied","Data":"c7499094a1a537c2efc732a5287adb62b5828b05a54961afa5d2ccc49f945ad4"}
Jul 17 08:07:41.865038 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:41.864858    2584 scope.go:117] "RemoveContainer" containerID="c7499094a1a537c2efc732a5287adb62b5828b05a54961afa5d2ccc49f945ad4"
Jul 17 08:07:42.666530 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.666499    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-w9f5k_6c4611d2-bbb8-440f-88a2-b67aea298503/main/0.log"
Jul 17 08:07:42.796758 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.796734    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4mxnb_ac641c8b-a596-4366-91b0-cb2a9ad176d3/main/0.log"
Jul 17 08:07:42.870767 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.870727    2584 generic.go:356] "Generic (PLEG): container finished" podID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerID="f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090" exitCode=6
Jul 17 08:07:42.871184 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.870786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerDied","Data":"f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090"}
Jul 17 08:07:42.871184 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.870868    2584 scope.go:117] "RemoveContainer" containerID="c7499094a1a537c2efc732a5287adb62b5828b05a54961afa5d2ccc49f945ad4"
Jul 17 08:07:42.871184 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:42.871111    2584 scope.go:117] "RemoveContainer" containerID="f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090"
Jul 17 08:07:42.871395 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:42.871366    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm_odh-ai-gateway-infra(9c5186cc-f380-4bfb-a07f-d4fd366b2cb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1"
Jul 17 08:07:43.877403 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:43.877375    2584 scope.go:117] "RemoveContainer" containerID="f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090"
Jul 17 08:07:43.877787 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:43.877554    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm_odh-ai-gateway-infra(9c5186cc-f380-4bfb-a07f-d4fd366b2cb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1"
Jul 17 08:07:49.851329 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:49.851260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:49.954077 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:49.954043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log" limit=1024
Jul 17 08:07:52.448783 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:52.448753    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hktn8_8ba69a4b-39b3-4d83-83be-47497ce7d246/global-pull-secret-syncer/0.log"
Jul 17 08:07:52.543419 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:52.543387    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8wl8r_1b94e9c3-49d0-41ef-9e47-4c53d4246c96/konnectivity-agent/0.log"
Jul 17 08:07:52.675240 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:52.675207    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-131.ec2.internal_ca0c79730331515f06ff8b1a5acdfdf5/haproxy/0.log"
Jul 17 08:07:54.446722 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.446692    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:07:54.447089 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.446992    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:54.447089 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.447003    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:54.447089 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.447063    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f78a72-09bd-4fb6-bdd5-211d8c3f47d2" containerName="revoke-keys"
Jul 17 08:07:54.447089 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.447072    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f46d6c-3849-4c2a-a481-76dd959f4268" containerName="revoke-keys"
Jul 17 08:07:54.451651 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.451632    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:07:54.457932 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.457908    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:07:54.512741 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.512711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z\" (UID: \"36467962-7274-4fa8-9c4e-194c1959a019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:07:54.613700 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.613667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z\" (UID: \"36467962-7274-4fa8-9c4e-194c1959a019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:07:54.614258 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.614237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z\" (UID: \"36467962-7274-4fa8-9c4e-194c1959a019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:07:54.762494 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.762401    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:07:54.902976 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.902949    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:07:54.904531 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:07:54.904504    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36467962_7274_4fa8_9c4e_194c1959a019.slice/crio-d95b140e3cdf0cf19b07965738b120c41c18c7f963504bece4a2665f9ed70272 WatchSource:0}: Error finding container d95b140e3cdf0cf19b07965738b120c41c18c7f963504bece4a2665f9ed70272: Status 404 returned error can't find the container with id d95b140e3cdf0cf19b07965738b120c41c18c7f963504bece4a2665f9ed70272
Jul 17 08:07:54.927438 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:54.927414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" event={"ID":"36467962-7274-4fa8-9c4e-194c1959a019","Type":"ContainerStarted","Data":"d95b140e3cdf0cf19b07965738b120c41c18c7f963504bece4a2665f9ed70272"}
Jul 17 08:07:55.933879 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:55.933842    2584 generic.go:356] "Generic (PLEG): container finished" podID="36467962-7274-4fa8-9c4e-194c1959a019" containerID="50c2e3221e209ef0659c0b8637cb71393fa4e2c519d8cec86d926510fcfcaf56" exitCode=6
Jul 17 08:07:55.934257 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:55.933930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" event={"ID":"36467962-7274-4fa8-9c4e-194c1959a019","Type":"ContainerDied","Data":"50c2e3221e209ef0659c0b8637cb71393fa4e2c519d8cec86d926510fcfcaf56"}
Jul 17 08:07:55.934257 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:55.934086    2584 scope.go:117] "RemoveContainer" containerID="50c2e3221e209ef0659c0b8637cb71393fa4e2c519d8cec86d926510fcfcaf56"
Jul 17 08:07:56.940242 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:56.940208    2584 generic.go:356] "Generic (PLEG): container finished" podID="36467962-7274-4fa8-9c4e-194c1959a019" containerID="a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9" exitCode=6
Jul 17 08:07:56.940732 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:56.940278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" event={"ID":"36467962-7274-4fa8-9c4e-194c1959a019","Type":"ContainerDied","Data":"a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9"}
Jul 17 08:07:56.940732 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:56.940357    2584 scope.go:117] "RemoveContainer" containerID="50c2e3221e209ef0659c0b8637cb71393fa4e2c519d8cec86d926510fcfcaf56"
Jul 17 08:07:56.940732 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:56.940534    2584 scope.go:117] "RemoveContainer" containerID="a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9"
Jul 17 08:07:56.940855 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:56.940727    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z_odh-ai-gateway-infra(36467962-7274-4fa8-9c4e-194c1959a019)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" podUID="36467962-7274-4fa8-9c4e-194c1959a019"
Jul 17 08:07:57.946536 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:57.946510    2584 scope.go:117] "RemoveContainer" containerID="a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9"
Jul 17 08:07:57.946910 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:57.946704    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z_odh-ai-gateway-infra(36467962-7274-4fa8-9c4e-194c1959a019)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" podUID="36467962-7274-4fa8-9c4e-194c1959a019"
Jul 17 08:07:58.286521 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.286465    2584 scope.go:117] "RemoveContainer" containerID="f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090"
Jul 17 08:07:58.385766 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.385738    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2sh6l_d3e910ce-7d62-4baa-ad7b-e506ab53ce37/limitador/0.log"
Jul 17 08:07:58.952301 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.952266    2584 generic.go:356] "Generic (PLEG): container finished" podID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerID="b5de0a7d1af76e826948056f405dade82395c22c807aa8e81b72d2374346e6f8" exitCode=6
Jul 17 08:07:58.952799 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.952361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerDied","Data":"b5de0a7d1af76e826948056f405dade82395c22c807aa8e81b72d2374346e6f8"}
Jul 17 08:07:58.952799 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.952423    2584 scope.go:117] "RemoveContainer" containerID="f2a3d049394acc19d75f0151ad69618f90a151a6a4925e79c9bf871a65730090"
Jul 17 08:07:58.952799 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:58.952700    2584 scope.go:117] "RemoveContainer" containerID="b5de0a7d1af76e826948056f405dade82395c22c807aa8e81b72d2374346e6f8"
Jul 17 08:07:58.952989 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:07:58.952934    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm_odh-ai-gateway-infra(9c5186cc-f380-4bfb-a07f-d4fd366b2cb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1"
Jul 17 08:07:59.983529 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:07:59.983443    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:08:00.113082 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.113059    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:08:00.259613 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.259527    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca\") pod \"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1\" (UID: \"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1\") "
Jul 17 08:08:00.259902 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.259879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" (UID: "9c5186cc-f380-4bfb-a07f-d4fd366b2cb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:08:00.361200 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.361146    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:08:00.798963 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.798929    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbsqc_450a10a1-b8ba-413d-bbcf-23cad9bc2461/node-exporter/0.log"
Jul 17 08:08:00.823819 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.823790    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbsqc_450a10a1-b8ba-413d-bbcf-23cad9bc2461/kube-rbac-proxy/0.log"
Jul 17 08:08:00.855883 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.855853    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbsqc_450a10a1-b8ba-413d-bbcf-23cad9bc2461/init-textfile/0.log"
Jul 17 08:08:00.963501 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.963462    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm" event={"ID":"9c5186cc-f380-4bfb-a07f-d4fd366b2cb1","Type":"ContainerDied","Data":"c9147bc3acc51a51157bf46e73a6791522abfb51e8715250a30ba0817b440b80"}
Jul 17 08:08:00.963501 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.963501    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"
Jul 17 08:08:00.963730 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.963565    2584 scope.go:117] "RemoveContainer" containerID="b5de0a7d1af76e826948056f405dade82395c22c807aa8e81b72d2374346e6f8"
Jul 17 08:08:00.982484 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.982459    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:08:00.986958 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:00.986936    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dfadf83-gb2mm"]
Jul 17 08:08:01.259291 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:01.259216    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-g688l_a9e5dd25-9460-45cb-b6cf-e656df275567/prometheus-operator-admission-webhook/0.log"
Jul 17 08:08:02.193435 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193400    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"]
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193710    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193721    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193735    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193740    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193746    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193752    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193810    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.193812 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.193816    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" containerName="revoke-keys"
Jul 17 08:08:02.198056 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.198038    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.201751 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.201724    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7q6q\"/\"kube-root-ca.crt\""
Jul 17 08:08:02.201927 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.201905    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7q6q\"/\"openshift-service-ca.crt\""
Jul 17 08:08:02.202739 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.202723    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p7q6q\"/\"default-dockercfg-9bq8c\""
Jul 17 08:08:02.207224 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.207190    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"]
Jul 17 08:08:02.277169 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.277126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-podres\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.277396 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.277192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prd5k\" (UniqueName: \"kubernetes.io/projected/f72a948c-d1d0-4790-a037-257ebf67c948-kube-api-access-prd5k\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.277396 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.277222    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-sys\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.277396 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.277237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-lib-modules\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.277396 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.277274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-proc\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.291029 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.291002    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5186cc-f380-4bfb-a07f-d4fd366b2cb1" path="/var/lib/kubelet/pods/9c5186cc-f380-4bfb-a07f-d4fd366b2cb1/volumes"
Jul 17 08:08:02.378605 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-podres\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd5k\" (UniqueName: \"kubernetes.io/projected/f72a948c-d1d0-4790-a037-257ebf67c948-kube-api-access-prd5k\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-sys\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-lib-modules\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378724    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-proc\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378760    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-podres\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-sys\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.378806 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-proc\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.379049 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.378854    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f72a948c-d1d0-4790-a037-257ebf67c948-lib-modules\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.387883 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.387859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd5k\" (UniqueName: \"kubernetes.io/projected/f72a948c-d1d0-4790-a037-257ebf67c948-kube-api-access-prd5k\") pod \"perf-node-gather-daemonset-wqntc\" (UID: \"f72a948c-d1d0-4790-a037-257ebf67c948\") " pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.508394 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.508317    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.639471 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.639440    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"]
Jul 17 08:08:02.641039 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:08:02.641007    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf72a948c_d1d0_4790_a037_257ebf67c948.slice/crio-3a66b3576f498c5e3ba27550d59207dfd02436df51bfd7772219d714007fde65 WatchSource:0}: Error finding container 3a66b3576f498c5e3ba27550d59207dfd02436df51bfd7772219d714007fde65: Status 404 returned error can't find the container with id 3a66b3576f498c5e3ba27550d59207dfd02436df51bfd7772219d714007fde65
Jul 17 08:08:02.739517 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.739485    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:02.742569 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.742552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:02.757956 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.757931    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:02.777329 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.777270    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-gm8nc_fdc295c0-e400-4da9-91fe-b23573af849e/networking-console-plugin/0.log"
Jul 17 08:08:02.884752 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.884720    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4\" (UID: \"8a646863-39d5-49a2-a29a-8b89e3b6319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:02.975072 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.975037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc" event={"ID":"f72a948c-d1d0-4790-a037-257ebf67c948","Type":"ContainerStarted","Data":"99fea30593e4249368d9eacf66be7518a8a2886c3718abca09b5ca19d27efa95"}
Jul 17 08:08:02.975245 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.975088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc" event={"ID":"f72a948c-d1d0-4790-a037-257ebf67c948","Type":"ContainerStarted","Data":"3a66b3576f498c5e3ba27550d59207dfd02436df51bfd7772219d714007fde65"}
Jul 17 08:08:02.975245 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.975117    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:02.985348 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.985308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4\" (UID: \"8a646863-39d5-49a2-a29a-8b89e3b6319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:02.985881 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.985863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4\" (UID: \"8a646863-39d5-49a2-a29a-8b89e3b6319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:02.994863 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:02.994818    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc" podStartSLOduration=0.994804998 podStartE2EDuration="994.804998ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 08:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 08:08:02.993733411 +0000 UTC m=+3379.247156249" watchObservedRunningTime="2026-07-17 08:08:02.994804998 +0000 UTC m=+3379.248227809"
Jul 17 08:08:03.052968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.052938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:03.193355 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.193314    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:03.193476 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:08:03.193349    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a646863_39d5_49a2_a29a_8b89e3b6319e.slice/crio-e226f7f877046215e213e9ea8561cd72f44d87a35a139a322bb3f318e7011ad4 WatchSource:0}: Error finding container e226f7f877046215e213e9ea8561cd72f44d87a35a139a322bb3f318e7011ad4: Status 404 returned error can't find the container with id e226f7f877046215e213e9ea8561cd72f44d87a35a139a322bb3f318e7011ad4
Jul 17 08:08:03.914769 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.914736    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-9nd6t_9c6a280d-1dc5-491c-b5ec-77ccd0af9f48/download-server/0.log"
Jul 17 08:08:03.981165 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.981131    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a646863-39d5-49a2-a29a-8b89e3b6319e" containerID="cc6c5fa47aeca6e37ba2e224b745d5bfb66f1a717b4db526519f117623b792d1" exitCode=6
Jul 17 08:08:03.981330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.981218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" event={"ID":"8a646863-39d5-49a2-a29a-8b89e3b6319e","Type":"ContainerDied","Data":"cc6c5fa47aeca6e37ba2e224b745d5bfb66f1a717b4db526519f117623b792d1"}
Jul 17 08:08:03.981330 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.981279    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" event={"ID":"8a646863-39d5-49a2-a29a-8b89e3b6319e","Type":"ContainerStarted","Data":"e226f7f877046215e213e9ea8561cd72f44d87a35a139a322bb3f318e7011ad4"}
Jul 17 08:08:03.981466 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:03.981415    2584 scope.go:117] "RemoveContainer" containerID="cc6c5fa47aeca6e37ba2e224b745d5bfb66f1a717b4db526519f117623b792d1"
Jul 17 08:08:04.987358 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:04.987303    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a646863-39d5-49a2-a29a-8b89e3b6319e" containerID="e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26" exitCode=6
Jul 17 08:08:04.987837 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:04.987374    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" event={"ID":"8a646863-39d5-49a2-a29a-8b89e3b6319e","Type":"ContainerDied","Data":"e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26"}
Jul 17 08:08:04.987837 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:04.987450    2584 scope.go:117] "RemoveContainer" containerID="cc6c5fa47aeca6e37ba2e224b745d5bfb66f1a717b4db526519f117623b792d1"
Jul 17 08:08:04.987837 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:04.987655    2584 scope.go:117] "RemoveContainer" containerID="e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26"
Jul 17 08:08:04.988025 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:04.987896    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4_odh-ai-gateway-infra(8a646863-39d5-49a2-a29a-8b89e3b6319e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" podUID="8a646863-39d5-49a2-a29a-8b89e3b6319e"
Jul 17 08:08:05.613986 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:05.613959    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnzc7_f8ea191a-a858-477f-8f34-31390a10e8a7/dns/0.log"
Jul 17 08:08:05.639285 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:05.639246    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnzc7_f8ea191a-a858-477f-8f34-31390a10e8a7/kube-rbac-proxy/0.log"
Jul 17 08:08:05.718986 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:05.718955    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbn4j_352cee13-e768-429f-ac8a-7e179831a97e/dns-node-resolver/0.log"
Jul 17 08:08:05.993803 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:05.993710    2584 scope.go:117] "RemoveContainer" containerID="e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26"
Jul 17 08:08:05.994151 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:05.993903    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4_odh-ai-gateway-infra(8a646863-39d5-49a2-a29a-8b89e3b6319e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" podUID="8a646863-39d5-49a2-a29a-8b89e3b6319e"
Jul 17 08:08:06.362233 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:06.362202    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-666fdc4999-9prdc_21d91072-feb2-4a89-b1b7-96cfaa7f5416/registry/0.log"
Jul 17 08:08:06.436721 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:06.436694    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pfz9q_ba41118a-688c-4ba8-93aa-c61b3ddbc615/node-ca/0.log"
Jul 17 08:08:07.518749 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:07.518713    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6nrdp_8b522f30-8d75-48da-b8ec-edf7f90e359b/discovery/0.log"
Jul 17 08:08:08.477757 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:08.477730    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8fp2_1daeaaed-fba4-4fbf-91ba-83ff2a7b28e4/serve-healthcheck-canary/0.log"
Jul 17 08:08:08.991145 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:08.991116    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p7q6q/perf-node-gather-daemonset-wqntc"
Jul 17 08:08:09.122491 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:09.122467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmxnh_b1a186c4-375f-4991-98b8-df6ccfd0d349/kube-rbac-proxy/0.log"
Jul 17 08:08:09.152405 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:09.152380    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmxnh_b1a186c4-375f-4991-98b8-df6ccfd0d349/exporter/0.log"
Jul 17 08:08:09.178968 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:09.178945    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jmxnh_b1a186c4-375f-4991-98b8-df6ccfd0d349/extractor/0.log"
Jul 17 08:08:10.284941 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:10.284906    2584 scope.go:117] "RemoveContainer" containerID="a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9"
Jul 17 08:08:11.015921 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:11.015891    2584 generic.go:356] "Generic (PLEG): container finished" podID="36467962-7274-4fa8-9c4e-194c1959a019" containerID="4c4fecf2644a0d2b86452b09f81847def6667c21ae5718787490a9a2313c296d" exitCode=6
Jul 17 08:08:11.016063 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:11.015934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" event={"ID":"36467962-7274-4fa8-9c4e-194c1959a019","Type":"ContainerDied","Data":"4c4fecf2644a0d2b86452b09f81847def6667c21ae5718787490a9a2313c296d"}
Jul 17 08:08:11.016063 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:11.015980    2584 scope.go:117] "RemoveContainer" containerID="a82ec5b1d74372ce79b22d36992d4d9bebf7930fab00a6156df193ff26a1a4b9"
Jul 17 08:08:11.016268 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:11.016243    2584 scope.go:117] "RemoveContainer" containerID="4c4fecf2644a0d2b86452b09f81847def6667c21ae5718787490a9a2313c296d"
Jul 17 08:08:11.016508 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:11.016479    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z_odh-ai-gateway-infra(36467962-7274-4fa8-9c4e-194c1959a019)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" podUID="36467962-7274-4fa8-9c4e-194c1959a019"
Jul 17 08:08:12.034108 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.034082    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2_fb05d8cf-34b6-4b5b-8be8-d19d288b3802/manager/0.log"
Jul 17 08:08:12.051496 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.051469    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:08:12.067888 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.067865    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59ffcf4dc8-bqkq2_fb05d8cf-34b6-4b5b-8be8-d19d288b3802/copy-manifests/0.log"
Jul 17 08:08:12.185855 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.185827    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:08:12.364666 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.364632    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca\") pod \"36467962-7274-4fa8-9c4e-194c1959a019\" (UID: \"36467962-7274-4fa8-9c4e-194c1959a019\") "
Jul 17 08:08:12.364959 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.364939    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36467962-7274-4fa8-9c4e-194c1959a019" (UID: "36467962-7274-4fa8-9c4e-194c1959a019"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:08:12.417267 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.417198    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6nrdp_8b522f30-8d75-48da-b8ec-edf7f90e359b/discovery/0.log"
Jul 17 08:08:12.465589 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:12.465566    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36467962-7274-4fa8-9c4e-194c1959a019-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:08:13.027820 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.027790    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"
Jul 17 08:08:13.028015 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.027834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z" event={"ID":"36467962-7274-4fa8-9c4e-194c1959a019","Type":"ContainerDied","Data":"d95b140e3cdf0cf19b07965738b120c41c18c7f963504bece4a2665f9ed70272"}
Jul 17 08:08:13.028015 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.027898    2584 scope.go:117] "RemoveContainer" containerID="4c4fecf2644a0d2b86452b09f81847def6667c21ae5718787490a9a2313c296d"
Jul 17 08:08:13.056612 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.056584    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:08:13.070192 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.070167    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f44a26e0-ftf4z"]
Jul 17 08:08:13.579273 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:13.579247    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fc85c6965-n8s6t_4fc6871d-ac6d-4ee5-a8af-e8616b62d536/manager/0.log"
Jul 17 08:08:14.027563 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:14.027533    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-6nrdp_8b522f30-8d75-48da-b8ec-edf7f90e359b/discovery/0.log"
Jul 17 08:08:14.289785 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:14.289758    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36467962-7274-4fa8-9c4e-194c1959a019" path="/var/lib/kubelet/pods/36467962-7274-4fa8-9c4e-194c1959a019/volumes"
Jul 17 08:08:16.820432 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820397    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"]
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820837    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820854    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820870    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820876    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820886    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.820911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820893    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.821198 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820973    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.821198 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.820988    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36467962-7274-4fa8-9c4e-194c1959a019" containerName="revoke-keys"
Jul 17 08:08:16.827687 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.827665    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"
Jul 17 08:08:16.833070 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:16.833043    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"]
Jul 17 08:08:17.003911 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.003882    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad337426-1741-4a7c-9eeb-3bc574db28f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq\" (UID: \"ad337426-1741-4a7c-9eeb-3bc574db28f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"
Jul 17 08:08:17.104582 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.104496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad337426-1741-4a7c-9eeb-3bc574db28f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq\" (UID: \"ad337426-1741-4a7c-9eeb-3bc574db28f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"
Jul 17 08:08:17.105095 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.105074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad337426-1741-4a7c-9eeb-3bc574db28f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq\" (UID: \"ad337426-1741-4a7c-9eeb-3bc574db28f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"
Jul 17 08:08:17.139583 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.139556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"
Jul 17 08:08:17.271557 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.271522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq"]
Jul 17 08:08:17.274990 ip-10-0-138-131 kubenswrapper[2584]: W0717 08:08:17.274962    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad337426_1741_4a7c_9eeb_3bc574db28f3.slice/crio-bc86aed7cacb5266e714399458ef9f9ec1d9a0a638ec42ef52d15bd044143c96 WatchSource:0}: Error finding container bc86aed7cacb5266e714399458ef9f9ec1d9a0a638ec42ef52d15bd044143c96: Status 404 returned error can't find the container with id bc86aed7cacb5266e714399458ef9f9ec1d9a0a638ec42ef52d15bd044143c96
Jul 17 08:08:17.284418 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:17.284402    2584 scope.go:117] "RemoveContainer" containerID="e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26"
Jul 17 08:08:18.052870 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.052845    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a646863-39d5-49a2-a29a-8b89e3b6319e" containerID="bba81876375b16f5b9766a50a3c72bb9db4642d72fdaa63fba42874ef6140e3a" exitCode=6
Jul 17 08:08:18.053238 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.052917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" event={"ID":"8a646863-39d5-49a2-a29a-8b89e3b6319e","Type":"ContainerDied","Data":"bba81876375b16f5b9766a50a3c72bb9db4642d72fdaa63fba42874ef6140e3a"}
Jul 17 08:08:18.053238 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.052980    2584 scope.go:117] "RemoveContainer" containerID="e5da16f94cfa021b84030af532b1504fd9f93937bc4c589e1335c269ad0f9c26"
Jul 17 08:08:18.053375 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.053286    2584 scope.go:117] "RemoveContainer" containerID="bba81876375b16f5b9766a50a3c72bb9db4642d72fdaa63fba42874ef6140e3a"
Jul 17 08:08:18.053519 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:18.053498    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4_odh-ai-gateway-infra(8a646863-39d5-49a2-a29a-8b89e3b6319e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" podUID="8a646863-39d5-49a2-a29a-8b89e3b6319e"
Jul 17 08:08:18.054456 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.054434    2584 generic.go:356] "Generic (PLEG): container finished" podID="ad337426-1741-4a7c-9eeb-3bc574db28f3" containerID="8e1a29071b36d9e10a2d34feaad785d490444778522266ca3c46f65674fbb9e3" exitCode=6
Jul 17 08:08:18.054563 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.054521    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq" event={"ID":"ad337426-1741-4a7c-9eeb-3bc574db28f3","Type":"ContainerDied","Data":"8e1a29071b36d9e10a2d34feaad785d490444778522266ca3c46f65674fbb9e3"}
Jul 17 08:08:18.054616 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.054574    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq" event={"ID":"ad337426-1741-4a7c-9eeb-3bc574db28f3","Type":"ContainerStarted","Data":"bc86aed7cacb5266e714399458ef9f9ec1d9a0a638ec42ef52d15bd044143c96"}
Jul 17 08:08:18.054657 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.054643    2584 scope.go:117] "RemoveContainer" containerID="8e1a29071b36d9e10a2d34feaad785d490444778522266ca3c46f65674fbb9e3"
Jul 17 08:08:18.518940 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.518916    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hmgdm_239fcabd-a0a1-4152-b8af-6097fbeaf5db/migrator/0.log"
Jul 17 08:08:18.547695 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:18.547670    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-hmgdm_239fcabd-a0a1-4152-b8af-6097fbeaf5db/graceful-termination/0.log"
Jul 17 08:08:19.062490 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.062460    2584 generic.go:356] "Generic (PLEG): container finished" podID="ad337426-1741-4a7c-9eeb-3bc574db28f3" containerID="910862767c20781c3ada5f4a08b9c6db6103dcfde9deccd898c099efbcdd6316" exitCode=6
Jul 17 08:08:19.062967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.062537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq" event={"ID":"ad337426-1741-4a7c-9eeb-3bc574db28f3","Type":"ContainerDied","Data":"910862767c20781c3ada5f4a08b9c6db6103dcfde9deccd898c099efbcdd6316"}
Jul 17 08:08:19.062967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.062597    2584 scope.go:117] "RemoveContainer" containerID="8e1a29071b36d9e10a2d34feaad785d490444778522266ca3c46f65674fbb9e3"
Jul 17 08:08:19.062967 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.062761    2584 scope.go:117] "RemoveContainer" containerID="910862767c20781c3ada5f4a08b9c6db6103dcfde9deccd898c099efbcdd6316"
Jul 17 08:08:19.063105 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:19.062984    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq_odh-ai-gateway-infra(ad337426-1741-4a7c-9eeb-3bc574db28f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq" podUID="ad337426-1741-4a7c-9eeb-3bc574db28f3"
Jul 17 08:08:19.090987 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.090963    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:19.227459 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.227426    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:19.424205 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.424174    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca\") pod \"8a646863-39d5-49a2-a29a-8b89e3b6319e\" (UID: \"8a646863-39d5-49a2-a29a-8b89e3b6319e\") "
Jul 17 08:08:19.424543 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.424522    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a646863-39d5-49a2-a29a-8b89e3b6319e" (UID: "8a646863-39d5-49a2-a29a-8b89e3b6319e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 08:08:19.525019 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:19.524943    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a646863-39d5-49a2-a29a-8b89e3b6319e-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\""
Jul 17 08:08:20.069262 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.069233    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"
Jul 17 08:08:20.069747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.069239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4" event={"ID":"8a646863-39d5-49a2-a29a-8b89e3b6319e","Type":"ContainerDied","Data":"e226f7f877046215e213e9ea8561cd72f44d87a35a139a322bb3f318e7011ad4"}
Jul 17 08:08:20.069747 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.069400    2584 scope.go:117] "RemoveContainer" containerID="bba81876375b16f5b9766a50a3c72bb9db4642d72fdaa63fba42874ef6140e3a"
Jul 17 08:08:20.071131 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.071114    2584 scope.go:117] "RemoveContainer" containerID="910862767c20781c3ada5f4a08b9c6db6103dcfde9deccd898c099efbcdd6316"
Jul 17 08:08:20.071308 ip-10-0-138-131 kubenswrapper[2584]: E0717 08:08:20.071288    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq_odh-ai-gateway-infra(ad337426-1741-4a7c-9eeb-3bc574db28f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-370a9d5a-88fvq" podUID="ad337426-1741-4a7c-9eeb-3bc574db28f3"
Jul 17 08:08:20.110593 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.110567    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:20.116528 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.116503    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b531d224-w6mn4"]
Jul 17 08:08:20.290169 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.290134    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a646863-39d5-49a2-a29a-8b89e3b6319e" path="/var/lib/kubelet/pods/8a646863-39d5-49a2-a29a-8b89e3b6319e/volumes"
Jul 17 08:08:20.419374 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.419271    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/kube-multus-additional-cni-plugins/0.log"
Jul 17 08:08:20.445007 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.444980    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/egress-router-binary-copy/0.log"
Jul 17 08:08:20.468804 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.468777    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/cni-plugins/0.log"
Jul 17 08:08:20.490719 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.490693    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/bond-cni-plugin/0.log"
Jul 17 08:08:20.513483 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.513454    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/routeoverride-cni/0.log"
Jul 17 08:08:20.534771 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.534751    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/whereabouts-cni-bincopy/0.log"
Jul 17 08:08:20.558700 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.558677    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf2cz_5f12d13b-a6ab-4340-a4f2-35501cb9566e/whereabouts-cni/0.log"
Jul 17 08:08:20.842457 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.842422    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pg8xv_ac1a6253-e8fe-4df5-8dc9-2c52026b78c1/kube-multus/0.log"
Jul 17 08:08:20.872144 ip-10-0-138-131 kubenswrapper[2584]: I0717 08:08:20.872104    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fv7dc_c11dcfbf-059f-4cac-b79a-4206b3e6e8b1/network-metrics-daemon/0.log"